Bug 1987046: Add pre-puller ds to reduce upgrade downtime #1167
Conversation
|
@tssurya: No Bugzilla bug is referenced in the title of this pull request. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
/retitle Bug 1987046: Add pre-puller ds to reduce upgrade downtime |
openshift-ci
bot
changed the title
openshift-ci
bot
added
bugzilla/severity-high
|
@tssurya: This pull request references Bugzilla bug 1987046, which is invalid:
Comment In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
/bugzilla refresh |
openshift-ci
bot
added
the
bugzilla/valid-bug
|
@vrutkovs: This pull request references Bugzilla bug 1987046, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker. 6 validation(s) were run on this bug
No GitHub users were found matching the public email listed for the QA contact in Bugzilla (anusaxen@redhat.com), skipping review request. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
openshift-ci
bot
removed
the
bugzilla/invalid-bug
|
/retest |
|
Still seeing a lot of disruption:
|
|
/retest |
1 similar comment
|
/retest |
|
Ultimately I think the best fix for this is that we do "pull through" caching in the registry instead of having each node pull each container. |
This solution of prepull is not perfect. The fact that there is disruption for 26 minutes cannot be solely due to OVN-K. It must be a combo of dependent components restarting during upgrades? |
I don't know how that would look, but then again on prod's I don't know if the registries will be co-located? This PR doesn't do much alongside a run in CI (not sure how the CI setup looks like), but in larger envs, we expect more benefit. |
The only difference between e2e-gcp-ovn-upgrade and e2e-gcp-upgrade is OVN-K |
|
@vrutkovs : Should we get this fix in into 4.8? Or do we think we need a new fix to solve the service disruption issue and close this? |
|
I think this fix would be nice in 4.8, but clearly its not sufficient to fix all disruptions |
|
/lgtm |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: abhat, tssurya The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
bot
added
the
approved
|
/test e2e-gcp-ovn-upgrade |
|
/retest-required Please review the full test history for this PR and help us cut down flakes. |
|
/retest |
jupierce
added
the
cherry-pick-approved
|
/retest-required Please review the full test history for this PR and help us cut down flakes. |
4 similar comments
|
/retest-required Please review the full test history for this PR and help us cut down flakes. |
|
/retest-required Please review the full test history for this PR and help us cut down flakes. |
|
/retest-required Please review the full test history for this PR and help us cut down flakes. |
|
/retest-required Please review the full test history for this PR and help us cut down flakes. |
|
@tssurya: The following test failed, say
Full PR test history. Your PR dashboard. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. |
|
/retest-required Please review the full test history for this PR and help us cut down flakes. |
8 similar comments
|
/retest-required Please review the full test history for this PR and help us cut down flakes. |
|
/retest-required Please review the full test history for this PR and help us cut down flakes. |
|
/retest-required Please review the full test history for this PR and help us cut down flakes. |
|
/retest-required Please review the full test history for this PR and help us cut down flakes. |
|
/retest-required Please review the full test history for this PR and help us cut down flakes. |
|
/retest-required Please review the full test history for this PR and help us cut down flakes. |
|
/retest-required Please review the full test history for this PR and help us cut down flakes. |
|
/retest-required Please review the full test history for this PR and help us cut down flakes. |
|
@tssurya: All pull requests linked via external trackers have merged: Bugzilla bug 1987046 has been moved to the MODIFIED state. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
Manual Cherry-pick of #1141
Fixed conflicts due to #1145
cc @vrutkovs