New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Bug 1866399: Shorten retry interval in must-gather #527
Bug 1866399: Shorten retry interval in must-gather #527
Conversation
@sallyom: This pull request references Bugzilla bug 1866399, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker. 3 validation(s) were run on this bug
In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
One nit
/approve
@@ -399,7 +399,7 @@ func newPrefixWriter(out io.Writer, prefix string) io.Writer { | |||
|
|||
func (o *MustGatherOptions) waitForPodRunning(pod *corev1.Pod) error { | |||
phase := pod.Status.Phase | |||
err := wait.PollImmediate(time.Minute, time.Duration(o.Timeout)*time.Second, func() (bool, error) { | |||
err := wait.PollImmediate(3*time.Second, time.Duration(o.Timeout)*time.Second, func() (bool, error) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Running means that the init container that does the heavy listing finished its job, which on some clusters might take a while. So instead of poking this every 3s, which is too much I'd bump this to 10s, maybe, wdyt @sallyom ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
from testing, I believe 3 sec is all we need, the cmd does this:
- wait for init container running (if you look at the must-gather pod at this point it will show
Init:0/1
while init container runs. (this is the 1st 3 second retry - I haven't seen it take longer than 2 retries) - gather container logs - this happens in the init container
- wait for must-gather pod Running - this doesn't start until the init container is done, so 3 seconds is also good for this one - I haven't seen this take longer than 2 retries, either. Once the init container completes (step 2 above) container turns to Running.
With this change, we actually cut a couple of unnecessary minutes off of must-gather!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
oki doke
/retest |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
/lgtm
@@ -399,7 +399,7 @@ func newPrefixWriter(out io.Writer, prefix string) io.Writer { | |||
|
|||
func (o *MustGatherOptions) waitForPodRunning(pod *corev1.Pod) error { | |||
phase := pod.Status.Phase | |||
err := wait.PollImmediate(time.Minute, time.Duration(o.Timeout)*time.Second, func() (bool, error) { | |||
err := wait.PollImmediate(3*time.Second, time.Duration(o.Timeout)*time.Second, func() (bool, error) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
oki doke
/retest |
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: sallyom, soltysh The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
/retest Please review the full test history for this PR and help us cut down flakes. |
2 similar comments
/retest Please review the full test history for this PR and help us cut down flakes. |
/retest Please review the full test history for this PR and help us cut down flakes. |
@sallyom: All pull requests linked via external trackers have merged: Bugzilla bug 1866399 has been moved to the MODIFIED state. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
This cuts down must-gather time by ~2 min.