Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bug 1888192: add rsync & log re-tries #631

Merged
merged 4 commits into from Feb 2, 2021
Merged

Bug 1888192: add rsync & log re-tries #631

merged 4 commits into from Feb 2, 2021

Conversation

soltysh
Copy link
Member

@soltysh soltysh commented Oct 30, 2020 •
edited

There are 2 main issues with oc adm must-gather that this PR addresses:

  1. On the env, the logs streaming from gather pod ends after ~10min, when the gather runs longer and on that cluster I was given access it was taking ~30mins, you'll only get info about the initial 10mins. The time-out is coming from server, so we just need to ensure we re-try streaming logs after that time-out if the pod is still running.
  2. Current copy logic is missing compression.
  3. Current sync logic is copying the artifacts only once, if it fails it doesn't re-try.

/assign @sallyom

@openshift-ci-robot openshift-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Oct 30, 2020
@openshift-ci-robot openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 30, 2020
@openshift-merge-robot
Copy link
Contributor

@soltysh: The following tests failed, say /retest to rerun all failed tests:

Test name Commit Details Rerun command
ci/prow/e2e-aws-upgrade a00be68 link /test e2e-aws-upgrade
ci/prow/e2e-aws a00be68 link /test e2e-aws
ci/prow/e2e-agnostic-cmd a00be68 link /test e2e-agnostic-cmd
ci/prow/e2e-aws-serial a00be68 link /test e2e-aws-serial

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@soltysh soltysh changed the title WIP: must-gather improvements Bug 1888192: add rsync & log re-tries Jan 18, 2021
@openshift-ci-robot openshift-ci-robot added bugzilla/severity-high Referenced Bugzilla bug's severity is high for the branch this PR is targeting. bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. and removed do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. labels Jan 18, 2021
@openshift-ci-robot
Copy link

@soltysh: This pull request references Bugzilla bug 1888192, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target release (4.7.0) matches configured target release for branch (4.7.0)
  • bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

In response to this:

Bug 1888192: add rsync & log re-tries

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@soltysh
Copy link
Member Author

soltysh commented Jan 19, 2021

/retest

@openshift-ci-robot
Copy link

@soltysh: This pull request references Bugzilla bug 1888192, which is valid.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target release (4.7.0) matches configured target release for branch (4.7.0)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

In response to this:

Bug 1888192: add rsync & log re-tries

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci-robot
Copy link

@soltysh: This pull request references Bugzilla bug 1888192, which is valid.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target release (4.7.0) matches configured target release for branch (4.7.0)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

In response to this:

Bug 1888192: add rsync & log re-tries

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@soltysh
Copy link
Member Author

soltysh commented Jan 19, 2021

/retest

sallyom
sallyom suggested changes Jan 21, 2021
View reviewed changes
Copy link
Contributor

@sallyom sallyom left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion to move the isGatherDone b4 the lost logs message, other than that lgtm.

pkg/cli/admin/mustgather/mustgather.go
// to ensure we don't print all of history set since to past 2 seconds
opts.Options.(*corev1.PodLogOptions).SinceSeconds = &since2s

if done, _ := o.isGatherDone(pod); done {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

move this above the klog "lost logs, re-trying", so that only prints if !isGatherDone -
(see easily w/ oc adm must-gather --dest-dir=/tmp/must-gather -v=4 -- date)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point, updated.

@soltysh soltysh mentioned this pull request Jan 22, 2021
Closed
@sallyom
Copy link
Contributor

sallyom commented Jan 25, 2021

/lgtm

@openshift-ci-robot openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Jan 25, 2021
@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

4 similar comments
@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

7 similar comments
@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@soltysh
Copy link
Member Author

soltysh commented Jan 26, 2021

/hold
I want to add timestamps to log lines

@openshift-ci-robot openshift-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jan 26, 2021
@soltysh
Copy link
Member Author

soltysh commented Jan 26, 2021

/hold cancel

@openshift-ci-robot openshift-ci-robot removed do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. lgtm Indicates that a PR is ready to be merged. labels Jan 26, 2021
@soltysh
Copy link
Member Author

soltysh commented Jan 26, 2021

/retest

3 similar comments
@soltysh
Copy link
Member Author

soltysh commented Jan 26, 2021

/retest

@soltysh
Copy link
Member Author

soltysh commented Jan 27, 2021

/retest

@soltysh
Copy link
Member Author

soltysh commented Feb 2, 2021

/retest

@sallyom
Copy link
Contributor

sallyom commented Feb 2, 2021

/lgtm

@openshift-ci-robot openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Feb 2, 2021
@openshift-ci-robot
Copy link

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: sallyom, soltysh

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-merge-robot openshift-merge-robot merged commit 8ad72f1 into openshift:master Feb 2, 2021
@openshift-ci-robot
Copy link

@soltysh: All pull requests linked via external trackers have merged:

Bugzilla bug 1888192 has been moved to the MODIFIED state.

In response to this:

Bug 1888192: add rsync & log re-tries

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@soltysh soltysh deleted the mustgather_fixes branch February 3, 2021 12:11
@damemi
Copy link

damemi commented Mar 9, 2021

/cherry-pick release-4.6

@openshift-cherrypick-robot openshift-cherrypick-robot mentioned this pull request Mar 9, 2021
Merged
@openshift-cherrypick-robot
Copy link

@damemi: new pull request created: #760

In response to this:

/cherry-pick release-4.6

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@wking wking mentioned this pull request Aug 17, 2021
Merged
wking added a commit to wking/openshift-release that referenced this pull request Nov 11, 2021
@wking
ci-operator: Replace --timeout with --request-timeout for rsh calls
b155fb1 
The --timeout approach landed back in ad2dbe0 (Dump iptables on
every node during teardown, 2019-01-24, openshift#2652).  But oc dropped the
deprecated --timeout option in openshift/oc@8931142584 (Remove
deprecated --timeout flag from oc rsh, 2021-01-25, openshift/oc#631).
That change landed in 4.7 and was backported to 4.6.23 [1], but not to
4.5.

I've confirmed with Dan Winship that we still want the iptables saves,
and don't just want to drop these blocks.

Generated with:

  $ sed -i 's/rsh --timeout=20/--request-timeout=20s rsh/' $(git --no-pager grep -l 'rsh.*timeout' ci-operator)

[1]: https://bugzilla.redhat.com/show_bug.cgi?id=1926172#c7
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sallyom sallyom sallyom requested changes

@deads2k deads2k Awaiting requested review from deads2k

@smarterclayton smarterclayton Awaiting requested review from smarterclayton

Assignees

@sallyom sallyom

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. bugzilla/severity-high Referenced Bugzilla bug's severity is high for the branch this PR is targeting. bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. lgtm Indicates that a PR is ready to be merged.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
7 participants
@soltysh @openshift-merge-robot @openshift-ci-robot @sallyom @openshift-bot @damemi @openshift-cherrypick-robot