bug 1942938: [release-4.7] inspect clusteroperators as a backup to must-gather if it fails #766

ingvagabund · 2021-03-15T15:07:36Z

Backporting:

openshift-ci-robot · 2021-03-15T15:07:38Z

@ingvagabund: No Bugzilla bug is referenced in the title of this pull request.
To reference a bug, add 'Bug XXX:' to the title of this pull request and request another bug refresh with /bugzilla refresh.

In response to this:

[release-4.7] inspect clusteroperators as a backup to must-gather if it fails

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot · 2021-03-15T18:28:07Z

@ingvagabund: No Bugzilla bug is referenced in the title of this pull request.
To reference a bug, add 'Bug XXX:' to the title of this pull request and request another bug refresh with /bugzilla refresh.

In response to this:

[release-4.7] inspect clusteroperators as a backup to must-gather if it fails

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

We now gather alerts from the monitoring stack and this takes a long time run. Also, when we have problems we normally also have networking issues so these rarely work when there is something actually broken.

ingvagabund · 2021-03-16T12:49:42Z

Waiting until #763 gets merged as well. After that getting:

$ ./oc adm must-gather
[must-gather      ] OUT Using must-gather plug-in image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:172a9f8aa34966b69dfa485179c6d5109db66d7d7018029dae12ff83c47a2c5d
Gathering data for ns/openshift-config...
Gathering data for ns/openshift-config-managed...
Gathering data for ns/openshift-authentication...
Gathering data for ns/openshift-authentication-operator...
Gathering data for ns/openshift-ingress...
Gathering data for ns/openshift-oauth-apiserver...
Gathering data for ns/openshift-machine-api...
Gathering data for ns/openshift-cloud-credential-operator...
Gathering data for ns/openshift-config-operator...
Gathering data for ns/openshift-console-operator...
Gathering data for ns/openshift-console...
Gathering data for ns/openshift-cluster-storage-operator...
Gathering data for ns/openshift-dns-operator...
Gathering data for ns/openshift-dns...
Gathering data for ns/openshift-etcd-operator...
Gathering data for ns/openshift-etcd...
Gathering data for ns/openshift-image-registry...
Gathering data for ns/openshift-ingress-operator...
Gathering data for ns/openshift-ingress-canary...
Gathering data for ns/openshift-insights...
Gathering data for ns/openshift-kube-apiserver-operator...
Gathering data for ns/openshift-kube-apiserver...
Gathering data for ns/openshift-kube-controller-manager...
Gathering data for ns/openshift-kube-controller-manager-operator...
Gathering data for ns/openshift-kube-scheduler...
Gathering data for ns/openshift-kube-scheduler-operator...
Gathering data for ns/openshift-kube-storage-version-migrator...
Gathering data for ns/openshift-kube-storage-version-migrator-operator...
Gathering data for ns/openshift-cluster-machine-approver...
Gathering data for ns/openshift-machine-config-operator...
Gathering data for ns/openshift-marketplace...
Gathering data for ns/openshift-monitoring...
Gathering data for ns/openshift-user-workload-monitoring...
Gathering data for ns/openshift-multus...
Gathering data for ns/openshift-sdn...
Gathering data for ns/openshift-network-diagnostics...
Gathering data for ns/openshift-network-operator...
Gathering data for ns/openshift-cluster-node-tuning-operator...
Gathering data for ns/openshift-apiserver-operator...
Gathering data for ns/openshift-apiserver...
Gathering data for ns/openshift-controller-manager-operator...
Gathering data for ns/openshift-controller-manager...
Gathering data for ns/openshift-cluster-samples-operator...
Gathering data for ns/openshift...
Gathering data for ns/openshift-operator-lifecycle-manager...
Gathering data for ns/openshift-service-ca-operator...
Gathering data for ns/openshift-service-ca...
Gathering data for ns/openshift-cluster-csi-drivers...
Wrote inspect data to must-gather.local.2241645993367180459.
error running backup collection: errors ocurred while gathering data:
    [skipping gathering secrets/support due to error: secrets "support" not found, skipping gathering EgressFirewall.k8s.ovn.org due to error: the server doesn't have a resource type "EgressFirewall", skipping gathering EgressIP.k8s.ovn.org due to error: the server doesn't have a resource type "EgressIP", skipping gathering endpoints/host-etcd-2 due to error: endpoints "host-etcd-2" not found]error: Failing to test BackupGathering

ingvagabund · 2021-03-25T10:52:13Z

/retest

openshift-ci-robot · 2021-03-25T11:21:18Z

@ingvagabund: This pull request references Bugzilla bug 1942938, which is invalid:

expected dependent Bugzilla bug 1942935 to be in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), CLOSED (CURRENTRELEASE), but it is MODIFIED instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

bug 1942938: [release-4.7] inspect clusteroperators as a backup to must-gather if it fails

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

ingvagabund · 2021-03-25T11:26:14Z

/retest

openshift-ci-robot · 2021-03-25T11:30:15Z

@ingvagabund: This pull request references Bugzilla bug 1942938, which is invalid:

expected dependent Bugzilla bug 1942935 to be in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), CLOSED (CURRENTRELEASE), but it is MODIFIED instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

bug 1942938: [release-4.7] inspect clusteroperators as a backup to must-gather if it fails

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot · 2021-03-25T12:46:33Z

@ingvagabund: This pull request references Bugzilla bug 1942938, which is invalid:

expected dependent Bugzilla bug 1942935 to be in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), CLOSED (CURRENTRELEASE), but it is ON_QA instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

bug 1942938: [release-4.7] inspect clusteroperators as a backup to must-gather if it fails

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot · 2021-03-25T12:55:51Z

@ingvagabund: This pull request references Bugzilla bug 1942938, which is invalid:

expected dependent Bugzilla bug 1942935 to be in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), CLOSED (CURRENTRELEASE), but it is ON_QA instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

bug 1942938: [release-4.7] inspect clusteroperators as a backup to must-gather if it fails

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

soltysh

/lgtm
/approve

ingvagabund · 2021-03-25T15:53:51Z

/retest

openshift-bot · 2021-03-25T19:20:45Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-03-26T01:00:20Z

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

openshift-ci-robot · 2021-03-26T01:00:25Z

@openshift-bot: This pull request references Bugzilla bug 1942938, which is invalid:

expected dependent Bugzilla bug 1942935 to be in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), CLOSED (CURRENTRELEASE), but it is ON_QA instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-bot · 2021-03-26T02:03:10Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-03-26T03:34:11Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-03-27T01:00:20Z

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

openshift-ci-robot · 2021-03-27T01:00:24Z

@openshift-bot: This pull request references Bugzilla bug 1942938, which is invalid:

expected dependent Bugzilla bug 1942935 to be in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), CLOSED (CURRENTRELEASE), but it is ON_QA instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-bot · 2021-03-28T01:01:05Z

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

openshift-ci-robot · 2021-03-28T01:01:08Z

@openshift-bot: This pull request references Bugzilla bug 1942938, which is invalid:

expected dependent Bugzilla bug 1942935 to be in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), CLOSED (CURRENTRELEASE), but it is ON_QA instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-bot · 2021-03-29T01:01:05Z

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

openshift-ci-robot · 2021-03-29T01:01:08Z

@openshift-bot: This pull request references Bugzilla bug 1942938, which is invalid:

expected dependent Bugzilla bug 1942935 to be in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), CLOSED (CURRENTRELEASE), but it is ON_QA instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-bot · 2021-03-30T01:00:21Z

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

openshift-ci-robot · 2021-03-30T01:00:25Z

@openshift-bot: This pull request references Bugzilla bug 1942938, which is invalid:

expected dependent Bugzilla bug 1942935 to be in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), CLOSED (CURRENTRELEASE), but it is ON_QA instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-bot · 2021-03-31T01:00:55Z

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

openshift-ci-robot · 2021-03-31T01:00:58Z

@openshift-bot: This pull request references Bugzilla bug 1942938, which is invalid:

expected dependent Bugzilla bug 1942935 to be in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), CLOSED (CURRENTRELEASE), but it is ON_QA instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-bot · 2021-04-01T01:01:00Z

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

openshift-ci-robot · 2021-04-01T01:01:10Z

@openshift-bot: This pull request references Bugzilla bug 1942938, which is valid. The bug has been moved to the POST state.

6 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.7.z) matches configured target release for branch (4.7.z)
bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)
dependent bug Bugzilla bug 1942935 is in the state VERIFIED, which is one of the valid states (VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), CLOSED (CURRENTRELEASE))
dependent Bugzilla bug 1942935 targets the "4.8.0" release, which is one of the valid target releases: 4.8.0
bug has dependents

Requesting review from QA contact:
/cc @zhouying7780

In response to this:

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

zhouying7780 · 2021-04-06T08:27:53Z

/lgtm

zhouying7780 · 2021-04-06T08:28:04Z

/label qe-approved

openshift-ci-robot · 2021-04-06T08:28:26Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ingvagabund, soltysh, zhouying7780

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [soltysh]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-bot · 2021-04-11T00:44:19Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-04-11T02:15:17Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-ci-robot · 2021-04-11T02:54:23Z

@ingvagabund: All pull requests linked via external trackers have merged:

openshift/oc#766

Bugzilla bug 1942938 has been moved to the MODIFIED state.

In response to this:

bug 1942938: [release-4.7] inspect clusteroperators as a backup to must-gather if it fails

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

inspect clusteroperators as a backup to must-gather if it fails

fa72722

openshift-ci-robot requested review from deads2k and soltysh March 15, 2021 15:08

stop trying to gather metrics and other endpoints directly from pods

30f90f3

We now gather alerts from the monitoring stack and this takes a long time run. Also, when we have problems we normally also have networking issues so these rarely work when there is something actually broken.

ingvagabund changed the title ~~[release-4.7] inspect clusteroperators as a backup to must-gather if it fails~~ bug 1942938: [release-4.7] inspect clusteroperators as a backup to must-gather if it fails Mar 25, 2021

openshift-ci-robot added bugzilla/severity-low Referenced Bugzilla bug's severity is low for the branch this PR is targeting. bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. labels Mar 25, 2021

deads2k and others added 4 commits March 25, 2021 13:38

prevent inspect from panic-ing if pods are missing

3600f4e

indicate how many bytes the hidden secret was

e6a8b11

add basic cluster summary to must-gather

2258995

update vendor

5e48711

This was referenced Mar 25, 2021

bug 1942938: [release-4.7] prevent inspect from panic-ing if pods are missing #776

Closed

bug 1942938: [release-4.7] indicate how many bytes the hidden secret key was #783

Closed

bug 1942938: [release-4.7] Add summary to oc must-gather #786

Closed

soltysh approved these changes Mar 25, 2021

View reviewed changes

openshift-ci-robot assigned soltysh Mar 25, 2021

openshift-ci-robot added lgtm Indicates that a PR is ready to be merged. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Mar 25, 2021

openshift-ci-robot added the bugzilla/severity-high Referenced Bugzilla bug's severity is high for the branch this PR is targeting. label Mar 26, 2021

openshift-ci-robot added the bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. label Apr 1, 2021

openshift-ci-robot removed the bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. label Apr 1, 2021

openshift-ci-robot requested a review from zhouying7780 April 1, 2021 01:01

openshift-ci-robot added the qe-approved Signifies that QE has signed off on this PR label Apr 6, 2021

openshift-ci-robot assigned zhouying7780 Apr 6, 2021

russellb added the cherry-pick-approved Indicates a cherry-pick PR into a release branch has been approved by the release branch manager. label Apr 9, 2021

openshift-merge-robot merged commit 95881af into openshift:release-4.7 Apr 11, 2021

ingvagabund deleted the backport-749 branch April 12, 2021 06:21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bug 1942938: [release-4.7] inspect clusteroperators as a backup to must-gather if it fails #766

bug 1942938: [release-4.7] inspect clusteroperators as a backup to must-gather if it fails #766

ingvagabund commented Mar 15, 2021 •

edited

openshift-ci-robot commented Mar 15, 2021

openshift-ci-robot commented Mar 15, 2021

ingvagabund commented Mar 16, 2021

ingvagabund commented Mar 25, 2021

openshift-ci-robot commented Mar 25, 2021

ingvagabund commented Mar 25, 2021

openshift-ci-robot commented Mar 25, 2021

openshift-ci-robot commented Mar 25, 2021

openshift-ci-robot commented Mar 25, 2021

soltysh left a comment

ingvagabund commented Mar 25, 2021

openshift-bot commented Mar 25, 2021

openshift-bot commented Mar 26, 2021

openshift-ci-robot commented Mar 26, 2021

openshift-bot commented Mar 26, 2021

openshift-bot commented Mar 26, 2021

openshift-bot commented Mar 27, 2021

openshift-ci-robot commented Mar 27, 2021

openshift-bot commented Mar 28, 2021

openshift-ci-robot commented Mar 28, 2021

openshift-bot commented Mar 29, 2021

openshift-ci-robot commented Mar 29, 2021

openshift-bot commented Mar 30, 2021

openshift-ci-robot commented Mar 30, 2021

openshift-bot commented Mar 31, 2021

openshift-ci-robot commented Mar 31, 2021

openshift-bot commented Apr 1, 2021

openshift-ci-robot commented Apr 1, 2021

zhouying7780 commented Apr 6, 2021

zhouying7780 commented Apr 6, 2021

openshift-ci-robot commented Apr 6, 2021

openshift-bot commented Apr 11, 2021

openshift-bot commented Apr 11, 2021

openshift-ci-robot commented Apr 11, 2021

bug 1942938: [release-4.7] inspect clusteroperators as a backup to must-gather if it fails #766

bug 1942938: [release-4.7] inspect clusteroperators as a backup to must-gather if it fails #766

Conversation

ingvagabund commented Mar 15, 2021 • edited

openshift-ci-robot commented Mar 15, 2021

openshift-ci-robot commented Mar 15, 2021

ingvagabund commented Mar 16, 2021

ingvagabund commented Mar 25, 2021

openshift-ci-robot commented Mar 25, 2021

ingvagabund commented Mar 25, 2021

openshift-ci-robot commented Mar 25, 2021

openshift-ci-robot commented Mar 25, 2021

openshift-ci-robot commented Mar 25, 2021

soltysh left a comment

Choose a reason for hiding this comment

ingvagabund commented Mar 25, 2021

openshift-bot commented Mar 25, 2021

openshift-bot commented Mar 26, 2021

openshift-ci-robot commented Mar 26, 2021

openshift-bot commented Mar 26, 2021

openshift-bot commented Mar 26, 2021

openshift-bot commented Mar 27, 2021

openshift-ci-robot commented Mar 27, 2021

openshift-bot commented Mar 28, 2021

openshift-ci-robot commented Mar 28, 2021

openshift-bot commented Mar 29, 2021

openshift-ci-robot commented Mar 29, 2021

openshift-bot commented Mar 30, 2021

openshift-ci-robot commented Mar 30, 2021

openshift-bot commented Mar 31, 2021

openshift-ci-robot commented Mar 31, 2021

openshift-bot commented Apr 1, 2021

openshift-ci-robot commented Apr 1, 2021

zhouying7780 commented Apr 6, 2021

zhouying7780 commented Apr 6, 2021

openshift-ci-robot commented Apr 6, 2021

openshift-bot commented Apr 11, 2021

openshift-bot commented Apr 11, 2021

openshift-ci-robot commented Apr 11, 2021

ingvagabund commented Mar 15, 2021 •

edited