OTA-362: pkg/monitortests/clusterversionoperator: Fatal unless Available=False in allow-list #27231

wking · 2022-06-08T00:05:51Z

Filling in known bugs from here (~~WIP: not actually full yet~~). This allows us to fail new regressions, and gradually ratchet tighter as we close out existing issues.

wking · 2022-06-08T16:08:15Z

Success:

I'll expand the exception list...

wking · 2022-06-08T17:18:28Z

WIP: not actually full yet

I've filled in the list of regexps in 78c5e91, so dropping the WIP.

wking · 2022-06-09T04:54:05Z

New phrasing:

jsafrane · 2022-06-28T08:17:37Z

pkg/synthetictests/operators.go

+		regexp.MustCompile(".*clusteroperator/kube-storage-version-migrator condition/Available status/False reason/KubeStorageVersionMigratorAvailable[ :].*"):                 "https://bugzilla.redhat.com/show_bug.cgi?id=1948087",
+		regexp.MustCompile(".*clusteroperator/kube-storage-version-migrator condition/Available status/False reason/KubeStorageVersionMigrator_Deploying[ :].*"):                "https://bugzilla.redhat.com/show_bug.cgi?id=1948087",
+		regexp.MustCompile(".*clusteroperator/machine-config condition/Available status/False reason/Cluster not available[ :].*"):                                              "https://bugzilla.redhat.com/show_bug.cgi?id=2051075",
+		regexp.MustCompile(".*clusteroperator/storage condition/Available status/False reason/AWSEBSCSIDriverOperatorCR_AWSEBSDriverControllerServiceController_Deploying .*"):  "https://bugzilla.redhat.com/show_bug.cgi?id=2081557",


I'd suggest reason/.*CSIDriverOperatorCR_.*DriverControllerServiceController_Deploying to catch all CSI driver operators. oVirt, IBM cloud, Alibaba, vSphere and OpenStack will have the same blips during upgrade.

and is this the right Bugzilla tracker for those blips?

Yes. I will flip it back to Assigned once the current fixes merge.

jsafrane · 2022-06-28T08:18:05Z

pkg/synthetictests/operators.go

+		regexp.MustCompile(".*clusteroperator/kube-storage-version-migrator condition/Available status/False reason/KubeStorageVersionMigrator_Deploying[ :].*"):                "https://bugzilla.redhat.com/show_bug.cgi?id=1948087",
+		regexp.MustCompile(".*clusteroperator/machine-config condition/Available status/False reason/Cluster not available[ :].*"):                                              "https://bugzilla.redhat.com/show_bug.cgi?id=2051075",
+		regexp.MustCompile(".*clusteroperator/storage condition/Available status/False reason/AWSEBSCSIDriverOperatorCR_AWSEBSDriverControllerServiceController_Deploying .*"):  "https://bugzilla.redhat.com/show_bug.cgi?id=2081557",
+		regexp.MustCompile(".*clusteroperator/storage condition/Available status/False reason/AWSEBSCSIDriverOperatorDeployment_Deploying[ :].*"):                               "https://bugzilla.redhat.com/show_bug.cgi?id=2081557",


dtto, reason/.*CSIDriverOperatorDeployment_Deploying

Is this the right Bugzilla tracker for those blips?

openshift-bot · 2023-02-04T01:00:22Z

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

openshift-bot · 2023-03-06T08:30:16Z

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

openshift-bot · 2023-04-06T00:00:17Z

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

openshift-ci · 2023-04-06T00:01:03Z

@openshift-bot: Closed this PR.

In response to this:

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

wking · 2023-10-03T17:50:12Z

/remove-lifecycle stale
/reopen

openshift-ci · 2023-10-03T17:50:27Z

@wking: Reopened this PR.

In response to this:

/remove-lifecycle stale
/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-trt-bot · 2023-12-01T07:31:25Z

Job Failure Risk Analysis for sha: 0b4bb1d

Job Name	Failure Risk
pull-ci-openshift-origin-master-e2e-aws-ovn-single-node-upgrade	IncompleteTests Tests for this run (17) are below the historical average (2167): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)
pull-ci-openshift-origin-master-e2e-aws-ovn-single-node-serial	IncompleteTests Tests for this run (15) are below the historical average (653): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)

wking · 2023-12-01T07:40:49Z

/retest-required

openshift-trt-bot · 2023-12-01T09:01:17Z

Job Failure Risk Analysis for sha: 0b4bb1d

Job Name	Failure Risk
pull-ci-openshift-origin-master-e2e-aws-ovn-single-node-upgrade	IncompleteTests Tests for this run (17) are below the historical average (2174): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)
pull-ci-openshift-origin-master-e2e-aws-ovn-single-node-serial	IncompleteTests Tests for this run (15) are below the historical average (653): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)

openshift-trt-bot · 2023-12-01T09:10:59Z

Job Failure Risk Analysis for sha: 0b4bb1d

Job Name	Failure Risk
pull-ci-openshift-origin-master-e2e-gcp-ovn	IncompleteTests Tests for this run (99) are below the historical average (1549): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)
pull-ci-openshift-origin-master-e2e-aws-ovn-single-node-upgrade	IncompleteTests Tests for this run (17) are below the historical average (2174): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)
pull-ci-openshift-origin-master-e2e-aws-ovn-single-node-serial	IncompleteTests Tests for this run (15) are below the historical average (653): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)

wking · 2023-12-01T15:33:29Z

openshift/cluster-storage-operator#426 might help.

/retest-required

openshift-trt-bot · 2023-12-01T18:14:40Z

Job Failure Risk Analysis for sha: 0b4bb1d

Job Name	Failure Risk
pull-ci-openshift-origin-master-e2e-aws-ovn-single-node-upgrade	IncompleteTests Tests for this run (17) are below the historical average (2205): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)
pull-ci-openshift-origin-master-e2e-aws-ovn-single-node-serial	IncompleteTests Tests for this run (15) are below the historical average (662): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)

wking · 2023-12-01T22:00:52Z

/retest-required

openshift-trt-bot · 2023-12-01T22:24:18Z

Job Failure Risk Analysis for sha: 0b4bb1d

Job Name	Failure Risk
pull-ci-openshift-origin-master-e2e-aws-ovn-single-node-upgrade	IncompleteTests Tests for this run (17) are below the historical average (2217): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)
pull-ci-openshift-origin-master-e2e-aws-ovn-single-node-serial	IncompleteTests Tests for this run (15) are below the historical average (664): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)

openshift-ci-robot · 2023-12-01T22:38:59Z

/retest-required

Remaining retests: 0 against base HEAD 80b0ef7 and 1 for PR HEAD 0b4bb1d in total

openshift-trt-bot · 2023-12-02T01:09:28Z

Job Failure Risk Analysis for sha: 0b4bb1d

Job Name	Failure Risk
pull-ci-openshift-origin-master-e2e-aws-ovn-single-node-upgrade	IncompleteTests Tests for this run (17) are below the historical average (2249): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)
pull-ci-openshift-origin-master-e2e-aws-ovn-single-node-serial	IncompleteTests Tests for this run (15) are below the historical average (655): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)

wking · 2023-12-02T04:52:33Z

openshift/cluster-cloud-controller-manager-operator#305 may have addressed the quay.io/openshift/origin-kube-rbac-proxy:4.2.0 failures.

/retest-required

openshift-trt-bot · 2023-12-02T10:40:31Z

Job Failure Risk Analysis for sha: 0b4bb1d

Job Name	Failure Risk
pull-ci-openshift-origin-master-e2e-aws-ovn-single-node-upgrade	IncompleteTests Tests for this run (17) are below the historical average (2260): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)
pull-ci-openshift-origin-master-e2e-aws-ovn-single-node-serial	IncompleteTests Tests for this run (15) are below the historical average (658): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)

openshift-trt-bot · 2023-12-02T11:09:26Z

Job Failure Risk Analysis for sha: 0b4bb1d

Job Name	Failure Risk
pull-ci-openshift-origin-master-e2e-aws-ovn-single-node-upgrade	IncompleteTests Tests for this run (17) are below the historical average (2260): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)
pull-ci-openshift-origin-master-e2e-aws-ovn-single-node-serial	IncompleteTests Tests for this run (15) are below the historical average (658): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)

wking · 2023-12-02T15:25:59Z

/retest-required

openshift-trt-bot · 2023-12-02T18:09:44Z

Job Failure Risk Analysis for sha: 0b4bb1d

Job Name	Failure Risk
pull-ci-openshift-origin-master-e2e-aws-ovn-single-node-upgrade	IncompleteTests Tests for this run (17) are below the historical average (2237): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)
pull-ci-openshift-origin-master-e2e-aws-ovn-single-node-serial	IncompleteTests Tests for this run (15) are below the historical average (661): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)

openshift-ci-robot · 2023-12-02T19:27:00Z

/retest-required

Remaining retests: 0 against base HEAD 5b587d9 and 0 for PR HEAD 0b4bb1d in total

openshift-ci-robot · 2023-12-02T22:35:46Z

/hold

Revision 0b4bb1d was retested 3 times: holding

openshift-trt-bot · 2023-12-03T00:34:42Z

Job Failure Risk Analysis for sha: 0b4bb1d

Job Name	Failure Risk
pull-ci-openshift-origin-master-e2e-aws-ovn-single-node-upgrade	IncompleteTests Tests for this run (17) are below the historical average (2247): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)
pull-ci-openshift-origin-master-e2e-aws-ovn-single-node-serial	IncompleteTests Tests for this run (15) are below the historical average (660): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)

openshift-trt-bot · 2023-12-03T01:09:32Z

Job Failure Risk Analysis for sha: 0b4bb1d

Job Name	Failure Risk
pull-ci-openshift-origin-master-e2e-aws-ovn-single-node-upgrade	IncompleteTests Tests for this run (17) are below the historical average (2247): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)
pull-ci-openshift-origin-master-e2e-aws-ovn-single-node-serial	IncompleteTests Tests for this run (15) are below the historical average (660): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)

wking · 2023-12-03T04:30:07Z

/hold cancel
/retest-required

openshift-ci-robot · 2023-12-03T05:35:52Z

/retest-required

Remaining retests: 0 against base HEAD 5b587d9 and 2 for PR HEAD 0b4bb1d in total

openshift-ci · 2023-12-03T07:08:45Z

@wking: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-agnostic-cmd	`054f30d`	link	false	`/test e2e-agnostic-cmd`
ci/prow/e2e-aws-single-node-upgrade	`054f30d`	link	false	`/test e2e-aws-single-node-upgrade`
ci/prow/e2e-gcp-ovn-image-ecosystem	`054f30d`	link	true	`/test e2e-gcp-ovn-image-ecosystem`
ci/prow/e2e-aws-ovn-single-node-serial	`0b4bb1d`	link	false	`/test e2e-aws-ovn-single-node-serial`
ci/prow/e2e-aws-csi	`0b4bb1d`	link	false	`/test e2e-aws-csi`
ci/prow/e2e-gcp-ovn-rt-upgrade	`0b4bb1d`	link	false	`/test e2e-gcp-ovn-rt-upgrade`
ci/prow/e2e-aws-ovn-single-node	`0b4bb1d`	link	false	`/test e2e-aws-ovn-single-node`
ci/prow/e2e-aws-ovn-upgrade	`0b4bb1d`	link	false	`/test e2e-aws-ovn-upgrade`
ci/prow/e2e-aws-ovn-single-node-upgrade	`0b4bb1d`	link	false	`/test e2e-aws-ovn-single-node-upgrade`
ci/prow/e2e-agnostic-ovn-cmd	`0b4bb1d`	link	false	`/test e2e-agnostic-ovn-cmd`
ci/prow/e2e-openstack-ovn	`0b4bb1d`	link	false	`/test e2e-openstack-ovn`
ci/prow/e2e-aws-ovn-cgroupsv2	`0b4bb1d`	link	false	`/test e2e-aws-ovn-cgroupsv2`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

openshift-trt-bot · 2023-12-03T07:14:38Z

Job Failure Risk Analysis for sha: 0b4bb1d

Job Name	Failure Risk
pull-ci-openshift-origin-master-e2e-aws-ovn-single-node-upgrade	IncompleteTests Tests for this run (17) are below the historical average (2230): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)
pull-ci-openshift-origin-master-e2e-aws-ovn-single-node-serial	IncompleteTests Tests for this run (15) are below the historical average (661): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)
pull-ci-openshift-origin-master-e2e-aws-ovn-serial	IncompleteTests Tests for this run (16) are below the historical average (728): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)
pull-ci-openshift-origin-master-e2e-aws-ovn-fips	IncompleteTests Tests for this run (16) are below the historical average (1770): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)

openshift-trt-bot · 2023-12-03T08:09:37Z

Job Failure Risk Analysis for sha: 0b4bb1d

Job Name	Failure Risk
pull-ci-openshift-origin-master-e2e-aws-ovn-single-node-upgrade	IncompleteTests Tests for this run (17) are below the historical average (2230): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)
pull-ci-openshift-origin-master-e2e-aws-ovn-single-node-serial	IncompleteTests Tests for this run (15) are below the historical average (661): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)
pull-ci-openshift-origin-master-e2e-aws-ovn-serial	IncompleteTests Tests for this run (16) are below the historical average (728): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)
pull-ci-openshift-origin-master-e2e-aws-ovn-fips	IncompleteTests Tests for this run (16) are below the historical average (1770): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)

wking · 2023-12-03T17:18:15Z

gmp-devel-1:6.1.2-10.el8.x86_64: Cannot download, all mirrors were already tried without success is unrelated to my pull, although searching doesn't turn up hits in other places. Maybe a temporary hiccup?

/retest-required

openshift-bot · 2023-12-04T08:30:29Z

[ART PR BUILD NOTIFIER]

This PR has been included in build openshift-enterprise-tests-container-v4.15.0-202312040732.p0.gd75092b.assembly.stream for distgit openshift-enterprise-tests.
All builds following this will include this PR.

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 8, 2022

openshift-ci bot requested review from jwforres and spadgett June 8, 2022 00:08

wking force-pushed the fatal-available-false branch from be29e72 to f2c36c4 Compare June 8, 2022 02:32

wking force-pushed the fatal-available-false branch from f2c36c4 to 29b556d Compare June 8, 2022 16:32

wking changed the title ~~WIP: pkg/synthetictests/operators: Fatal unless Available=False in allow-list~~ pkg/synthetictests/operators: Fatal unless Available=False in allow-list Jun 8, 2022

openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 8, 2022

wking force-pushed the fatal-available-false branch from 29b556d to 78c5e91 Compare June 8, 2022 17:18

wking force-pushed the fatal-available-false branch 2 times, most recently from 6dc7c2b to bedc17e Compare June 8, 2022 23:19

wking force-pushed the fatal-available-false branch 2 times, most recently from 8c71586 to 054f30d Compare June 9, 2022 14:04

jsafrane reviewed Jun 28, 2022

View reviewed changes

openshift-ci bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Feb 4, 2023

openshift-ci bot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Mar 6, 2023

openshift-ci bot closed this Apr 6, 2023

openshift-ci bot reopened this Oct 3, 2023

openshift-merge-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 3, 2023

wking force-pushed the fatal-available-false branch from 054f30d to 594f368 Compare October 3, 2023 17:52

openshift-merge-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 3, 2023

openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Dec 2, 2023

openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Dec 3, 2023

openshift-merge-bot bot merged commit d75092b into openshift:master Dec 3, 2023
13 of 22 checks passed

wking deleted the fatal-available-false branch December 3, 2023 21:12

OTA-362: pkg/monitortests/clusterversionoperator: Fatal unless Available=False in allow-list #27231

OTA-362: pkg/monitortests/clusterversionoperator: Fatal unless Available=False in allow-list #27231

Conversation

wking commented Jun 8, 2022 • edited

wking commented Jun 8, 2022

wking commented Jun 8, 2022

wking commented Jun 9, 2022

jsafrane Jun 28, 2022

Choose a reason for hiding this comment

wking Jun 29, 2022

Choose a reason for hiding this comment

jsafrane Jun 30, 2022

Choose a reason for hiding this comment

jsafrane Jun 28, 2022

Choose a reason for hiding this comment

wking Jun 29, 2022

Choose a reason for hiding this comment

openshift-bot commented Feb 4, 2023

openshift-bot commented Mar 6, 2023

openshift-bot commented Apr 6, 2023

openshift-ci bot commented Apr 6, 2023

wking commented Oct 3, 2023

openshift-ci bot commented Oct 3, 2023

openshift-trt-bot commented Dec 1, 2023

wking commented Dec 1, 2023

openshift-trt-bot commented Dec 1, 2023

openshift-trt-bot commented Dec 1, 2023

wking commented Dec 1, 2023

openshift-trt-bot commented Dec 1, 2023

wking commented Dec 1, 2023

openshift-trt-bot commented Dec 1, 2023

openshift-ci-robot commented Dec 1, 2023

openshift-trt-bot commented Dec 2, 2023

wking commented Dec 2, 2023

openshift-trt-bot commented Dec 2, 2023

openshift-trt-bot commented Dec 2, 2023

wking commented Dec 2, 2023

openshift-trt-bot commented Dec 2, 2023

openshift-ci-robot commented Dec 2, 2023

openshift-ci-robot commented Dec 2, 2023

openshift-trt-bot commented Dec 3, 2023

openshift-trt-bot commented Dec 3, 2023

wking commented Dec 3, 2023

openshift-ci-robot commented Dec 3, 2023

openshift-ci bot commented Dec 3, 2023 • edited

openshift-trt-bot commented Dec 3, 2023

openshift-trt-bot commented Dec 3, 2023

wking commented Dec 3, 2023

openshift-bot commented Dec 4, 2023

wking commented Jun 8, 2022 •

edited

openshift-ci bot commented Dec 3, 2023 •

edited