Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TRT-1567: Revert #28589 "AUTH-483: Add a monitor test to check the existence of required-scc annotation in platform workloads" #28659

Conversation

DennisPeriquet
Copy link
Contributor

@DennisPeriquet DennisPeriquet commented Mar 17, 2024

Reverts #28589 ; tracked by TRT-1567

Per OpenShift policy, we are reverting this breaking change to get CI and/or nightly payloads flowing again.

4.16.0-0.ci-2024-03-16-025152 is the first payload where this test failed the payload

To unrevert this, revert this PR, and layer an additional separate commit on top that addresses the problem. Before merging the unrevert, please run these jobs on the PR and check the result of these jobs to confirm the fix has corrected the problem:

/payload-aggregate periodic-ci-openshift-release-master-ci-4.16-upgrade-from-stable-4.15-e2e-aws-ovn-upgrade

CC: @liouk

ps. I noticed on https://prow.ci.openshift.org/view/gs/test-platform-results/logs/aggregated-aws-ovn-upgrade-4.16-minor-release-openshift-release-analysis-aggregator/1768832918135771136 that although the aggregation points to https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-ci-4.16-upgrade-from-stable-4.15-e2e-aws-ovn-upgrade/1768832917330464768 as a failed job, the job passes. So now, I'm wondering if this is an aggregation problem and not a test problem. I see in your original PR that this should flake.

PR created by Revertomatic™️

…nitor-test"

This reverts commit d002496, reversing
changes made to 0de0977.
@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Mar 17, 2024
@openshift-ci-robot
Copy link

openshift-ci-robot commented Mar 17, 2024

@DennisPeriquet: This pull request references TRT-1567 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.16.0" version, but no target version was set.

In response to this:

Reverts #28589 ; tracked by TRT-1567

Per OpenShift policy, we are reverting this breaking change to get CI and/or nightly payloads flowing again.

4.16.0-0.ci-2024-03-16-025152 is the first payload where this test failed the payload

To unrevert this, revert this PR, and layer an additional separate commit on top that addresses the problem. Before merging the unrevert, please run these jobs on the PR and check the result of these jobs to confirm the fix has corrected the problem:

/payload-aggregate periodic-ci-openshift-release-master-ci-4.16-upgrade-from-stable-4.15-e2e-aws-ovn-upgrade

CC: @liouk

PR created by Revertomatic™️

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci openshift-ci bot requested review from deads2k and soltysh March 17, 2024 17:51
@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 17, 2024
@openshift-ci-robot
Copy link

openshift-ci-robot commented Mar 17, 2024

@DennisPeriquet: This pull request references TRT-1567 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.16.0" version, but no target version was set.

In response to this:

Reverts #28589 ; tracked by TRT-1567

Per OpenShift policy, we are reverting this breaking change to get CI and/or nightly payloads flowing again.

4.16.0-0.ci-2024-03-16-025152 is the first payload where this test failed the payload

To unrevert this, revert this PR, and layer an additional separate commit on top that addresses the problem. Before merging the unrevert, please run these jobs on the PR and check the result of these jobs to confirm the fix has corrected the problem:

/payload-aggregate periodic-ci-openshift-release-master-ci-4.16-upgrade-from-stable-4.15-e2e-aws-ovn-upgrade

CC: @liouk

ps. I noticed on https://prow.ci.openshift.org/view/gs/test-platform-results/logs/aggregated-aws-ovn-upgrade-4.16-minor-release-openshift-release-analysis-aggregator/1768832918135771136 that although the aggregation points to https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-ci-4.16-upgrade-from-stable-4.15-e2e-aws-ovn-upgrade/1768832917330464768 as a failed job, the job passes. So now, I'm wondering if this is an aggregation problem and not a test problem. I see in your original PR that this should flake.

PR created by Revertomatic™️

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@DennisPeriquet
Copy link
Contributor Author

/hold

I'm leaning toward this PR not being at fault. But still not sure why the aggregator is calling this test a failure.

@openshift-ci openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 17, 2024
@DennisPeriquet
Copy link
Contributor Author

i see this in the aggregated job summary:

: [sig-auth] all workloads in ns/openshift-must-gather-smq72 must set the 'openshift.io/required-scc' annotation expand_less | 0s
-- | --
{Passed 1 times, failed 0 times, skipped 0 times: we require at least 6 attempts to have a chance at success  name: '[sig-auth] all workloads in ns/openshift-must-gather-smq72 must set the ''openshift.io/required-scc''   annotation' testsuitename: openshift-tests summary: 'Passed 1 times, failed 0 times, skipped 0 times: we require at least 6 attempts   to have a chance at success' passes: - jobrunid: "1768832917330464768"   humanurl: https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-ci-4.16-upgrade-from-stable-4.15-e2e-aws-ovn-upgrade/1768832917330464768   gcsartifacturl: https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/logs/periodic-ci-openshift-release-master-ci-4.16-upgrade-from-stable-4.15-e2e-aws-ovn-upgrade/1768832917330464768/artifacts failures: [] skips: [] }

I think what's happening is that particular test passes only in 1 job but does not appear as passing in any other job; I checked 4 of the jobs and didn't see it in flaking or passing and that's probably what's causing the failure.

So now I wonder why is this test not running for all jobs? and if it is supposed to run for all jobs, then this revert makes sense.

@stbenjam
Copy link
Member

stbenjam commented Mar 17, 2024

/hold cancel
/approve
/lgtm

This needs to be reverted, dynamic names of namespaces should NOT be in test names. So, it's causing payloads to fail and blowing up our table of known test names since each one is unique since it contains a namespace.

The must-gather failures like this:

Failed: suite=[openshift-tests], [sig-auth] all workloads in ns/openshift-must-gather-qng8h must set the 'openshift.io/required-scc' annotation
Passed 1 times, failed 0 times, skipped 0 times: we require at least 6 attempts to have a chance at success

are happening when the test is unlucky enough to be running at the same time the must-gather pods exist.

@openshift-ci openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 17, 2024
@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Mar 17, 2024
Copy link
Contributor

openshift-ci bot commented Mar 17, 2024

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: DennisPeriquet, stbenjam

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:
  • OWNERS [DennisPeriquet,stbenjam]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@stbenjam
Copy link
Member

/label acknowledge-critical-fixes-only
/skip
/override ci/prow/e2e-metal-ipi-sdn
/override ci/prow/e2e-metal-ipi-ovn-ipv6
/override ci/prow/e2e-gcp-ovn-upgrade
/override ci/prow/e2e-agnostic-ovn-cmd
/override ci/prow/e2e-gcp-csi
/override ci/prow/e2e-aws-ovn-single-node
/override ci/prow/e2e-aws-ovn-cgroupsv2
/override ci/prow/e2e-openstack-ovn
/override ci/prow/e2e-gcp-ovn
/override ci/prow/e2e-aws-ovn-fips
/override ci/prow/e2e-aws-csi
/override ci/prow/e2e-gcp-ovn-rt-upgrade
/override ci/prow/e2e-aws-ovn-single-node-serial

@openshift-ci openshift-ci bot added the acknowledge-critical-fixes-only Indicates if the issuer of the label is OK with the policy. label Mar 17, 2024
Copy link
Contributor

openshift-ci bot commented Mar 17, 2024

@stbenjam: Overrode contexts on behalf of stbenjam: ci/prow/e2e-agnostic-ovn-cmd, ci/prow/e2e-aws-csi, ci/prow/e2e-aws-ovn-cgroupsv2, ci/prow/e2e-aws-ovn-fips, ci/prow/e2e-aws-ovn-single-node, ci/prow/e2e-aws-ovn-single-node-serial, ci/prow/e2e-gcp-csi, ci/prow/e2e-gcp-ovn, ci/prow/e2e-gcp-ovn-rt-upgrade, ci/prow/e2e-gcp-ovn-upgrade, ci/prow/e2e-metal-ipi-ovn-ipv6, ci/prow/e2e-metal-ipi-sdn, ci/prow/e2e-openstack-ovn

In response to this:

/label acknowledge-critical-fixes-only
/skip
/override ci/prow/e2e-metal-ipi-sdn
/override ci/prow/e2e-metal-ipi-ovn-ipv6
/override ci/prow/e2e-gcp-ovn-upgrade
/override ci/prow/e2e-agnostic-ovn-cmd
/override ci/prow/e2e-gcp-csi
/override ci/prow/e2e-aws-ovn-single-node
/override ci/prow/e2e-aws-ovn-cgroupsv2
/override ci/prow/e2e-openstack-ovn
/override ci/prow/e2e-gcp-ovn
/override ci/prow/e2e-aws-ovn-fips
/override ci/prow/e2e-aws-csi
/override ci/prow/e2e-gcp-ovn-rt-upgrade
/override ci/prow/e2e-aws-ovn-single-node-serial

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci-robot
Copy link

/retest-required

Remaining retests: 0 against base HEAD d002496 and 2 for PR HEAD 289896b in total

@DennisPeriquet
Copy link
Contributor Author

/override ci/prow/e2e-aws-ovn-serial

Copy link
Contributor

openshift-ci bot commented Mar 18, 2024

@DennisPeriquet: Overrode contexts on behalf of DennisPeriquet: ci/prow/e2e-aws-ovn-serial

In response to this:

/override ci/prow/e2e-aws-ovn-serial

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-merge-bot openshift-merge-bot bot merged commit 2cecd5f into openshift:master Mar 18, 2024
20 of 22 checks passed
Copy link
Contributor

openshift-ci bot commented Mar 18, 2024

@DennisPeriquet: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-aws-ovn-single-node-upgrade 289896b link false /test e2e-aws-ovn-single-node-upgrade

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@liouk liouk mentioned this pull request Mar 18, 2024
@openshift-bot
Copy link
Contributor

[ART PR BUILD NOTIFIER]

This PR has been included in build openshift-enterprise-tests-container-v4.16.0-202403180813.p0.g2cecd5f.assembly.stream.el8 for distgit openshift-enterprise-tests.
All builds following this will include this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
acknowledge-critical-fixes-only Indicates if the issuer of the label is OK with the policy. approved Indicates a PR has been approved by an approver from all required OWNERS files. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants