Reduce WinC presubmit test count by excluding Disruptive and Serial tests#76620
Conversation
openshift-ci
bot
added
the
needs-rebase
rrasouli
force-pushed
the
add-filters-to-presubmit
branch
from
031b710 to
905226b
Compare
openshift-ci
bot
removed
the
needs-rebase
openshift-ci-robot
added
rehearsals-ack
rrasouli
force-pushed
the
add-filters-to-presubmit
branch
from
905226b to
ea4b561
Compare
|
/test debug-winc-aws-ipi debug-winc-gcp-ipi debug-winc-azure-ipi periodic-ci-openshift-openshift-tests-private-release-4.22-amd64-nightly-aws-ipi-proxy-ovn-winc-f28 |
|
/pj-rehearse debug-winc-aws-ipi debug-winc-gcp-ipi debug-winc-azure-ipi periodic-ci-openshift-openshift-tests-private-release-4.22-amd64-nightly-aws-ipi-proxy-ovn-winc-f28 |
|
@rrasouli: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
@rrasouli: job(s): debug-winc-aws-ipi, debug-winc-gcp-ipi, debug-winc-azure-ipi, periodic-ci-openshift-openshift-tests-private-release-4.22-amd64-nightly-aws-ipi-proxy-ovn-winc-f28 either don't exist or were not found to be affected, and cannot be rehearsed |
|
/pj-rehearse debug-winc-azure-ipi |
|
@rrasouli: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
@rrasouli: job(s): debug-winc-azure-ipi either don't exist or were not found to be affected, and cannot be rehearsed |
|
/pj-rehearse |
|
@rrasouli: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
rrasouli
force-pushed
the
add-filters-to-presubmit
branch
2 times, most recently
from
a9276ee to
431c6ab
Compare
rrasouli
changed the title
|
@weinliu @jrvaldes @mansikulkarni96 please have a look |
rrasouli
force-pushed
the
add-filters-to-presubmit
branch
3 times, most recently
from
b62116d to
fe8c96f
Compare
|
/lgtm |
rrasouli
force-pushed
the
add-filters-to-presubmit
branch
from
fe8c96f to
2239c46
Compare
|
@jianlinliu @weinliu please approve |
Do we need to change them in advance? we don't have WINC 4.23 image yet |
| cluster_profile: nutanix-qe | ||
| env: | ||
| EXTRACT_MANIFEST_INCLUDED: "true" | ||
| OVERRIDE_RHCOS_IMAGE: https://rhcos.mirror.openshift.com/art/storage/prod/streams/rhel-9.6/builds/9.6.20260117-0/x86_64/rhcos-9.6.20260117-0-nutanix.x86_64.qcow2 |
There was a problem hiding this comment.
from https://rhcos.mirror.openshift.com/art/storage/prod/streams/rhel-9.6/builds/builds.json, I see there are more qcow2 versions, why use rhcos-9.6.20260117-0-nutanix.x86_64.qcow2, is it on purpose?
There was a problem hiding this comment.
what version do the other jobs use - should I use latest image?
There was a problem hiding this comment.
TBH, I don't know the answer, just found there are more qcow2 versions. maybe @weinliu knows?
There was a problem hiding this comment.
checked on 4.22.0-0.nightly-2026-03-24-224450 without Windows node, node os is rhel-9.8
$ oc debug node/**
sh-5.1# chroot /host
sh-5.1# cat /etc/os-release
NAME="Red Hat Enterprise Linux CoreOS"
VERSION="9.8.20260324-0 (Plow)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="9.8"
PLATFORM_ID="platform:el9"
PRETTY_NAME="Red Hat Enterprise Linux CoreOS 9.8.20260324-0 (Plow)"
ANSI_COLOR="0;31"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:redhat:enterprise_linux:9::baseos"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9"
BUG_REPORT_URL="https://issues.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 9"
REDHAT_BUGZILLA_PRODUCT_VERSION=9.8
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.8 Beta"
OSTREE_VERSION='9.8.20260324-0'
IMAGE_VERSION='9.8.20260324-0'
VARIANT=CoreOS
VARIANT_ID=coreos
OPENSHIFT_VERSION="4.22"
sh-5.1# cat /etc/redhat-release
Red Hat Enterprise Linux release 9.8 Beta (Plow)
checked on 4.21.0-0.nightly-2026-03-22-203205 without Windows node, node os is rhel-9.6
$ oc debug node/**
sh-5.1# chroot /host
sh-5.1# cat /etc/os-release
NAME="Red Hat Enterprise Linux CoreOS"
VERSION="9.6.20260319-1 (Plow)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="9.6"
PLATFORM_ID="platform:el9"
PRETTY_NAME="Red Hat Enterprise Linux CoreOS 9.6.20260319-1 (Plow)"
ANSI_COLOR="0;31"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:redhat:enterprise_linux:9::baseos"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9"
BUG_REPORT_URL="https://issues.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 9"
REDHAT_BUGZILLA_PRODUCT_VERSION=9.6
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.6"
OSTREE_VERSION='9.6.20260319-1'
VARIANT=CoreOS
VARIANT_ID=coreos
OPENSHIFT_VERSION="4.21"
checked on 4.20.0-0.nightly-2026-03-23-130518 without Windows node, node os is rhel-9.6
$ oc debug node/**
sh-5.1# chroot /host
sh-5.1# cat /etc/os-release
NAME="Red Hat Enterprise Linux CoreOS"
VERSION="9.6.20260319-1 (Plow)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="9.6"
PLATFORM_ID="platform:el9"
PRETTY_NAME="Red Hat Enterprise Linux CoreOS 9.6.20260319-1 (Plow)"
ANSI_COLOR="0;31"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:redhat:enterprise_linux:9::baseos"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9"
BUG_REPORT_URL="https://issues.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 9"
REDHAT_BUGZILLA_PRODUCT_VERSION=9.6
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.6"
OSTREE_VERSION='9.6.20260319-1'
VARIANT=CoreOS
VARIANT_ID=coreos
OPENSHIFT_VERSION="4.20"
seems no need to change, the main branch is still 4.22 now, we can ignore 4.23/5.0 |
rrasouli
force-pushed
the
add-filters-to-presubmit
branch
from
2239c46 to
78dd4bb
Compare
openshift-ci-robot
removed
the
rehearsals-ack
@juzhao Anyway I've update 4.23 jobs with the new filter that will save us another PR |
rrasouli
force-pushed
the
add-filters-to-presubmit
branch
from
78dd4bb to
036e6f6
Compare
|
|
/pj-rehearse ack |
|
@rrasouli: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
openshift-ci-robot
added
the
rehearsals-ack
| cluster_profile: nutanix-qe | ||
| env: | ||
| EXTRACT_MANIFEST_INCLUDED: "true" | ||
| OVERRIDE_RHCOS_IMAGE: https://rhcos.mirror.openshift.com/art/storage/prod/streams/rhel-9.6/builds/9.6.20260323-1/x86_64/rhcos-9.6.20260323-1-nutanix.x86_64.qcow2 |
There was a problem hiding this comment.
see from #76620 (comment), 4.22 node os is rhel-9.8, is there someone who can confirm use the rhel-9.6 is fine for 4.22(main branch now)
There was a problem hiding this comment.
Good catch!
Removed OVERRIDE_RHCOS_IMAGE from the newly added nutanix-ipi jobs (main, release-4.22, release-4.23).
- Nightly nutanix jobs don't use
OVERRIDE_RHCOS_IMAGE- I checked the nightly configs (e.g.,nutanix-ipi-ovn-winc-f7inrelease-4.22__amd64-nightly.yaml) and they work fine without this override. - New jobs should match nightly pattern - Since the newly added presubmit jobs (main, 4.22, 4.23) didn't exist before, I aligned them with the nightly configs by removing the RHCOS override.
…ests This change updates all WinC debug-* presubmit jobs to exclude Disruptive and Serial tests, reducing the test count from ~40 to ~23 tests. Changes: - Updated TEST_FILTERS to add ~Disruptive&;~Serial& exclusions on 37 existing jobs - Removed ineffective FILTERS_ADDITIONAL: ~CPaasrunOnly&;Smokerun& (no WinC tests have CPaasrunOnly tag) - Added debug-winc-nutanix-ipi to main, release-4.22, and release-4.23 (6 new jobs) - Added debug-winc-aws-upi to main, release-4.22, and release-4.23 (6 new jobs) - Applied across all branches: main, release-4.17 through release-4.23 Test breakdown after this change: - ~23 tests: All Smokerun tests excluding Disruptive/Serial - Priority distribution: 10 Critical, 1 High, 11 Medium, 1 Low - Safer for presubmit: no cluster disruption, no serial execution delays Total: 49 presubmit jobs modified/added across 8 branches Nightly jobs remain unchanged and continue to run full Smokerun suite (~40 tests).
rrasouli
force-pushed
the
add-filters-to-presubmit
branch
from
036e6f6 to
c6a84c9
Compare
openshift-ci-robot
removed
the
rehearsals-ack
|
[REHEARSALNOTIFIER]
A total of 43 jobs have been affected by this change. The above listing is non-exhaustive and limited to 25 jobs. A full list of affected jobs can be found here Interacting with pj-rehearseComment: Once you are satisfied with the results of the rehearsals, comment: |
|
@rrasouli: all tests passed! Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
...config/openshift/openshift-tests-private/openshift-openshift-tests-private-release-4.21.yaml
Show resolved
Hide resolved
|
/approve |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: juzhao, rrasouli, weinliu The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
bot
added
the
approved
|
/pj-rehearse ack |
|
@rrasouli: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
openshift-ci-robot
added
the
rehearsals-ack
PR openshift#76620 added debug-winc-nutanix-ipi jobs to main, release-4.22, and release-4.23 but omitted the critical OVERRIDE_RHCOS_IMAGE environment variable that was already present in release-4.21 (added by PR openshift#73946). Without this override, Nutanix IPI installations fail during bootstrap because of RHCOS version mismatches between the installer's embedded metadata and the release payload's expected machine-os version. This manifests as timeout errors (exit 28) in the ipi-conf-nutanix-context step when trying to communicate with the Nutanix Prism Central API. This fix adds OVERRIDE_RHCOS_IMAGE pointing to RHCOS 9.6.20260323-1 (same version used in working 4.21 jobs) to prevent API server startup failures during cluster bootstrap. Fixes: openshift#76620
4 participants
Problem
WinC presubmit jobs (debug-winc-*) run all ~40 Smokerun tests, making PR feedback slow and resource-intensive.
PR #71726 attempted to reduce this by adding
FILTERS_ADDITIONAL: ~CPaasrunOnly&;Smokerun&, but this filter has no effect because no WinC tests have the CPaasrunOnly tag.Root Cause
Analysis of the test suite shows:
~CPaasrunOnly&;Smokerun&= exclude CPaasrunOnly from Smokerun = no exclusionsSolution
Before:
After:
Test Count Impact
Current state (~40 tests):
With this PR (~23 tests):
Nightly jobs: Unchanged - continue to run full ~40 test Smokerun suite
Coverage Summary
Total: 49 presubmit jobs across 8 branches
✨ = newly added in this PR
Files Modified
All WinC presubmit job configurations:
Changes:
Jira: https://redhat.atlassian.net/browse/WINC-1507