Skip to content

METAL-1745: Add 5.0 jobs for rhcos10 worker nodes#79338

Open
sgoveas wants to merge 1 commit into
openshift:mainfrom
sgoveas:rhcos10
Open

METAL-1745: Add 5.0 jobs for rhcos10 worker nodes#79338
sgoveas wants to merge 1 commit into
openshift:mainfrom
sgoveas:rhcos10

Conversation

@sgoveas
Copy link
Copy Markdown
Contributor

@sgoveas sgoveas commented May 15, 2026

Overview

This PR configures CI infrastructure to support testing OpenShift with a RHCOS 10 (rhel-10) worker OS stream variant. It extends the baremetal IPI test pipeline to allow worker nodes to use an alternative CoreOS stream during cluster installation and testing.

CI Configuration Changes

New test jobs added for RHCOS 10 worker testing:

  • Added multi-nightly baremetal IPI test variants for release-4.23 and 5.0 that include rhcos10-worker in their names, supporting both standard and virtual-media deployments
  • These jobs are configured with WORKER_COREOS_STREAM: rhel-10 to target the RHCOS 10 stream

Baremetal IPI installation flow:

  • Extended the baremetal-lab-ipi-install step to support an optional WORKER_COREOS_STREAM variable
  • When this variable is set, the installation script patches generated manifests to override the worker node OS stream:
    • Worker BareMetalHost manifests receive the custom CoreOS stream configuration
    • Worker MachineSet manifests are updated to reference the alternate stream
    • A new MachineConfigPool is generated for workers to bind them to the custom OS image stream

Test orchestration adjustments:

  • Disabled the baremetal-lab-agent-gather step in the post-installation chain
  • Updated the baremetal-lab-wait step to use the baremetal-qe-base container image instead of the CLI image
  • Increased the cluster wait timeout from 9000 to 259200 seconds (3 days)
  • Simplified the QE test chain to only run cluster health checks

Image registry configuration:

  • Updated to use a mirrored registry (registry.build10.ci.openshift.org) for release images during installation

Impact

The infrastructure now supports testing OpenShift cluster deployments where worker nodes run RHCOS 10 instead of the standard stream, enabling validation of compatibility and stability with the newer OS variant in baremetal environments.

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented May 15, 2026

Walkthrough

This pull request introduces CoreOS image stream configuration for baremetal worker nodes and updates test infrastructure chains. It adds a WORKER_COREOS_STREAM environment variable that patches worker manifests with stream annotations, exports release image overrides, defines two new RHCOS 10 test jobs, disables agent gathering in the post-install chain, and reconfigures the wait step with an explicit image reference and extended timeout.

Changes

RHCOS 10 worker stream configuration

Layer / File(s) Summary
WORKER_COREOS_STREAM environment integration
ci-operator/step-registry/baremetal/lab/ipi/install/baremetal-lab-ipi-install-ref.yaml, ci-operator/step-registry/baremetal/lab/ipi/install/baremetal-lab-ipi-install-commands.sh, ci-operator/config/openshift/openshift-tests-private/openshift-openshift-tests-private-release-5.0__multi-nightly.yaml
WORKER_COREOS_STREAM environment variable is declared with empty default, exported release image override references are set to mirrored registry, conditional script block patches worker BareMetalHost and MachineSet manifests with stream annotations and generates MachineConfigPool, and two new Metal IPI test jobs are configured with WORKER_COREOS_STREAM: rhel-10.
Test chain and wait step reconfiguration
ci-operator/step-registry/baremetal/lab/post/baremetal-lab-post-chain.yaml, ci-operator/step-registry/baremetal/lab/wait/baremetal-lab-wait-ref.yaml, ci-operator/step-registry/openshift/e2e/test/qe/openshift-e2e-test-qe-chain.yaml
Post-install chain disables agent-gather step and activates wait step, wait step replaces cli image source with explicit baremetal-qe-base:latest reference and extends cluster duration to 259200 seconds, and QE test chain comments out multiple test refs leaving only health check active.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~12 minutes

Suggested labels

rehearsals-ack

🚥 Pre-merge checks | ✅ 12
✅ Passed checks (12 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed Check not applicable. This PR modifies CI configuration files (YAML) and bash scripts only, not Ginkgo test code. The test name check requires actual test code files.
Test Structure And Quality ✅ Passed The custom check for Ginkgo test structure and quality is not applicable to this PR. The PR contains only shell scripts and YAML CI/CD configuration files—no Go test code or Ginkgo tests are present.
Microshift Test Compatibility ✅ Passed PR modifies CI configuration in openshift/release, not test code. Check applies to new Ginkgo tests in openshift/origin. No new Go test files.
Single Node Openshift (Sno) Test Compatibility ✅ Passed PR modifies only CI configuration and orchestration files (YAML and shell scripts) in the openshift/release repository. No new Ginkgo e2e test code is added. The custom check is not applicable.
Topology-Aware Scheduling Compatibility ✅ Passed PR modifies CI infrastructure configuration, not pod scheduling constraints. The MachineConfigPool nodeSelector is topology-compatible.
Ote Binary Stdout Contract ✅ Passed OTE Binary Stdout Contract check applies to Go test binaries. This PR modifies only CI configuration (YAML) and bash scripts in openshift/release. No Go test code or OTE binaries present.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed PR contains only CI configuration changes. No new Ginkgo e2e tests are being added, making this check not applicable.
Title check ✅ Passed The title 'METAL-1745: Add 5.0 jobs for rhcos10 worker nodes' directly describes the main change—adding CI test jobs for RHEL CoreOS 10 worker nodes in release 5.0, matching the new jobs added to the release configuration.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@openshift-ci openshift-ci Bot requested review from asood-rh and oliver-smakal May 15, 2026 10:12
@sgoveas sgoveas marked this pull request as draft May 15, 2026 10:13
@openshift-ci openshift-ci Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 15, 2026
@sgoveas
Copy link
Copy Markdown
Contributor Author

sgoveas commented May 15, 2026

/pj-rehearse periodic-ci-openshift-openshift-tests-private-release-4.23-multi-nightly-metal-ipi-ovn-ipv4-rhcos10-worker-tp-amd-f7 periodic-ci-openshift-openshift-tests-private-release-4.23-multi-nightly-metal-ipi-ovn-ipv4-vmedia-rhcos10-worker-tp-arm-f7

@openshift-merge-bot
Copy link
Copy Markdown
Contributor

@sgoveas: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 4

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In
`@ci-operator/step-registry/baremetal/lab/ipi/install/baremetal-lab-ipi-install-commands.sh`:
- Around line 261-290: The code uses WORKER_COREOS_STREAM directly in sed
replacements and an unquoted heredoc which can break manifests if the value
contains unexpected characters; add validation that WORKER_COREOS_STREAM matches
a safe pattern (e.g. ^[A-Za-z0-9._-]+$) and fail early if it doesn't, then use
the validated value (e.g. WORKER_COREOS_STREAM_SANITIZED) in the sed commands
and the heredoc for 99_worker-osimagestream.yaml; also switch sed to a delimiter
that reduces escaping issues (for example use |) and ensure all replacements
reference the sanitized variable (occurrences in the for loops and the heredoc
that create MachineConfigPool).

In `@ci-operator/step-registry/baremetal/lab/post/baremetal-lab-post-chain.yaml`:
- Around line 4-7: The shared post chain currently includes the step ref
"baremetal-lab-wait" which can block teardown for up to 72h; remove
"baremetal-lab-wait" from the common post chain in baremetal-lab-post-chain.yaml
so the shared post-cleanup no longer references that ref, and instead add a
separate dedicated post chain or workflow (e.g.,
"baremetal-lab-post-manual-wait") that includes "baremetal-lab-wait" for
manual/debug runs or make its inclusion conditional so normal teardown is never
delayed.

In
`@ci-operator/step-registry/openshift/e2e/test/qe/openshift-e2e-test-qe-chain.yaml`:
- Line 16: The chain description "Execute e2e tests from QE, which include
golang (openshift-extended-test), cucushift (cucushift-e2e), cypress
(openshift-extended-web-tests), ...(more to add)" is outdated; update the
docstring in openshift-e2e-test-qe-chain.yaml (the top description/comment
containing that text) to accurately state the current execution scope (e.g.,
"Runs installer health checks only") and optionally note that
extended/cucushift/web suites are not executed by this chain so future
maintainers aren’t misled.
- Around line 5-14: The QE refs for this chain have been commented out, which
disables QE coverage; restore the original list by uncommenting or re-adding the
refs so the chain runs QE tests: ensure the entries idp-htpasswd,
fips-check-fips-or-die, fips-check-node-scan, cucushift-pre,
openshift-extended-test, cucushift-e2e, openshift-extended-web-tests,
openshift-extended-test-supplementary, openshift-e2e-test-clusterinfra-qe, and
openshift-e2e-test-qe-report appear (in the same logical order) in the YAML
instead of being commented so the openshift-e2e-test-qe coverage is executed.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 9dec86da-0be9-46d2-a504-0a17bcdc526f

📥 Commits

Reviewing files that changed from the base of the PR and between da6e46c and 95d2da8.

⛔ Files ignored due to path filters (1)
  • ci-operator/jobs/openshift/openshift-tests-private/openshift-openshift-tests-private-release-4.23-periodics.yaml is excluded by !ci-operator/jobs/**
📒 Files selected for processing (6)
  • ci-operator/config/openshift/openshift-tests-private/openshift-openshift-tests-private-release-4.23__multi-nightly.yaml
  • ci-operator/step-registry/baremetal/lab/ipi/install/baremetal-lab-ipi-install-commands.sh
  • ci-operator/step-registry/baremetal/lab/ipi/install/baremetal-lab-ipi-install-ref.yaml
  • ci-operator/step-registry/baremetal/lab/post/baremetal-lab-post-chain.yaml
  • ci-operator/step-registry/baremetal/lab/wait/baremetal-lab-wait-ref.yaml
  • ci-operator/step-registry/openshift/e2e/test/qe/openshift-e2e-test-qe-chain.yaml

Comment on lines +261 to +290
if [[ -n "${WORKER_COREOS_STREAM:-}" ]]; then
for bmh_file in "${INSTALL_DIR}"/openshift/99_openshift-cluster-api_hosts-*.yaml; do
if ! grep -q 'installer.openshift.io/role: control-plane' "${bmh_file}"; then
sed -i "s/coreos.openshift.io\/stream: .*/coreos.openshift.io\/stream: ${WORKER_COREOS_STREAM}/" "${bmh_file}"
fi
done
# Patch worker MachineSet hostSelector to match the new stream
for ms_file in "${INSTALL_DIR}"/openshift/99_openshift-cluster-api_worker-machineset-*.yaml; do
sed -i "s/coreos.openshift.io\/stream: .*/coreos.openshift.io\/stream: ${WORKER_COREOS_STREAM}/" "${ms_file}"
done
# Set the worker MachineConfigPool to use the specified stream for
# the on-disk OS. Requires the OSStreams feature gate (TechPreviewNoUpgrade).
cat > "${INSTALL_DIR}/openshift/99_worker-osimagestream.yaml" <<EOF
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfigPool
metadata:
labels:
machineconfiguration.openshift.io/mco-built-in: ""
pools.operator.machineconfiguration.openshift.io/worker: ""
name: worker
spec:
machineConfigSelector:
matchLabels:
machineconfiguration.openshift.io/role: worker
nodeSelector:
matchLabels:
node-role.kubernetes.io/worker: ""
osImageStream:
name: ${WORKER_COREOS_STREAM}
EOF
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Validate and escape WORKER_COREOS_STREAM before using it in sed/YAML.

WORKER_COREOS_STREAM is used unescaped in replacements and heredoc output; unexpected characters can break manifest generation or patching.

Suggested hardening
 if [[ -n "${WORKER_COREOS_STREAM:-}" ]]; then
+    if [[ ! "${WORKER_COREOS_STREAM}" =~ ^[a-zA-Z0-9._-]+$ ]]; then
+        echo "[ERROR] Invalid WORKER_COREOS_STREAM: ${WORKER_COREOS_STREAM}"
+        exit 1
+    fi
+    esc_stream="$(printf '%s' "${WORKER_COREOS_STREAM}" | sed 's/[\/&]/\\&/g')"
     for bmh_file in "${INSTALL_DIR}"/openshift/99_openshift-cluster-api_hosts-*.yaml; do
         if ! grep -q 'installer.openshift.io/role: control-plane' "${bmh_file}"; then
-            sed -i "s/coreos.openshift.io\/stream: .*/coreos.openshift.io\/stream: ${WORKER_COREOS_STREAM}/" "${bmh_file}"
+            sed -i "s/coreos.openshift.io\/stream: .*/coreos.openshift.io\/stream: ${esc_stream}/" "${bmh_file}"
         fi
     done
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
if [[ -n "${WORKER_COREOS_STREAM:-}" ]]; then
for bmh_file in "${INSTALL_DIR}"/openshift/99_openshift-cluster-api_hosts-*.yaml; do
if ! grep -q 'installer.openshift.io/role: control-plane' "${bmh_file}"; then
sed -i "s/coreos.openshift.io\/stream: .*/coreos.openshift.io\/stream: ${WORKER_COREOS_STREAM}/" "${bmh_file}"
fi
done
# Patch worker MachineSet hostSelector to match the new stream
for ms_file in "${INSTALL_DIR}"/openshift/99_openshift-cluster-api_worker-machineset-*.yaml; do
sed -i "s/coreos.openshift.io\/stream: .*/coreos.openshift.io\/stream: ${WORKER_COREOS_STREAM}/" "${ms_file}"
done
# Set the worker MachineConfigPool to use the specified stream for
# the on-disk OS. Requires the OSStreams feature gate (TechPreviewNoUpgrade).
cat > "${INSTALL_DIR}/openshift/99_worker-osimagestream.yaml" <<EOF
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfigPool
metadata:
labels:
machineconfiguration.openshift.io/mco-built-in: ""
pools.operator.machineconfiguration.openshift.io/worker: ""
name: worker
spec:
machineConfigSelector:
matchLabels:
machineconfiguration.openshift.io/role: worker
nodeSelector:
matchLabels:
node-role.kubernetes.io/worker: ""
osImageStream:
name: ${WORKER_COREOS_STREAM}
EOF
if [[ -n "${WORKER_COREOS_STREAM:-}" ]]; then
if [[ ! "${WORKER_COREOS_STREAM}" =~ ^[a-zA-Z0-9._-]+$ ]]; then
echo "[ERROR] Invalid WORKER_COREOS_STREAM: ${WORKER_COREOS_STREAM}"
exit 1
fi
esc_stream="$(printf '%s' "${WORKER_COREOS_STREAM}" | sed 's/[\/&]/\\&/g')"
for bmh_file in "${INSTALL_DIR}"/openshift/99_openshift-cluster-api_hosts-*.yaml; do
if ! grep -q 'installer.openshift.io/role: control-plane' "${bmh_file}"; then
sed -i "s/coreos.openshift.io\/stream: .*/coreos.openshift.io\/stream: ${esc_stream}/" "${bmh_file}"
fi
done
# Patch worker MachineSet hostSelector to match the new stream
for ms_file in "${INSTALL_DIR}"/openshift/99_openshift-cluster-api_worker-machineset-*.yaml; do
sed -i "s/coreos.openshift.io\/stream: .*/coreos.openshift.io\/stream: ${esc_stream}/" "${ms_file}"
done
# Set the worker MachineConfigPool to use the specified stream for
# the on-disk OS. Requires the OSStreams feature gate (TechPreviewNoUpgrade).
cat > "${INSTALL_DIR}/openshift/99_worker-osimagestream.yaml" <<EOF
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfigPool
metadata:
labels:
machineconfiguration.openshift.io/mco-built-in: ""
pools.operator.machineconfiguration.openshift.io/worker: ""
name: worker
spec:
machineConfigSelector:
matchLabels:
machineconfiguration.openshift.io/role: worker
nodeSelector:
matchLabels:
node-role.kubernetes.io/worker: ""
osImageStream:
name: ${WORKER_COREOS_STREAM}
EOF
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@ci-operator/step-registry/baremetal/lab/ipi/install/baremetal-lab-ipi-install-commands.sh`
around lines 261 - 290, The code uses WORKER_COREOS_STREAM directly in sed
replacements and an unquoted heredoc which can break manifests if the value
contains unexpected characters; add validation that WORKER_COREOS_STREAM matches
a safe pattern (e.g. ^[A-Za-z0-9._-]+$) and fail early if it doesn't, then use
the validated value (e.g. WORKER_COREOS_STREAM_SANITIZED) in the sed commands
and the heredoc for 99_worker-osimagestream.yaml; also switch sed to a delimiter
that reduces escaping issues (for example use |) and ensure all replacements
reference the sanitized variable (occurrences in the for loops and the heredoc
that create MachineConfigPool).

Comment on lines 4 to 7
# - ref: baremetal-lab-agent-gather
# - chain: gather
- ref: baremetal-lab-wait
- ref: baremetal-lab-post-dhcp-pxe-conf
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Do not block shared post-cleanup with baremetal-lab-wait.

Injecting baremetal-lab-wait into the common post chain can delay teardown by up to 72h (current default), which risks lab capacity exhaustion and delayed cleanup.

Safer direction
-   # - ref: baremetal-lab-agent-gather
-   # - chain: gather
-    - ref: baremetal-lab-wait
+    - ref: baremetal-lab-agent-gather
+    - chain: gather

If the wait is only for manual/debug workflows, keep it in a dedicated post chain/workflow rather than the shared one.

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
# - ref: baremetal-lab-agent-gather
# - chain: gather
- ref: baremetal-lab-wait
- ref: baremetal-lab-post-dhcp-pxe-conf
- ref: baremetal-lab-agent-gather
- chain: gather
- ref: baremetal-lab-post-dhcp-pxe-conf
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@ci-operator/step-registry/baremetal/lab/post/baremetal-lab-post-chain.yaml`
around lines 4 - 7, The shared post chain currently includes the step ref
"baremetal-lab-wait" which can block teardown for up to 72h; remove
"baremetal-lab-wait" from the common post chain in baremetal-lab-post-chain.yaml
so the shared post-cleanup no longer references that ref, and instead add a
separate dedicated post chain or workflow (e.g.,
"baremetal-lab-post-manual-wait") that includes "baremetal-lab-wait" for
manual/debug runs or make its inclusion conditional so normal teardown is never
delayed.

Comment on lines +5 to +14
# - ref: idp-htpasswd
# - ref: fips-check-fips-or-die
# - ref: fips-check-node-scan
# - ref: cucushift-pre
# - ref: openshift-extended-test
# - ref: cucushift-e2e
# - ref: openshift-extended-web-tests
# - ref: openshift-extended-test-supplementary
# - ref: openshift-e2e-test-clusterinfra-qe
# - ref: openshift-e2e-test-qe-report
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

This effectively disables QE coverage for this chain.

With these refs commented, the chain only performs cluster-health checks, so openshift-e2e-test-qe no longer runs QE tests and can produce false-green signal.

Suggested fix
-#  - ref: idp-htpasswd
-#  - ref: fips-check-fips-or-die
-#  - ref: fips-check-node-scan
-#  - ref: cucushift-pre
-#  - ref: openshift-extended-test
-#  - ref: cucushift-e2e
-#  - ref: openshift-extended-web-tests
-#  - ref: openshift-extended-test-supplementary
-#  - ref: openshift-e2e-test-clusterinfra-qe
-#  - ref: openshift-e2e-test-qe-report
+  - ref: idp-htpasswd
+  - ref: fips-check-fips-or-die
+  - ref: fips-check-node-scan
+  - ref: cucushift-pre
+  - ref: openshift-extended-test
+  - ref: cucushift-e2e
+  - ref: openshift-extended-web-tests
+  - ref: openshift-extended-test-supplementary
+  - ref: openshift-e2e-test-clusterinfra-qe
+  - ref: openshift-e2e-test-qe-report
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
# - ref: idp-htpasswd
# - ref: fips-check-fips-or-die
# - ref: fips-check-node-scan
# - ref: cucushift-pre
# - ref: openshift-extended-test
# - ref: cucushift-e2e
# - ref: openshift-extended-web-tests
# - ref: openshift-extended-test-supplementary
# - ref: openshift-e2e-test-clusterinfra-qe
# - ref: openshift-e2e-test-qe-report
- ref: idp-htpasswd
- ref: fips-check-fips-or-die
- ref: fips-check-node-scan
- ref: cucushift-pre
- ref: openshift-extended-test
- ref: cucushift-e2e
- ref: openshift-extended-web-tests
- ref: openshift-extended-test-supplementary
- ref: openshift-e2e-test-clusterinfra-qe
- ref: openshift-e2e-test-qe-report
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@ci-operator/step-registry/openshift/e2e/test/qe/openshift-e2e-test-qe-chain.yaml`
around lines 5 - 14, The QE refs for this chain have been commented out, which
disables QE coverage; restore the original list by uncommenting or re-adding the
refs so the chain runs QE tests: ensure the entries idp-htpasswd,
fips-check-fips-or-die, fips-check-node-scan, cucushift-pre,
openshift-extended-test, cucushift-e2e, openshift-extended-web-tests,
openshift-extended-test-supplementary, openshift-e2e-test-clusterinfra-qe, and
openshift-e2e-test-qe-report appear (in the same logical order) in the YAML
instead of being commented so the openshift-e2e-test-qe coverage is executed.

# - ref: openshift-e2e-test-clusterinfra-qe
# - ref: openshift-e2e-test-qe-report
documentation: |-
Execute e2e tests from QE, which include golang (openshift-extended-test), cucushift (cucushift-e2e), cypress (openshift-extended-web-tests), ...(more to add)
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Update documentation to match the new execution scope.

The docstring still says this chain executes extended/cucushift/web QE suites, but current steps run only installer health checks.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@ci-operator/step-registry/openshift/e2e/test/qe/openshift-e2e-test-qe-chain.yaml`
at line 16, The chain description "Execute e2e tests from QE, which include
golang (openshift-extended-test), cucushift (cucushift-e2e), cypress
(openshift-extended-web-tests), ...(more to add)" is outdated; update the
docstring in openshift-e2e-test-qe-chain.yaml (the top description/comment
containing that text) to accurately state the current execution scope (e.g.,
"Runs installer health checks only") and optionally note that
extended/cucushift/web suites are not executed by this chain so future
maintainers aren’t misled.

@sgoveas
Copy link
Copy Markdown
Contributor Author

sgoveas commented May 15, 2026

/pj-rehearse periodic-ci-openshift-openshift-tests-private-release-4.23-multi-nightly-metal-ipi-ovn-ipv4-rhcos10-worker-tp-amd-f7 periodic-ci-openshift-openshift-tests-private-release-4.23-multi-nightly-metal-ipi-ovn-ipv4-vmedia-rhcos10-worker-tp-arm-f7

@openshift-merge-bot
Copy link
Copy Markdown
Contributor

@sgoveas: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In
`@ci-operator/step-registry/baremetal/lab/ipi/install/baremetal-lab-ipi-install-commands.sh`:
- Around line 95-96: The script currently hard-codes CI release image
coordinates into OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE and
MULTI_RELEASE_IMAGE; change it to use the ci-operator provided release image
environment variable(s) instead: if OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE or
MULTI_RELEASE_IMAGE are not already set, assign them from the ci-operator
release variable (e.g., RELEASE_IMAGE) by using something like
OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE="${OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE:-${RELEASE_IMAGE}}"
and MULTI_RELEASE_IMAGE="${MULTI_RELEASE_IMAGE:-${RELEASE_IMAGE}}", so you don’t
force a specific ci-ln-* image and you respect any externally-provided values.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 4c73af61-2ede-4da8-affa-7b54b06930a4

📥 Commits

Reviewing files that changed from the base of the PR and between 95d2da8 and b7b7921.

⛔ Files ignored due to path filters (1)
  • ci-operator/jobs/openshift/openshift-tests-private/openshift-openshift-tests-private-release-4.23-periodics.yaml is excluded by !ci-operator/jobs/**
📒 Files selected for processing (6)
  • ci-operator/config/openshift/openshift-tests-private/openshift-openshift-tests-private-release-4.23__multi-nightly.yaml
  • ci-operator/step-registry/baremetal/lab/ipi/install/baremetal-lab-ipi-install-commands.sh
  • ci-operator/step-registry/baremetal/lab/ipi/install/baremetal-lab-ipi-install-ref.yaml
  • ci-operator/step-registry/baremetal/lab/post/baremetal-lab-post-chain.yaml
  • ci-operator/step-registry/baremetal/lab/wait/baremetal-lab-wait-ref.yaml
  • ci-operator/step-registry/openshift/e2e/test/qe/openshift-e2e-test-qe-chain.yaml
🚧 Files skipped from review as they are similar to previous changes (5)
  • ci-operator/step-registry/baremetal/lab/post/baremetal-lab-post-chain.yaml
  • ci-operator/step-registry/openshift/e2e/test/qe/openshift-e2e-test-qe-chain.yaml
  • ci-operator/step-registry/baremetal/lab/wait/baremetal-lab-wait-ref.yaml
  • ci-operator/step-registry/baremetal/lab/ipi/install/baremetal-lab-ipi-install-ref.yaml
  • ci-operator/config/openshift/openshift-tests-private/openshift-openshift-tests-private-release-4.23__multi-nightly.yaml

Comment on lines +95 to +96
export OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE="registry.build10.ci.openshift.org/ci-ln-lgvqh3t/release:latest"
export MULTI_RELEASE_IMAGE="registry.build10.ci.openshift.org/ci-ln-lgvqh3t/release:latest"
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Avoid hard-coding CI release image coordinates in this shared install step.

Line 95 and Line 96 pin to a specific ci-ln-* namespace/image, which is brittle for shared step-registry usage and can break installs when that namespace/image is absent. Prefer ci-operator-provided release image env vars instead of a fixed value.

Suggested fix
-export OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE="registry.build10.ci.openshift.org/ci-ln-lgvqh3t/release:latest"
-export MULTI_RELEASE_IMAGE="registry.build10.ci.openshift.org/ci-ln-lgvqh3t/release:latest"
+: "${OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE:=${RELEASE_IMAGE_INITIAL}}"
+: "${MULTI_RELEASE_IMAGE:=${RELEASE_IMAGE_LATEST}}"
+export OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE
+export MULTI_RELEASE_IMAGE
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@ci-operator/step-registry/baremetal/lab/ipi/install/baremetal-lab-ipi-install-commands.sh`
around lines 95 - 96, The script currently hard-codes CI release image
coordinates into OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE and
MULTI_RELEASE_IMAGE; change it to use the ci-operator provided release image
environment variable(s) instead: if OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE or
MULTI_RELEASE_IMAGE are not already set, assign them from the ci-operator
release variable (e.g., RELEASE_IMAGE) by using something like
OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE="${OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE:-${RELEASE_IMAGE}}"
and MULTI_RELEASE_IMAGE="${MULTI_RELEASE_IMAGE:-${RELEASE_IMAGE}}", so you don’t
force a specific ci-ln-* image and you respect any externally-provided values.

@sgoveas
Copy link
Copy Markdown
Contributor Author

sgoveas commented May 18, 2026

/pj-rehearse periodic-ci-openshift-openshift-tests-private-release-4.23-multi-nightly-metal-ipi-ovn-ipv4-rhcos10-worker-tp-amd-f7 periodic-ci-openshift-openshift-tests-private-release-4.23-multi-nightly-metal-ipi-ovn-ipv4-vmedia-rhcos10-worker-tp-amd-f7

@openshift-merge-bot
Copy link
Copy Markdown
Contributor

@sgoveas: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@sgoveas
Copy link
Copy Markdown
Contributor Author

sgoveas commented May 18, 2026

/pj-rehearse periodic-ci-openshift-openshift-tests-private-release-5.0-multi-nightly-metal-ipi-ovn-ipv4-rhcos10-worker-tp-amd-f7 periodic-ci-openshift-openshift-tests-private-release-5.0-multi-nightly-metal-ipi-ovn-ipv4-vmedia-rhcos10-worker-tp-amd-f7

@openshift-merge-bot
Copy link
Copy Markdown
Contributor

@sgoveas: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci Bot commented May 20, 2026

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: sgoveas
Once this PR has been reviewed and has the lgtm label, please assign memodi for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@sgoveas sgoveas changed the title Rhcos10 METAL-1745: Add 5.0 jobs for rhcos10 worker nodes May 20, 2026
@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label May 20, 2026
@openshift-ci-robot
Copy link
Copy Markdown
Contributor

openshift-ci-robot commented May 20, 2026

@sgoveas: This pull request references METAL-1745 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "5.0.0" version, but no target version was set.

Details

In response to this:

testing worker os stream

Overview

This PR configures CI infrastructure to support testing OpenShift with a RHCOS 10 (rhel-10) worker OS stream variant. It extends the baremetal IPI test pipeline to allow worker nodes to use an alternative CoreOS stream during cluster installation and testing.

CI Configuration Changes

New test jobs added for RHCOS 10 worker testing:

  • Added multi-nightly baremetal IPI test variants for release-4.23 and 5.0 that include rhcos10-worker in their names, supporting both standard and virtual-media deployments
  • These jobs are configured with WORKER_COREOS_STREAM: rhel-10 to target the RHCOS 10 stream

Baremetal IPI installation flow:

  • Extended the baremetal-lab-ipi-install step to support an optional WORKER_COREOS_STREAM variable
  • When this variable is set, the installation script patches generated manifests to override the worker node OS stream:
  • Worker BareMetalHost manifests receive the custom CoreOS stream configuration
  • Worker MachineSet manifests are updated to reference the alternate stream
  • A new MachineConfigPool is generated for workers to bind them to the custom OS image stream

Test orchestration adjustments:

  • Disabled the baremetal-lab-agent-gather step in the post-installation chain
  • Updated the baremetal-lab-wait step to use the baremetal-qe-base container image instead of the CLI image
  • Increased the cluster wait timeout from 9000 to 259200 seconds (3 days)
  • Simplified the QE test chain to only run cluster health checks

Image registry configuration:

  • Updated to use a mirrored registry (registry.build10.ci.openshift.org) for release images during installation

Impact

The infrastructure now supports testing OpenShift cluster deployments where worker nodes run RHCOS 10 instead of the standard stream, enabling validation of compatibility and stability with the newer OS variant in baremetal environments.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@sgoveas sgoveas marked this pull request as ready for review May 20, 2026 18:56
@openshift-ci openshift-ci Bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 20, 2026
@openshift-ci openshift-ci Bot requested review from jechen0648 and pamoedom May 20, 2026 18:57
@openshift-merge-bot
Copy link
Copy Markdown
Contributor

[REHEARSALNOTIFIER]
@sgoveas: the pj-rehearse plugin accommodates running rehearsal tests for the changes in this PR. Expand 'Interacting with pj-rehearse' for usage details. The following rehearsable tests have been affected by this change:

Test name Repo Type Reason
periodic-ci-openshift-openshift-tests-private-release-4.18-multi-nightly-metal-ipi-ovn-ipv4-arm-f60 N/A periodic Registry content changed
periodic-ci-openshift-cluster-kube-apiserver-operator-main-periodics-e2e-metal-encryption-kms N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.17-multi-nightly-4.17-upgrade-from-stable-4.17-metal-ipi-ovn-ipv4-arm-f360 N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.21-multi-nightly-metal-ipi-ovn-ipv4-vmedia-multi-arch-day2-amd-f28 N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.21-multi-nightly-metal-ipi-ovn-ipv4-vmedia-disruptive-arm-f28 N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.20-multi-nightly-metal-ipi-ovn-ipv4-vmedia-multi-arch-amd-f28 N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.19-multi-nightly-4.19-upgrade-from-stable-4.18-metal-ipi-ovn-ipv4-vmedia-amd-f28 N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.22-multi-nightly-metal-short-cert-rotation-arm-f7 N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.17-multi-nightly-metal-ipi-ovn-dualstack-arm-vmedia-f360 N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.16-multi-nightly-metal-ipi-ovn-ipv4-basecap-none-arm-f360 N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.22-amd64-nightly-metal-ipi-ovn-vmedia-disk-encryption-f7 N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.15-multi-nightly-metal-ipi-ovn-dualstack-amd-vmedia-f999 N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.17-multi-nightly-metal-ipi-ovn-ipv4-arm-f360 N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.16-multi-nightly-4.16-upgrade-from-stable-4.16-metal-ipi-ovn-ipv4-basecap-none-arm-f360 N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.17-multi-nightly-metal-ipi-ovn-ipv4-amd-f360 N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.17-multi-nightly-4.17-upgrade-from-stable-4.16-metal-ipi-ovn-ipv4-fips-vmedia-amd-f360 N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.20-amd64-rollback-nightly-metal-ipi-ovn-ipv4-f28 N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.13-multi-nightly-metal-ipi-ovn-ipv4-fips-vmedia-amd-f999 N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.22-multi-nightly-4.22-upgrade-from-stable-4.22-metal-ipi-ovn-ipv4-external-lb-arm-f60 N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.22-arm64-nightly-metal-ipi-ovn-ipv4-disconnected-f7 N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.20-multi-nightly-metal-ipi-ovn-dualstack-arm-vmedia-f28 N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.17-arm64-nightly-metal-ipi-ovn-vmedia-disk-encryption-f360 N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.19-multi-nightly-metal-ipi-ovn-ipv4-amd-f28 N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.21-multi-nightly-4.21-upgrade-from-stable-4.18-metal-ipi-ovn-ipv4-basecap-none-arm-f28 N/A periodic Registry content changed
periodic-ci-openshift-openshift-tests-private-release-4.22-amd64-nightly-metal-ipi-ovn-vmedia-disk-encryption-rhcos10-tp-f7 N/A periodic Registry content changed

A total of 183 jobs have been affected by this change. The above listing is non-exhaustive and limited to 25 jobs.

A full list of affected jobs can be found here

Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

@sgoveas
Copy link
Copy Markdown
Contributor Author

sgoveas commented May 20, 2026

/pj-rehearse periodic-ci-openshift-openshift-tests-private-release-5.0-multi-nightly-metal-ipi-ovn-ipv4-rhcos10-worker-tp-amd-f14 periodic-ci-openshift-openshift-tests-private-release-5.0-multi-nightly-metal-ipi-ovn-ipv4-vmedia-rhcos10-worker-tp-arm-f14

@openshift-merge-bot
Copy link
Copy Markdown
Contributor

@sgoveas: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci Bot commented May 21, 2026

@sgoveas: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/rehearse/periodic-ci-openshift-openshift-tests-private-release-4.23-multi-nightly-metal-ipi-ovn-ipv4-vmedia-rhcos10-worker-tp-arm-f7 95d2da8 link unknown /pj-rehearse periodic-ci-openshift-openshift-tests-private-release-4.23-multi-nightly-metal-ipi-ovn-ipv4-vmedia-rhcos10-worker-tp-arm-f7
ci/rehearse/periodic-ci-openshift-openshift-tests-private-release-5.0-multi-nightly-metal-ipi-ovn-ipv4-rhcos10-worker-tp-amd-f7 7127cee link unknown /pj-rehearse periodic-ci-openshift-openshift-tests-private-release-5.0-multi-nightly-metal-ipi-ovn-ipv4-rhcos10-worker-tp-amd-f7
ci/rehearse/periodic-ci-openshift-openshift-tests-private-release-4.23-multi-nightly-metal-ipi-ovn-ipv4-vmedia-rhcos10-worker-tp-amd-f7 41b0718 link unknown /pj-rehearse periodic-ci-openshift-openshift-tests-private-release-4.23-multi-nightly-metal-ipi-ovn-ipv4-vmedia-rhcos10-worker-tp-amd-f7
ci/rehearse/periodic-ci-openshift-openshift-tests-private-release-5.0-multi-nightly-metal-ipi-ovn-ipv4-vmedia-rhcos10-worker-tp-amd-f7 7127cee link unknown /pj-rehearse periodic-ci-openshift-openshift-tests-private-release-5.0-multi-nightly-metal-ipi-ovn-ipv4-vmedia-rhcos10-worker-tp-amd-f7
ci/rehearse/periodic-ci-openshift-openshift-tests-private-release-5.0-multi-nightly-metal-ipi-ovn-ipv4-rhcos10-worker-tp-amd-f14 8575dcf link unknown /pj-rehearse periodic-ci-openshift-openshift-tests-private-release-5.0-multi-nightly-metal-ipi-ovn-ipv4-rhcos10-worker-tp-amd-f14
ci/rehearse/periodic-ci-openshift-openshift-tests-private-release-5.0-multi-nightly-metal-ipi-ovn-ipv4-vmedia-rhcos10-worker-tp-arm-f14 8575dcf link unknown /pj-rehearse periodic-ci-openshift-openshift-tests-private-release-5.0-multi-nightly-metal-ipi-ovn-ipv4-vmedia-rhcos10-worker-tp-arm-f14

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

jira/valid-reference Indicates that this PR references a valid Jira ticket of any type.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants