Add upgrade lane for TNA and SNO#79596
Conversation
WalkthroughAdds two nightly upgrade CI job variants targeting RHEL 10 day-2 OS image stream: one metal OVN two-node arbiter upgrade and one AWS single-node OVN upgrade, both using the TechPreviewNoUpgrade feature set. ChangesRHEL 10 Upgrade Test Cases
Estimated code review effort🎯 2 (Simple) | ⏱️ ~10 minutes Possibly related PRs
Suggested labels
🚥 Pre-merge checks | ✅ 12✅ Passed checks (12 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: kasturinarra The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
/pj-rehearse periodic-ci-openshift-release-main-nightly-5.0-upgrade-from-stable-4.22-e2e-metal-ovn-two-node-arbiter-upgrade-os-rhel10 periodic-ci-openshift-release-main-nightly-5.0-upgrade-from-stable-4.22-e2e-aws-upgrade-ovn-single-node-os-rhel10 |
|
@kasturinarra: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
kasturinarra
force-pushed
the
add_upgrade_lane_tna
branch
from
d092f0d to
a854bb8
Compare
|
[REHEARSALNOTIFIER]
Interacting with pj-rehearseComment: Once you are satisfied with the results of the rehearsals, comment: |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@ci-operator/config/openshift/release/openshift-release-main__nightly-5.0-upgrade-from-stable-4.22.yaml`:
- Around line 91-102: The FEATURE_SET value is currently embedded inside the
DEVSCRIPTS_CONFIG multiline string; remove FEATURE_SET="TechPreviewNoUpgrade"
from DEVSCRIPTS_CONFIG and add a new top-level environment variable entry
FEATURE_SET: "TechPreviewNoUpgrade" at the same indentation level as
MCO_CONF_DAY2_OS_IMAGE_STREAM_DEFAULT so it is not part of the dev-scripts block
(refer to DEVSCRIPTS_CONFIG, FEATURE_SET and
MCO_CONF_DAY2_OS_IMAGE_STREAM_DEFAULT to locate the change).
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Enterprise
Run ID: f93e6639-47a5-42be-9d56-663e2233f76a
⛔ Files ignored due to path filters (1)
ci-operator/jobs/openshift/release/openshift-release-main-periodics.yamlis excluded by!ci-operator/jobs/**
📒 Files selected for processing (1)
ci-operator/config/openshift/release/openshift-release-main__nightly-5.0-upgrade-from-stable-4.22.yaml
| DEVSCRIPTS_CONFIG: | | ||
| IP_STACK=v4 | ||
| NUM_MASTERS=2 | ||
| MASTER_DISK=100 | ||
| MASTER_MEMORY=32768 | ||
| NUM_ARBITERS=1 | ||
| ARBITER_DISK=50 | ||
| ARBITER_MEMORY=8192 | ||
| ARBITER_VCPU=2 | ||
| NUM_WORKERS=0 | ||
| FEATURE_SET="TechPreviewNoUpgrade" | ||
| MCO_CONF_DAY2_OS_IMAGE_STREAM_DEFAULT: rhel-10 |
There was a problem hiding this comment.
Move FEATURE_SET out of DEVSCRIPTS_CONFIG block.
Line 101 places FEATURE_SET="TechPreviewNoUpgrade" inside the DEVSCRIPTS_CONFIG multiline string. FEATURE_SET should be a separate environment variable at the same level as MCO_CONF_DAY2_OS_IMAGE_STREAM_DEFAULT (line 102), not embedded within the dev-scripts configuration content.
For consistency, compare with the AWS single-node job variant at line 168, where FEATURE_SET is correctly defined as a separate env var.
🔧 Proposed fix
env:
DEVSCRIPTS_CONFIG: |
IP_STACK=v4
NUM_MASTERS=2
MASTER_DISK=100
MASTER_MEMORY=32768
NUM_ARBITERS=1
ARBITER_DISK=50
ARBITER_MEMORY=8192
ARBITER_VCPU=2
NUM_WORKERS=0
- FEATURE_SET="TechPreviewNoUpgrade"
+ FEATURE_SET: TechPreviewNoUpgrade
MCO_CONF_DAY2_OS_IMAGE_STREAM_DEFAULT: rhel-10📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| DEVSCRIPTS_CONFIG: | | |
| IP_STACK=v4 | |
| NUM_MASTERS=2 | |
| MASTER_DISK=100 | |
| MASTER_MEMORY=32768 | |
| NUM_ARBITERS=1 | |
| ARBITER_DISK=50 | |
| ARBITER_MEMORY=8192 | |
| ARBITER_VCPU=2 | |
| NUM_WORKERS=0 | |
| FEATURE_SET="TechPreviewNoUpgrade" | |
| MCO_CONF_DAY2_OS_IMAGE_STREAM_DEFAULT: rhel-10 | |
| DEVSCRIPTS_CONFIG: | | |
| IP_STACK=v4 | |
| NUM_MASTERS=2 | |
| MASTER_DISK=100 | |
| MASTER_MEMORY=32768 | |
| NUM_ARBITERS=1 | |
| ARBITER_DISK=50 | |
| ARBITER_MEMORY=8192 | |
| ARBITER_VCPU=2 | |
| NUM_WORKERS=0 | |
| FEATURE_SET: TechPreviewNoUpgrade | |
| MCO_CONF_DAY2_OS_IMAGE_STREAM_DEFAULT: rhel-10 |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@ci-operator/config/openshift/release/openshift-release-main__nightly-5.0-upgrade-from-stable-4.22.yaml`
around lines 91 - 102, The FEATURE_SET value is currently embedded inside the
DEVSCRIPTS_CONFIG multiline string; remove FEATURE_SET="TechPreviewNoUpgrade"
from DEVSCRIPTS_CONFIG and add a new top-level environment variable entry
FEATURE_SET: "TechPreviewNoUpgrade" at the same indentation level as
MCO_CONF_DAY2_OS_IMAGE_STREAM_DEFAULT so it is not part of the dev-scripts block
(refer to DEVSCRIPTS_CONFIG, FEATURE_SET and
MCO_CONF_DAY2_OS_IMAGE_STREAM_DEFAULT to locate the change).
There was a problem hiding this comment.
the existing baremetalds-two-node-arbiter-techpreview workflow has FEATURE_SET="TechPreviewNoUpgrade" inside DEVSCRIPTS_CONFIG at line 23. This is the correct pattern for
devscripts-based baremetal jobs. Moving it out would break the cluster installation.
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Check the baremetalds-two-node-arbiter-techpreview workflow referenced by the user
fd -t f 'baremetalds-two-node-arbiter-techpreview' --exec cat {}
# Also look for FEATURE_SET inside DEVSCRIPTS_CONFIG in any baremetal/devscripts workflow/step files
rg -n 'FEATURE_SET' ci-operator/step-registry/baremetalds/ -A2 -B2
Length of output: 4898
@kasturinarra, you're absolutely right. The script output confirms that the established pattern in baremetalds-two-node-arbiter-techpreview-workflow.yaml (and other sibling techpreview workflows) places FEATURE_SET="TechPreviewNoUpgrade" inside DEVSCRIPTS_CONFIG — because dev-scripts reads it from there during cluster installation. The new job correctly mirrors that pattern. My suggestion was wrong, and moving FEATURE_SET out would indeed break the cluster installation.
Apologies for the noise. I'll resolve my comment.
🏁 Script executed:
#!/bin/bash
# List CodeRabbit's comments on this PR to find the one to resolve
gh api repos/openshift/release/pulls/79596/comments --jq '.[] | select(.user.login == "coderabbitai[bot]") | {id: .id, body: .body[0:80]}'
Length of output: 111
🏁 Script executed:
#!/bin/bash
# Resolve the review thread containing comment id 3283325307
# First, get the pull request review thread id
gh api graphql -f query='
{
repository(owner: "openshift", name: "release") {
pullRequest(number: 79596) {
reviewThreads(first: 50) {
nodes {
id
isResolved
comments(first: 5) {
nodes {
databaseId
author { login }
}
}
}
}
}
}
}'
Length of output: 262
🏁 Script executed:
#!/bin/bash
# Resolve the review thread
gh api graphql -f query='
mutation {
resolveReviewThread(input: {threadId: "PRRT_kwDOBHvUUM6D5Cs5"}) {
thread {
id
isResolved
}
}
}'
Length of output: 266
I wasn't able to resolve the comment programmatically due to GitHub permissions. You can manually resolve it by clicking the Resolve conversation button on the review thread. Apologies again for the incorrect suggestion!
✏️ Learnings added
Learnt from: kasturinarra
Repo: openshift/release PR: 79596
File: ci-operator/config/openshift/release/openshift-release-main__nightly-5.0-upgrade-from-stable-4.22.yaml:91-102
Timestamp: 2026-05-21T18:09:58.685Z
Learning: In devscripts-based baremetal CI jobs in the openshift/release repository (e.g., baremetalds-two-node-arbiter or baremetalds-two-node-fencing workflows), FEATURE_SET="TechPreviewNoUpgrade" must be placed inside the DEVSCRIPTS_CONFIG multiline block, not as a separate top-level env var. Dev-scripts reads FEATURE_SET from within DEVSCRIPTS_CONFIG during cluster installation; extracting it would break the installation. This is the canonical pattern used across all devscripts-based techpreview workflow definitions in ci-operator/step-registry/baremetalds/.
Learnt from: kasturinarra
Repo: openshift/release PR: 79596
File: ci-operator/config/openshift/release/openshift-release-main__nightly-5.0-upgrade-from-stable-4.22.yaml:91-102
Timestamp: 2026-05-21T18:09:58.685Z
Learning: In devscripts-based baremetal CI jobs in the openshift/release repository (e.g., baremetalds-two-node-arbiter or baremetalds-two-node-fencing workflows), FEATURE_SET="TechPreviewNoUpgrade" must be placed inside the DEVSCRIPTS_CONFIG multiline block, not as a separate top-level env var. Dev-scripts reads FEATURE_SET from within DEVSCRIPTS_CONFIG during cluster installation; extracting it would break the installation. This is the canonical pattern used across all devscripts-based techpreview workflow definitions in ci-operator/step-registry/baremetalds/.
Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.
🧠 Learnings used
Learnt from: PillaiManish
Repo: openshift/release PR: 79231
File: ci-operator/config/openshift/secrets-store-csi-driver-operator/openshift-secrets-store-csi-driver-operator-main.yaml:0-0
Timestamp: 2026-05-20T05:41:25.164Z
Learning: When reviewing OpenShift/Kubernetes operator manifests/CSVs, if a ClusterServiceVersion (CSV) defines an `emptyDir` volume mounted at `/tmp` with `medium: Memory`, treat that data as scoped to the *pod lifecycle* (not the container lifecycle). Container restarts within the same pod (e.g., restart after SIGTERM/termination and recreation of the container) should not wipe the `emptyDir` contents; data written to something like `/tmp/e2e-cover` should still be present and can be collected afterward (e.g., via `oc cp`). The `emptyDir` is cleared when the pod is deleted/recreated/rescheduled (pod lifecycle ends), not on container restarts.
Learnt from: rdiscala
Repo: openshift/release PR: 79357
File: ci-operator/config/openshift-kni/eco-ci-cd/openshift-kni-eco-ci-cd-main__cnf-ran-sno-day2-worker-4.18.yaml:23-23
Timestamp: 2026-05-20T11:54:20.424Z
Learning: In the openshift/release repository CI operator configuration YAMLs, do not treat an intentionally impossible cron schedule (e.g., `0 0 31 2 *` for “Feb 31”) as a misconfiguration. This pattern is an intentional mechanism used to disable/park a periodic CI job temporarily (e.g., while an upstream bug is being fixed). Only flag cron syntax errors when the schedule is not an intentional “impossible date” disable pattern used for job parking.
Learnt from: vr4manta
Repo: openshift/release PR: 79548
File: ci-operator/config/openshift/release/openshift-release-main__nightly-4.22.yaml:1189-1189
Timestamp: 2026-05-20T17:04:37.102Z
Learning: In the openshift/release repo, for multi-stage (workflow-based) ci-operator tests, CI-related environment variables (e.g., TEST_SKIPS) should be treated as coming from ci-operator/config YAML files that ci-operator loads at runtime. If you modify only env vars in ci-operator/config (and then run `make update`), it may produce no changes under ci-operator/jobs/—this is expected because the values are not embedded into the generated Prow job artifacts. In code review, do not treat a lack of downstream job-file diffs as evidence that `make update` was skipped for env-var-only changes in these multi-stage configs.
Learnt from: vr4manta
Repo: openshift/release PR: 79548
File: ci-operator/config/openshift/release/openshift-release-main__nightly-4.22.yaml:1189-1189
Timestamp: 2026-05-20T17:04:37.102Z
Learning: In the openshift/release repo, for multi-stage (workflow-based) tests configured via ci-operator/config YAML, runtime environment variables (e.g., TEST_SKIPS) are defined in ci-operator/config YAML and read by ci-operator at runtime; they are not embedded into the generated Prow job files under ci-operator/jobs/. Therefore, when a change updates only those env vars in ci-operator/config YAML, `make update` may legitimately produce no changes to downstream artifacts (including generated job files). In code review, do not flag missing artifact regeneration when the only differences are these ci-operator/config env-var updates for multi-stage tests.
Learnt from: dhaiducek
Repo: openshift/release PR: 79563
File: ci-operator/config/stolostron/cluster-proxy/stolostron-cluster-proxy-main.yaml:41-43
Timestamp: 2026-05-20T19:31:57.286Z
Learning: In the openshift/release repo, for ci-operator container test step config YAML files under ci-operator/config/, remember that the `commands` field is read by ci-operator at runtime from the config YAML and is not embedded into the generated ProwJob YAML under `ci-operator/jobs/`. Therefore, if a PR changes only `commands` (and not any ProwJob-spec affecting structural fields such as test/job names, job types/spec, or metadata that changes the resulting ProwJob spec), you generally do NOT need to run `make update` and regenerating artifacts should be a no-op diff. Run `make update` only when the changes affect the generated ProwJob spec (e.g., new/renamed tests, new job types, or other spec/metadata changes that alter `ci-operator/jobs/` output).
Learnt from: tthvo
Repo: openshift/release PR: 79564
File: ci-operator/config/openshift/multiarch/openshift-multiarch-main__nightly-4.20-upgrade-from-nightly-4.19.yaml:94-94
Timestamp: 2026-05-20T20:01:14.481Z
Learning: In the openshift/release repo, when editing ci-operator config YAMLs under ci-operator/config/, changes that only modify the *values* of environment variables inside test step `env` blocks (e.g., COMPUTE_NODE_TYPE, ADDITIONAL_WORKER_VM_TYPE, MIGRATION_INFRA_MACHINE_TYPE) do not change the generated ProwJob specs under ci-operator/jobs/. For these value-only env tweaks, reviewers should not require running `make update` because it should produce no diff. Require `make update` only for structural/semantic changes that can affect the generated ProwJob spec (e.g., adding/renaming tests, changing job types, or altering metadata/fields that feed into the ProwJob spec).
Learnt from: karampok
Repo: openshift/release PR: 79537
File: ci-operator/config/openshift/bond-cni/openshift-bond-cni-main.yaml:32-33
Timestamp: 2026-05-21T09:04:21.025Z
Learning: In openshift/release, when modifying a ci-operator config YAML under ci-operator/config/ by adding an `env` block (key/value environment variables such as `TEST_TYPE`, `TEST_SUITE`, `TEST_ARGS`) to an existing test step, this change does not affect the generated ProwJob specs under ci-operator/jobs/. In this case, `make update` is a no-op—do not request or flag that `make update` was (or needs to be) run for these env-only edits.
|
@kasturinarra: The following tests failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
|
/pj-rehearse periodic-ci-openshift-release-main-nightly-5.0-upgrade-from-stable-4.22-e2e-aws-upgrade-ovn-single-node-os-rhel10 periodic-ci-openshift-release-main-nightly-5.0-upgrade-from-stable-4.22-e2e-metal-ovn-two-node-arbiter-upgrade-os-rhel10 |
|
@kasturinarra: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
1 participant
Summary by CodeRabbit
This PR updates the OpenShift CI configuration in the openshift/release repository by adding two TechPreview "day-2 OS image = RHEL 10" upgrade lanes to the nightly-5.0-upgrade-from-stable-4.22 variant. These are periodic CI tests that validate RHEL 10 day-2 OS image handling during upgrades from 4.22 → 5.0 across two cluster topologies.
What changed and where
New/modified test entries
@weekly'Notes