Clear ClusterVersion channel in ROSA CI to prevent CannotRetrieveUpdatesSRE - #83056
Clear ClusterVersion channel in ROSA CI to prevent CannotRetrieveUpdatesSRE#83056dustman9000 wants to merge 4 commits into
Conversation
WalkthroughThe wait-ready step now optionally clears ChangesROSA ClusterVersion channel clearing
Estimated code review effort: 2 (Simple) | ~10 minutes Suggested reviewers: 🚥 Pre-merge checks | ✅ 15✅ Passed checks (15 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@ci-operator/step-registry/rosa/cluster/wait-ready/operators/rosa-cluster-wait-ready-operators-commands.sh`:
- Around line 90-93: Update the CLEAR_CLUSTERVERSION_CHANNEL branch to stop
suppressing failures from the oc patch command. Remove the trailing “|| true”,
or implement bounded retries that cause the step to fail if patching
clusterversion version does not succeed.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Enterprise
Run ID: f341175b-7f8b-4bdc-a71b-545cd573d06d
📒 Files selected for processing (4)
ci-operator/step-registry/rosa/aws/hcp/conformance/rosa-aws-hcp-conformance-workflow.yamlci-operator/step-registry/rosa/aws/sts/conformance/rosa-aws-sts-conformance-workflow.yamlci-operator/step-registry/rosa/cluster/wait-ready/operators/rosa-cluster-wait-ready-operators-commands.shci-operator/step-registry/rosa/cluster/wait-ready/operators/rosa-cluster-wait-ready-operators-ref.yaml
| if [[ "${CLEAR_CLUSTERVERSION_CHANNEL:-}" == "true" ]]; then | ||
| echo "Clearing ClusterVersion channel to prevent CannotRetrieveUpdatesSRE alert..." | ||
| oc patch clusterversion version --type merge -p '{"spec":{"channel":""}}' || true | ||
| fi |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
Do not suppress a failed channel patch.
When CLEAR_CLUSTERVERSION_CHANNEL is enabled, this patch is the required mitigation. || true hides API, permission, and admission failures. The step can then succeed while spec.channel remains set, so the alert can still fire and the conformance job can fail. Remove || true, or use bounded retries and fail when the patch does not succeed.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@ci-operator/step-registry/rosa/cluster/wait-ready/operators/rosa-cluster-wait-ready-operators-commands.sh`
around lines 90 - 93, Update the CLEAR_CLUSTERVERSION_CHANNEL branch to stop
suppressing failures from the oc patch command. Remove the trailing “|| true”,
or implement bounded retries that cause the step to fail if patching
clusterversion version does not succeed.
|
[REHEARSALNOTIFIER]
A total of 317 jobs have been affected by this change. The above listing is non-exhaustive and limited to 25 jobs. A full list of affected jobs can be found here Interacting with pj-rehearseComment: Once you are satisfied with the results of the rehearsals, comment: |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: dustman9000 The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
Bot
added
the
approved
|
@dustman9000: all tests passed! Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
1 participant
Summary
CLEAR_CLUSTERVERSION_CHANNELenv var torosa-cluster-wait-ready-operatorssteprosa-aws-sts-conformanceandrosa-aws-hcp-conformanceworkflowsCannotRetrieveUpdatesSREfrom firing during conformance test runs on nightly payloadsProblem
ROSA CI conformance jobs on 4.20 nightlies are failing the
[sig-trt][invariant] No new alerts should be firingmonitor test becauseCannotRetrieveUpdatesSREfires during the ~1h test run.The alert (from managed-cluster-config) fires when the CVO hasn't retrieved updates in 1+ hour. On nightly CI clusters there's no valid update graph, so the CVO retrieval timestamp goes stale and trips the threshold. The openshift-tests alert framework has no historical data for this ROSA-only alert, so it flags it as "new and should not be firing."
Failing jobs:
periodic-ci-openshift-release-main-nightly-4.20-e2e-rosa-sts-ovn(run 2085209519696121856)Fix
When
CLEAR_CLUSTERVERSION_CHANNEL=true, the step clearsspec.channelon ClusterVersion after operators are ready. This makes the CVO reportreason=NoChannelon theRetrievedUpdatescondition, which the alert expression explicitly excludes (reason!="NoChannel").The env var defaults to
false, so existing jobs are unaffected. Only the two conformance workflows opt in.Changes
rosa-cluster-wait-ready-operators-commands.shrosa-cluster-wait-ready-operators-ref.yamlCLEAR_CLUSTERVERSION_CHANNELenv var (default:false)rosa-aws-sts-conformance-workflow.yamlCLEAR_CLUSTERVERSION_CHANNEL: "true"rosa-aws-hcp-conformance-workflow.yamlCLEAR_CLUSTERVERSION_CHANNEL: "true"Test plan
rosa-cluster-wait-ready-operatorsstep succeeds withCLEAR_CLUSTERVERSION_CHANNEL=trueCannotRetrieveUpdatesSREno longer fires during conformance runsrosa-cluster-wait-ready-operatorsare unaffected (env var defaults tofalse)Summary by CodeRabbit
CLEAR_CLUSTERVERSION_CHANNELsetting torosa-cluster-wait-ready-operators.ClusterVersion.spec.channelafter operators become ready. This setsRetrievedUpdatestoreason=NoChanneland prevents theCannotRetrieveUpdatesSREalert during ROSA conformance tests.false.