Automate Reliability tests on Prow - #83100
Conversation
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: mehabhalodiya The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
WalkthroughAdded four OpenShift 5.0 nightly reliability-v2 CI variants for AWS, Azure, ROSA, and ROSA HCP. Each variant defines 68-hour, 20-hour, and 1-hour jobs. Reliability and Plank timeouts now allow 72-hour runs. ChangesReliability v2 nightly CI
Estimated code review effort: 3 (Moderate) | ~25 minutes Suggested reviewers: Sequence Diagram(s)sequenceDiagram
participant CI as CI configuration
participant Workflow as Installer workflow
participant Cluster as Cloud cluster
participant Chain as Reliability v2 chain
CI->>Workflow: Start 68-hour, 20-hour, or 1-hour job
Workflow->>Cluster: Provision provider-specific cluster
Cluster->>Chain: Run reliability-v2 deployment and test chain
Important Pre-merge checks failedPlease resolve all errors before merging. Addressing warnings is optional. ❌ Failed checks (1 error)
✅ Passed checks (14 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@ci-operator/config/openshift/svt/openshift-svt-master__reliability-v2-aws-5.0-nightly-x86.yaml`:
- Around line 25-72: Add the intended periodic cron schedule to each
reliability-v2-68h, reliability-v2-20h, and reliability-v2-1h test in
ci-operator/config/openshift/svt/openshift-svt-master__reliability-v2-aws-5.0-nightly-x86.yaml
lines 25-72,
ci-operator/config/openshift/svt/openshift-svt-master__reliability-v2-azure-5.0-nightly-x86.yaml
lines 25-78,
ci-operator/config/openshift/svt/openshift-svt-master__reliability-v2-rosa-5.0-nightly-x86.yaml
lines 29-89, and
ci-operator/config/openshift/svt/openshift-svt-master__reliability-v2-rosa_hcp-5.0-nightly-x86.yaml
lines 29-92, preserving their existing job configuration while ensuring all 12
tests run periodically.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Enterprise
Run ID: b371c976-b598-4145-b981-df097669f711
⛔ Files ignored due to path filters (1)
ci-operator/jobs/openshift/svt/openshift-svt-master-presubmits.yamlis excluded by!ci-operator/jobs/**
📒 Files selected for processing (6)
ci-operator/config/openshift/svt/openshift-svt-master__reliability-v2-aws-5.0-nightly-x86.yamlci-operator/config/openshift/svt/openshift-svt-master__reliability-v2-azure-5.0-nightly-x86.yamlci-operator/config/openshift/svt/openshift-svt-master__reliability-v2-rosa-5.0-nightly-x86.yamlci-operator/config/openshift/svt/openshift-svt-master__reliability-v2-rosa_hcp-5.0-nightly-x86.yamlci-operator/step-registry/openshift/svt/reliability-v2/openshift-svt-reliability-v2-ref.yamlcore-services/prow/02_config/_config.yaml
Signed-off-by: mehabhalodiya <mehabhalodiya@gmail.com>
b1876d1 to
9855cb2
Compare
|
Note GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer. |
|
[REHEARSALNOTIFIER]
Interacting with pj-rehearseComment: Once you are satisfied with the results of the rehearsals, comment: |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@ci-operator/step-registry/openshift/svt/reliability-v2/openshift-svt-reliability-v2-commands.sh`:
- Around line 21-24: Disable xtrace for the child process by changing the
command that invokes ./start.sh from “bash -x” to an untraced invocation, while
preserving the existing token-loading guard. Add a comment immediately before
the set +x block explaining that tracing is disabled to prevent SLACK_API_TOKEN
from appearing in CI logs during configuration generation.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Enterprise
Run ID: 52772d4a-dab1-44bd-baed-a139250c30fe
⛔ Files ignored due to path filters (1)
ci-operator/jobs/openshift/svt/openshift-svt-master-presubmits.yamlis excluded by!ci-operator/jobs/**
📒 Files selected for processing (7)
ci-operator/config/openshift/svt/openshift-svt-master__reliability-v2-aws-5.0-nightly-x86.yamlci-operator/config/openshift/svt/openshift-svt-master__reliability-v2-azure-5.0-nightly-x86.yamlci-operator/config/openshift/svt/openshift-svt-master__reliability-v2-rosa-5.0-nightly-x86.yamlci-operator/config/openshift/svt/openshift-svt-master__reliability-v2-rosa_hcp-5.0-nightly-x86.yamlci-operator/step-registry/openshift/svt/reliability-v2/openshift-svt-reliability-v2-commands.shci-operator/step-registry/openshift/svt/reliability-v2/openshift-svt-reliability-v2-ref.yamlcore-services/prow/02_config/_config.yaml
🚧 Files skipped from review as they are similar to previous changes (6)
- core-services/prow/02_config/_config.yaml
- ci-operator/step-registry/openshift/svt/reliability-v2/openshift-svt-reliability-v2-ref.yaml
- ci-operator/config/openshift/svt/openshift-svt-master__reliability-v2-rosa_hcp-5.0-nightly-x86.yaml
- ci-operator/config/openshift/svt/openshift-svt-master__reliability-v2-rosa-5.0-nightly-x86.yaml
- ci-operator/config/openshift/svt/openshift-svt-master__reliability-v2-aws-5.0-nightly-x86.yaml
- ci-operator/config/openshift/svt/openshift-svt-master__reliability-v2-azure-5.0-nightly-x86.yaml
| set +x | ||
| SLACK_API_TOKEN=$(cat "/token/reliability-v2-slack-api-token") | ||
| export SLACK_API_TOKEN | ||
| set -x |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
Do not trace the child process that consumes SLACK_API_TOKEN.
The guard protects only the cat command and the export. Line 32 then runs bash -x ./start.sh. The current upstream reliability-v2/start.sh expands $SLACK_API_TOKEN in a sed command while generating its configuration. Xtrace can print the expanded command and expose the token in Prow logs. (raw.githubusercontent.com)
Remove -x from Line 32. Add a comment before Line 21 that explains why tracing is disabled.
As per coding guidelines, step-registry command scripts must avoid leaking tokens into CI logs and must document tracing-disabled scopes.
Proposed fix
+# Disable tracing while reading and exporting the Slack API token.
set +x
SLACK_API_TOKEN=$(cat "/token/reliability-v2-slack-api-token")
export SLACK_API_TOKEN
set -x
...
-bash -x ./start.sh -n reliability -t $RELIABILITY_DURATION -c $CONFIG_TEMPLATE -r $TOLERANCE_RATE
+bash ./start.sh -n reliability -t $RELIABILITY_DURATION -c $CONFIG_TEMPLATE -r $TOLERANCE_RATE🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@ci-operator/step-registry/openshift/svt/reliability-v2/openshift-svt-reliability-v2-commands.sh`
around lines 21 - 24, Disable xtrace for the child process by changing the
command that invokes ./start.sh from “bash -x” to an untraced invocation, while
preserving the existing token-loading guard. Add a comment immediately before
the set +x block explaining that tracing is disabled to prevent SLACK_API_TOKEN
from appearing in CI logs during configuration generation.
Sources: Coding guidelines, MCP tools
|
@mehabhalodiya: all tests passed! Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
1 participant
Each 5.0 config now has 3 tests:
reliability-v2-68h,reliability-v2-20h,reliability-v2-1h.The 68h reliability duration leaves ~4h buffer within the 72h job timeout for cluster provisioning, Dittybopper deployment, must-gather collection, and teardown.
12 Prow jobs total (3 durations x 4 platforms) are now generated and ready to trigger with commands like:
/test reliability-v2-aws-5.0-nightly-x86-reliability-v2-68h/test reliability-v2-azure-5.0-nightly-x86-reliability-v2-68h/test reliability-v2-rosa-5.0-nightly-x86-reliability-v2-68h/test reliability-v2-rosa_hcp-5.0-nightly-x86-reliability-v2-68hWhat each test automates end-to-end:
idp-htpasswdfor self-managed; ROSA workflows handle it nativelyopenshift-qe-deploy-dittybopperstep deploys monitoring dashboardsopenshift-svt-reliability-v2runs the test for the configured durationSummary by CodeRabbit
Adds 12 Prow jobs for OpenShift 5.0 Reliability v2 nightly tests across AWS, Azure, ROSA, and ROSA HCP. Each platform runs 1-hour, 20-hour, and 68-hour tests with platform-specific provisioning and deployment workflows.
The jobs automate cluster provisioning, identity provider setup where required, Dittybopper deployment, reliability testing, must-gather collection, and cluster teardown. The 68-hour tests reserve time for setup and cleanup within the 72-hour job limit.
Increases the Reliability v2 step timeout and the
openshift/svtPlank timeout to 72 hours. ProtectsSLACK_API_TOKENwhile the test command reads and exports it.