Skip to content

[WIP] Jv perf scale baseline only has nightlies#79848

Open
JoukoVirtanen wants to merge 7 commits into
openshift:mainfrom
JoukoVirtanen:jv-perf-scale-baseline-only-has-nightlies
Open

[WIP] Jv perf scale baseline only has nightlies#79848
JoukoVirtanen wants to merge 7 commits into
openshift:mainfrom
JoukoVirtanen:jv-perf-scale-baseline-only-has-nightlies

Conversation

@JoukoVirtanen
Copy link
Copy Markdown
Contributor

@JoukoVirtanen JoukoVirtanen commented May 28, 2026

Summary by CodeRabbit

This PR makes Orion baseline selection configurable and centralizes the baseline-detection logic used by OpenShift CI perf/scale steps.

What changed in practical terms:

  • Adds a new environment variable ORION_BASELINE_JOB_TYPE to several ci-operator step-ref manifests used by OpenShift QE and StackRox perfscale. When set, this variable forces Orion to use the provided job_type as the baseline filter (e.g., set to "periodic" to restrict baselines to nightly runs). Affected refs:
    • openshift-qe-orion-ref, cluster-density, node-density, crd-scale
    • stackrox-perfscale-chain (here ORION_BASELINE_JOB_TYPE defaults to "periodic")
  • Refactors the Orion helper script (ci-operator/step-registry/openshift-qe/orion/openshift-qe-orion-commands.sh) by introducing three helper functions:
    • classify_baseline_job(): classifies the baseline category (override, periodic, periodic_with_pull, presubmit_pull, rehearse, rehearse_with_pull) based on ORION_BASELINE_JOB_TYPE, JOB_TYPE, JOB_NAME, and PULL_NUMBER.
    • resolve_baseline_job_type(baseline_category): returns the job_type filter expression used for Orion queries (honors the override).
    • resolve_pull_number(baseline_category): returns the pull_number expression used for Orion queries (handles cases where pull numbers should be included).
      The script now uses these helpers instead of inline case/if logic to compute job_type and pull_number for Orion queries.

Effect and rationale:

  • Teams can explicitly control which job runs Orion uses as the baseline (improving consistency by allowing e.g., baseline-only nightly comparisons and excluding PR/rehearsal runs).
  • The refactor centralizes and documents the baseline-selection behavior and makes it straightforward to opt a step into baseline-only comparisons via ORION_BASELINE_JOB_TYPE.

Other notes:

  • Most refs keep ORION_BASELINE_JOB_TYPE default empty (preserving automatic detection); StackRox’s chain sets it to "periodic".
  • The PR is WIP and currently held by the author.

@openshift-ci openshift-ci Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 28, 2026
@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci Bot commented May 28, 2026

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented May 28, 2026

Walkthrough

Extracts Orion baseline job classification into three Bash helpers, replaces inline computation with those helpers, and adds an ORION_BASELINE_JOB_TYPE env var to several Orion steps and the StackRox perfscale chain; also adds a new StackRox perf-scale CI config file.

Changes

Orion baseline job type classification refactoring

Layer / File(s) Summary
Helper function extraction and integration
ci-operator/step-registry/openshift-qe/orion/openshift-qe-orion-commands.sh
Adds classify_baseline_job(), resolve_baseline_job_type(), and resolve_pull_number() and replaces prior inline case/if logic with calls that compute baseline_category, job_type, and pull_number.
Env var additions to Orion step refs and chain
ci-operator/step-registry/openshift-qe/orion/openshift-qe-orion-ref.yaml, ci-operator/step-registry/openshift-qe/orion/cluster-density/openshift-qe-orion-cluster-density-ref.yaml, ci-operator/step-registry/openshift-qe/orion/crd-scale/openshift-qe-orion-crd-scale-ref.yaml, ci-operator/step-registry/openshift-qe/orion/node-density/openshift-qe-orion-node-density-ref.yaml, ci-operator/step-registry/stackrox/perfscale/stackrox-perfscale-chain.yaml
Adds ORION_BASELINE_JOB_TYPE env entries (default "" in Orion steps, default "periodic" in the StackRox perfscale chain) and documents that it overrides automatic baseline job_type detection.
New perf-scale CI Operator config
ci-operator/config/stackrox/stackrox/stackrox-stackrox-jv-test-perf-scale__perf-scale.yaml
Adds a new perf-scale CI config defining images, release target, resource defaults, a 24-node scale test job, env/config for perf-scale runs, hooks, timeout, and generated metadata.

🎯 3 (Moderate) | ⏱️ ~20 minutes

🚥 Pre-merge checks | ✅ 13 | ❌ 2

❌ Failed checks (1 warning, 1 inconclusive)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
Title check ❓ Inconclusive The title is vague and uses non-descriptive language ("baseline only has nightlies") that doesn't clearly convey the substantive changes being made. Revise the title to clearly describe the main change, such as 'Add ORION_BASELINE_JOB_TYPE environment variable to support baseline job type filtering' or 'Refactor Orion baseline job type detection and resolution logic'.
✅ Passed checks (13 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed PR contains only CI/CD configuration files (Bash scripts and YAML files) with no Ginkgo tests, making this check inapplicable.
Test Structure And Quality ✅ Passed The custom check for Ginkgo test structure is not applicable to this PR, which only modifies Bash scripts and YAML configuration files with no test code.
Microshift Test Compatibility ✅ Passed PR does not add any Ginkgo e2e tests; all changes are CI/CD infrastructure (bash scripts, YAML configs). The MicroShift test compatibility check is not applicable to non-test code changes.
Single Node Openshift (Sno) Test Compatibility ✅ Passed No Ginkgo e2e tests added in this PR. Changes are limited to Bash shell script helpers and YAML CI/CD configuration files, which do not contain or define Ginkgo test cases.
Topology-Aware Scheduling Compatibility ✅ Passed PR only modifies CI/CD infrastructure files (ci-operator configs and bash scripts), not deployment manifests, operators, or pod specs. No scheduling constraints present. Check is not applicable.
Ote Binary Stdout Contract ✅ Passed PR contains no OTE binaries—only Bash CI scripts and YAML configs. OTE binary stdout contract check is not applicable.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed PR does not add Ginkgo e2e tests. Changes are CI/CD YAML configs and shell script helpers only, not Go test files.
No-Weak-Crypto ✅ Passed No weak cryptography detected: comprehensive search found 0 instances of MD5, SHA1, DES, RC4, 3DES, Blowfish, ECB, custom crypto, or non-constant-time secret comparisons in PR files.
Container-Privileges ✅ Passed No container privilege escalation patterns found in any of the 7 modified files. The PR adds environment variables and Bash functions without any privileged container configurations.
No-Sensitive-Data-In-Logs ✅ Passed PR's new functions output only safe values (job classifications, PR numbers). They don't log passwords, tokens, API keys, or PII. Pre-existing credential logging is orthogonal to the refactoring.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@openshift-merge-bot
Copy link
Copy Markdown
Contributor

@JoukoVirtanen, pj-rehearse: unable to determine affected jobs. This could be due to a branch that needs to be rebased. ERROR:

could not determine changed registry steps: could not load step registry: [chain/stackrox-perfscale: parameter "ORION_BASELINE_JOB_TYPE" is overridden in [chain/stackrox-perfscale] but not declared in any step, workflow/stackrox-perfscale: chain/stackrox-perfscale: parameter "ORION_BASELINE_JOB_TYPE" is overridden in [chain/stackrox-perfscale] but not declared in any step]
Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented May 28, 2026

Actionable comments posted: 0

@JoukoVirtanen JoukoVirtanen marked this pull request as ready for review May 28, 2026 22:32
@JoukoVirtanen
Copy link
Copy Markdown
Contributor Author

/hold

@openshift-ci openshift-ci Bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label May 28, 2026
@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci Bot commented May 29, 2026

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: JoukoVirtanen
Once this PR has been reviewed and has the lgtm label, please assign sandeepyadav93 for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-merge-bot
Copy link
Copy Markdown
Contributor

[REHEARSALNOTIFIER]
@JoukoVirtanen: the pj-rehearse plugin accommodates running rehearsal tests for the changes in this PR. Expand 'Interacting with pj-rehearse' for usage details. The following rehearsable tests have been affected by this change:

Test name Repo Type Reason
pull-ci-cloud-bulldozer-orion-main-payload-control-plane-6nodes cloud-bulldozer/orion presubmit Registry content changed
pull-ci-openshift-cluster-network-operator-master-qe-perfscale-aws-ovn-medium-cluster-density openshift/cluster-network-operator presubmit Registry content changed
pull-ci-openshift-cluster-network-operator-master-qe-perfscale-aws-ovn-small-cluster-density openshift/cluster-network-operator presubmit Registry content changed
pull-ci-openshift-cluster-network-operator-release-5.1-qe-perfscale-aws-ovn-medium-cluster-density openshift/cluster-network-operator presubmit Registry content changed
pull-ci-openshift-cluster-network-operator-release-5.1-qe-perfscale-aws-ovn-small-cluster-density openshift/cluster-network-operator presubmit Registry content changed
pull-ci-openshift-cluster-network-operator-release-5.0-qe-perfscale-aws-ovn-medium-cluster-density openshift/cluster-network-operator presubmit Registry content changed
pull-ci-openshift-cluster-network-operator-release-5.0-qe-perfscale-aws-ovn-small-cluster-density openshift/cluster-network-operator presubmit Registry content changed
pull-ci-openshift-cluster-network-operator-release-4.23-qe-perfscale-aws-ovn-medium-cluster-density openshift/cluster-network-operator presubmit Registry content changed
pull-ci-openshift-cluster-network-operator-release-4.23-qe-perfscale-aws-ovn-small-cluster-density openshift/cluster-network-operator presubmit Registry content changed
pull-ci-openshift-cluster-network-operator-release-4.22-qe-perfscale-aws-ovn-medium-cluster-density openshift/cluster-network-operator presubmit Registry content changed
pull-ci-openshift-cluster-network-operator-release-4.22-qe-perfscale-aws-ovn-small-cluster-density openshift/cluster-network-operator presubmit Registry content changed
pull-ci-openshift-cluster-network-operator-release-4.21-qe-perfscale-aws-ovn-medium-cluster-density openshift/cluster-network-operator presubmit Registry content changed
pull-ci-openshift-cluster-network-operator-release-4.21-qe-perfscale-aws-ovn-small-cluster-density openshift/cluster-network-operator presubmit Registry content changed
pull-ci-openshift-cluster-network-operator-release-4.20-qe-perfscale-aws-ovn-medium-cluster-density openshift/cluster-network-operator presubmit Registry content changed
pull-ci-openshift-cluster-network-operator-release-4.20-qe-perfscale-aws-ovn-small-cluster-density openshift/cluster-network-operator presubmit Registry content changed
pull-ci-openshift-cluster-network-operator-release-4.19-qe-perfscale-aws-ovn-medium-cluster-density openshift/cluster-network-operator presubmit Registry content changed
pull-ci-openshift-cluster-network-operator-release-4.19-qe-perfscale-aws-ovn-small-cluster-density openshift/cluster-network-operator presubmit Registry content changed
pull-ci-openshift-cluster-network-operator-release-4.18-qe-perfscale-aws-ovn-medium-cluster-density openshift/cluster-network-operator presubmit Registry content changed
pull-ci-openshift-cluster-network-operator-release-4.18-qe-perfscale-aws-ovn-small-cluster-density openshift/cluster-network-operator presubmit Registry content changed
pull-ci-openshift-cluster-network-operator-release-4.17-qe-perfscale-aws-ovn-medium-cluster-density openshift/cluster-network-operator presubmit Registry content changed
pull-ci-openshift-cluster-network-operator-release-4.17-qe-perfscale-aws-ovn-small-cluster-density openshift/cluster-network-operator presubmit Registry content changed
pull-ci-openshift-cluster-network-operator-release-4.16-qe-perfscale-aws-ovn-medium-cluster-density openshift/cluster-network-operator presubmit Registry content changed
pull-ci-openshift-cluster-network-operator-release-4.16-qe-perfscale-aws-ovn-small-cluster-density openshift/cluster-network-operator presubmit Registry content changed
pull-ci-openshift-cluster-network-operator-release-4.15-qe-perfscale-aws-ovn-medium-cluster-density openshift/cluster-network-operator presubmit Registry content changed
pull-ci-openshift-cluster-network-operator-release-4.15-qe-perfscale-aws-ovn-small-cluster-density openshift/cluster-network-operator presubmit Registry content changed

A total of 276 jobs have been affected by this change. The above listing is non-exhaustive and limited to 25 jobs.

A full list of affected jobs can be found here

Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In
`@ci-operator/config/stackrox/stackrox/stackrox-stackrox-jv-test-perf-scale__perf-scale.yaml`:
- Around line 66-70: The zz_generated_metadata block contains a temporary
personal branch (branch: jv-test-perf-scale) and should not be merged as-is;
either remove this temporary test config file entirely or update the
zz_generated_metadata (branch, org, repo, variant) to point to the correct
production values (or a tracked canonical branch) and add a PR comment
confirming the change before unblocking the PR; specifically edit the
zz_generated_metadata entry in
stackrox-stackrox-jv-test-perf-scale__perf-scale.yaml to remove or replace the
branch: jv-test-perf-scale value and ensure the file is intended for mainline
config before merge.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 2f15cffa-9d9f-4a3c-ae00-7527ceaca646

📥 Commits

Reviewing files that changed from the base of the PR and between bf165c9 and c5aee7d.

⛔ Files ignored due to path filters (1)
  • ci-operator/jobs/stackrox/stackrox/stackrox-stackrox-jv-test-perf-scale-presubmits.yaml is excluded by !ci-operator/jobs/**
📒 Files selected for processing (1)
  • ci-operator/config/stackrox/stackrox/stackrox-stackrox-jv-test-perf-scale__perf-scale.yaml

Comment on lines +66 to +70
zz_generated_metadata:
branch: jv-test-perf-scale
org: stackrox
repo: stackrox
variant: perf-scale
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Confirm this temporary test config is removed before merge.

The metadata targets a personal branch (jv-test-perf-scale) and the commit message states this job was "Temporarily adding a job to test changes". This file looks like a scratch artifact for exercising the Orion baseline refactor, not a production config destined for main. The PR is already WIP//hold, so this is just a reminder to drop it (or repoint to a real branch) before lifting the hold.

Want me to open a tracking issue so this temporary config isn't accidentally merged?

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@ci-operator/config/stackrox/stackrox/stackrox-stackrox-jv-test-perf-scale__perf-scale.yaml`
around lines 66 - 70, The zz_generated_metadata block contains a temporary
personal branch (branch: jv-test-perf-scale) and should not be merged as-is;
either remove this temporary test config file entirely or update the
zz_generated_metadata (branch, org, repo, variant) to point to the correct
production values (or a tracked canonical branch) and add a PR comment
confirming the change before unblocking the PR; specifically edit the
zz_generated_metadata entry in
stackrox-stackrox-jv-test-perf-scale__perf-scale.yaml to remove or replace the
branch: jv-test-perf-scale value and ensure the file is intended for mainline
config before merge.

@JoukoVirtanen
Copy link
Copy Markdown
Contributor Author

/test pull-ci-stackrox-stackrox-jv-test-perf-scale-perf-scale-24nodes-scale-test

@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci Bot commented May 29, 2026

@JoukoVirtanen: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

/test boskos-config
/test boskos-config-generation
/test check-gh-automation
/test check-gh-automation-tide
/test check-trigger-trusted-apps
/test ci-operator-config
/test ci-operator-config-metadata
/test ci-operator-registry
/test ci-secret-bootstrap-config-validation
/test ci-testgrid-allow-list
/test clusterimageset-validate
/test config
/test core-valid
/test generated-config
/test generated-dashboards
/test hyperfleet-risk-scorer-test
/test image-mirroring-config-validation
/test jira-lifecycle-config
/test labels
/test openshift-image-mirror-mappings
/test ordered-prow-config
/test owners
/test pr-reminder-config
/test prow-config
/test prow-config-filenames
/test prow-config-semantics
/test pylint
/test release-config
/test release-controller-config
/test rover-groups-config-validation
/test secret-generator-config-valid
/test services-valid
/test stackrox-stackrox-stackrox-stackrox-check
/test step-registry-metadata
/test step-registry-shellcheck
/test sync-rover-groups
/test verified-config
/test yamllint

The following commands are available to trigger optional jobs:

/test check-cluster-profiles-config

Use /test all to run the following jobs that were automatically triggered:

pull-ci-openshift-release-check-gh-automation
pull-ci-openshift-release-main-ci-operator-config
pull-ci-openshift-release-main-ci-operator-config-metadata
pull-ci-openshift-release-main-ci-operator-registry
pull-ci-openshift-release-main-config
pull-ci-openshift-release-main-core-valid
pull-ci-openshift-release-main-generated-config
pull-ci-openshift-release-main-ordered-prow-config
pull-ci-openshift-release-main-owners
pull-ci-openshift-release-main-prow-config-filenames
pull-ci-openshift-release-main-prow-config-semantics
pull-ci-openshift-release-main-release-controller-config
pull-ci-openshift-release-main-stackrox-stackrox-stackrox-stackrox-check
pull-ci-openshift-release-main-step-registry-metadata
pull-ci-openshift-release-main-step-registry-shellcheck
pull-ci-openshift-release-openshift-image-mirror-mappings
pull-ci-openshift-release-yamllint
Details

In response to this:

/test pull-ci-stackrox-stackrox-jv-test-perf-scale-perf-scale-24nodes-scale-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@JoukoVirtanen
Copy link
Copy Markdown
Contributor Author

/pj-rehearse pull-ci-stackrox-stackrox-jv-test-perf-scale-perf-scale-24nodes-scale-test

@openshift-merge-bot
Copy link
Copy Markdown
Contributor

@JoukoVirtanen: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@JoukoVirtanen
Copy link
Copy Markdown
Contributor Author

/pj-rehearse pull-ci-stackrox-stackrox-jv-test-perf-scale-perf-scale-24nodes-scale-test

@openshift-merge-bot
Copy link
Copy Markdown
Contributor

@JoukoVirtanen: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@JoukoVirtanen
Copy link
Copy Markdown
Contributor Author

/pj-rehearse pull-ci-stackrox-stackrox-jv-test-perf-scale-perf-scale-24nodes-scale-test

@openshift-merge-bot
Copy link
Copy Markdown
Contributor

@JoukoVirtanen: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci Bot commented May 30, 2026

@JoukoVirtanen: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant