Skip to content

SPLAT-2668: debug removing skipped CCM tests on hypershift#79439

Draft
mtulio wants to merge 1 commit into
openshift:mainfrom
mtulio:SPLAT-2668-validate-OCPBUGS-85414
Draft

SPLAT-2668: debug removing skipped CCM tests on hypershift#79439
mtulio wants to merge 1 commit into
openshift:mainfrom
mtulio:SPLAT-2668-validate-OCPBUGS-85414

Conversation

@mtulio
Copy link
Copy Markdown
Contributor

@mtulio mtulio commented May 18, 2026

This PR updates Hypershift CI configuration in the openshift/release repository to debug removal of skipped Cloud Controller Manager (CCM) tests on Hypershift (refs SPLAT-2668 / OCPBUGS-85414).

What changed (practical terms)

  • Release-5.0 job configuration: In ci-operator/config/openshift/hypershift/openshift-hypershift-release-5.0.yaml the e2e-aws-ovn-conformance-ccm-techpreview job’s TEST_SKIPS was reformatted and the skip that matched the NLB test "NLB should be reachable with target-node-labels|loadbalancer" was removed while the skip for the NLB hairpinning check remains. Practically this enables the previously-skipped "NLB should be reachable with target-node-labels" test for the techpreview CCM conformance run on Hypershift.
  • Main CI job definitions: ci-operator/config/openshift/hypershift/openshift-hypershift-main.yaml gained two optional job entries — e2e-aws-ovn-conformance-ccm and e2e-aws-ovn-conformance-ccm-techpreview — that target the hypershift-aws cluster_profile, set PUBLIC_ONLY: "true" and leave TEST_SKIPS empty; the techpreview variant sets GUEST_FEATURE_SET: TechPreviewNoUpgrade. These additions make the CCM conformance variants available to job generation (optional jobs that can be rehearsed/run).
  • Release-5.0 periodics: ci-operator/config/openshift/hypershift/openshift-hypershift-release-5.0__periodics.yaml adds two periodic jobs:
    • e2e-aws-ovn-conformance-ccm scheduled @monthly with TEST_SKIPS: ""
    • e2e-aws-ovn-conformance-ccm-techpreview scheduled @daily with GUEST_FEATURE_SET: TechPreviewNoUpgrade and TEST_SKIPS: ""
      These schedule the CCM conformance variants for automatic runs on the 5.0 release branch.
  • 4.23 periodics: ci-operator/config/openshift/hypershift/openshift-hypershift-release-4.23__periodics.yaml clears TEST_SKIPS for the e2e-aws-ovn-conformance-ccm job and changes the techpreview job’s schedule from @monthly to @daily and clears its TEST_SKIPS as well — increasing frequency and ensuring the CCM variants exercise previously-skipped tests on 4.23.

Impact

  • The techpreview CCM conformance job on Hypershift will now run the previously-skipped NLB target-node-labels test, supporting debugging and validation of CCM-related test skipping issues.
  • Adding optional job entries in main plus the periodics ensures these CCM conformance variants are present in job generation and will run on the configured schedules for both 5.0 and 4.23 branches.

Other notes

  • The PR references JIRA SPLAT-2668; openshift-ci-robot confirmed the JIRA reference but flagged the JIRA as missing a target version for the target branch (it expected "5.0").
  • The author (mtulio) manually triggered rehearsals and issued Prow commands (/test all and /ph-rehearse ...) and linked to rehearsal job logs; one prior rehearse run failed due to an outdated image per their comment.

@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label May 18, 2026
@openshift-ci openshift-ci Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 18, 2026
@openshift-ci-robot
Copy link
Copy Markdown
Contributor

openshift-ci-robot commented May 18, 2026

@mtulio: This pull request references SPLAT-2668 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "5.0.0" version, but no target version was set.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci Bot commented May 18, 2026

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@mtulio
Copy link
Copy Markdown
Contributor Author

mtulio commented May 18, 2026

/test all

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented May 18, 2026

Walkthrough

Adds two AWS OVN conformance CCM e2e jobs (regular and techpreview) to hypershift CI and periodics, clears/condenses TEST_SKIPS for those jobs across 4.23 and 5.0 configs, and schedules the techpreview periodic as daily for 5.0.

Changes

Hypershift CI jobs and periodics

Layer / File(s) Summary
Add CI job entries
ci-operator/config/openshift/hypershift/openshift-hypershift-main.yaml
Insert e2e-aws-ovn-conformance-ccm and e2e-aws-ovn-conformance-ccm-techpreview into main CI config with PUBLIC_ONLY: "true", TEST_SKIPS: ""; techpreview sets GUEST_FEATURE_SET: TechPreviewNoUpgrade.
Add 5.0 periodics
ci-operator/config/openshift/hypershift/openshift-hypershift-release-5.0__periodics.yaml
Add monthly (@monthly) periodic for e2e-aws-ovn-conformance-ccm and daily (@daily) periodic for e2e-aws-ovn-conformance-ccm-techpreview, both on hypershift-aws with PUBLIC_ONLY: "true" and TEST_SKIPS: ""; techpreview sets GUEST_FEATURE_SET.
Condense 5.0 techpreview TEST_SKIPS
ci-operator/config/openshift/hypershift/openshift-hypershift-release-5.0.yaml
Rewrite the TEST_SKIPS for e2e-aws-ovn-conformance-ccm-techpreview to a single-line value and remove the `target-node-labels
Update 4.23 periodics
ci-operator/config/openshift/hypershift/openshift-hypershift-release-4.23__periodics.yaml
Clear TEST_SKIPS for e2e-aws-ovn-conformance-ccm and change e2e-aws-ovn-conformance-ccm-techpreview schedule from @monthly to @daily with TEST_SKIPS: "".

Sequence Diagram(s)

sequenceDiagram
  participant Developer
  participant CI-Operator
  participant ci_config_main as "openshift-hypershift-main.yaml"
  participant periodics_5_0 as "openshift-hypershift-release-5.0__periodics.yaml"
  participant periodics_4_23 as "openshift-hypershift-release-4.23__periodics.yaml"
  Developer->>CI-Operator: add/modify job definitions and periodics
  CI-Operator->>ci_config_main: insert new e2e-aws-ovn-conformance-ccm jobs (PUBLIC_ONLY, TEST_SKIPS, GUEST_FEATURE_SET)
  CI-Operator->>periodics_5_0: register `@monthly` and `@daily` periodics for new jobs
  CI-Operator->>periodics_4_23: update schedules and clear TEST_SKIPS
Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related PRs

  • openshift/release#79505: Updates the same e2e-aws-ovn-conformance-ccm* job definitions to use reporter_config for reporting.

Suggested labels

lgtm, approved, rehearsals-ack

Suggested reviewers

  • sosiouxme
  • xueqzhan
🚥 Pre-merge checks | ✅ 12
✅ Passed checks (12 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title references JIRA issue SPLAT-2668 and mentions 'debug removing skipped CCM tests on hypershift', which directly aligns with the changeset that modifies TEST_SKIPS configurations and adds/removes test skip conditions across multiple hypershift configuration files.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed PR modifies CI configuration files (YAML), not Ginkgo test code. The custom check applies only to test titles in It()/Describe()/Context() calls; no such code is present in these files.
Test Structure And Quality ✅ Passed PR contains only CI/CD configuration YAML files, not Ginkgo test code. The custom check requires reviewing Ginkgo test structures, making it not applicable to this PR.
Microshift Test Compatibility ✅ Passed PR contains only CI configuration changes (YAML files) that configure which test jobs to run. No new Ginkgo e2e test code (It(), Describe(), etc.) is being added. Check does not apply.
Single Node Openshift (Sno) Test Compatibility ✅ Passed PR only modifies CI configuration YAML files. No new Ginkgo test code was added, so the SNO compatibility check does not apply.
Topology-Aware Scheduling Compatibility ✅ Passed PR only modifies CI operator test job configurations, not deployment manifests or operator code. No pod scheduling constraints, affinity rules, or topology-aware resource definitions are introduced.
Ote Binary Stdout Contract ✅ Passed The OTE Binary Stdout Contract check applies to source code (main functions, logging, I/O). This PR modifies only YAML CI configuration files, not executable code.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed PR modifies CI configuration files (YAML), not Ginkgo e2e test code. Custom check applies only when new tests with It(), Describe(), etc. are added, which is not the case here.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands and usage tips.

@mtulio
Copy link
Copy Markdown
Contributor Author

mtulio commented May 18, 2026

/ph-rehearse periodic-ci-openshift-hypershift-release-5.0-e2e-aws-ovn-conformance-ccm-techpreview

1 similar comment
@mtulio
Copy link
Copy Markdown
Contributor Author

mtulio commented May 18, 2026

/ph-rehearse periodic-ci-openshift-hypershift-release-5.0-e2e-aws-ovn-conformance-ccm-techpreview

@mtulio
Copy link
Copy Markdown
Contributor Author

mtulio commented May 18, 2026

@mtulio
Copy link
Copy Markdown
Contributor Author

mtulio commented May 19, 2026

@mtulio mtulio force-pushed the SPLAT-2668-validate-OCPBUGS-85414 branch from b879e72 to 501cf43 Compare May 21, 2026 01:57
@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci Bot commented May 21, 2026

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: mtulio
Once this PR has been reviewed and has the lgtm label, please assign muraee for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In
`@ci-operator/config/openshift/hypershift/openshift-hypershift-release-5.0__periodics.yaml`:
- Around line 159-177: The periodic job entries for "as:
e2e-aws-ovn-conformance-ccm" and "as: e2e-aws-ovn-conformance-ccm-techpreview"
are missing the required mirrored sections from the main config; copy the full
set of variant sections (base_images, build_root, images, promotion, and
releases) from the main config into each of these periodic job blocks so their
structure matches the main config exactly, ensuring you place the copied
build_root, images, promotion, base_images and releases under the same
steps/job-level scope as the other fields for those two jobs.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 4c60cda5-c9dd-46a2-a881-d997f083c3b3

📥 Commits

Reviewing files that changed from the base of the PR and between b879e72 and 501cf43.

⛔ Files ignored due to path filters (2)
  • ci-operator/jobs/openshift/hypershift/openshift-hypershift-main-presubmits.yaml is excluded by !ci-operator/jobs/**
  • ci-operator/jobs/openshift/hypershift/openshift-hypershift-release-5.0-periodics.yaml is excluded by !ci-operator/jobs/**
📒 Files selected for processing (3)
  • ci-operator/config/openshift/hypershift/openshift-hypershift-main.yaml
  • ci-operator/config/openshift/hypershift/openshift-hypershift-release-5.0.yaml
  • ci-operator/config/openshift/hypershift/openshift-hypershift-release-5.0__periodics.yaml
💤 Files with no reviewable changes (1)
  • ci-operator/config/openshift/hypershift/openshift-hypershift-release-5.0.yaml

@mtulio mtulio force-pushed the SPLAT-2668-validate-OCPBUGS-85414 branch from 501cf43 to 1ad913a Compare May 21, 2026 04:07
@openshift-ci openshift-ci Bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 21, 2026
@openshift-merge-bot
Copy link
Copy Markdown
Contributor

@mtulio, pj-rehearse: unable to determine affected jobs. This could be due to a branch that needs to be rebased. ERROR:

couldn't prepare candidate: couldn't rebase candidate onto 881070c21e9d0cc1fba5b736c5ca88be51d186bf due to conflicts
Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

Review CCM tests by keeping only main presubmit, and adding 5.0
periodics for jobs required to ensure CCM e2e readines on Hypershift.

Removing TEST_SKIPS which is manually tested and working in local dev
env in HCP clusters:
openshift/cluster-cloud-controller-manager-operator#464 (comment)
@mtulio mtulio force-pushed the SPLAT-2668-validate-OCPBUGS-85414 branch from 1ad913a to 64535e9 Compare May 21, 2026 04:13
@mtulio
Copy link
Copy Markdown
Contributor Author

mtulio commented May 21, 2026

/test all

@openshift-ci openshift-ci Bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 21, 2026
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

♻️ Duplicate comments (1)
ci-operator/config/openshift/hypershift/openshift-hypershift-release-5.0__periodics.yaml (1)

159-175: 🛠️ Refactor suggestion | 🟠 Major

Periodic variant is missing required mirrored sections from main config.

This __periodics.yaml file still lacks build_root, images, and promotion sections at the file level. While this is a pre-existing structural issue (not introduced by the current PR), it should be addressed to comply with variant file structure requirements.

As per coding guidelines: "Variant periodic configuration files should include base_images, build_root, images, promotion, and releases sections copied from main config, then add only tests: entries with interval: or cron: scheduling"

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@ci-operator/config/openshift/hypershift/openshift-hypershift-release-5.0__periodics.yaml`
around lines 159 - 175, This periodic variant is missing required top-level
sections copied from the main config; add the base_images, build_root, images,
promotion, and releases sections from the main config into this variant file and
keep only tests entries (like the existing e2e-aws-ovn-conformance-ccm and
e2e-aws-ovn-conformance-ccm-techpreview blocks) with their cron/interval
scheduling; ensure the inserted sections match the main config exactly
(including any base_images keys and build_root settings) so the variant
structure follows the guideline that only tests are varied while
build/images/promotion/releases remain identical.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Duplicate comments:
In
`@ci-operator/config/openshift/hypershift/openshift-hypershift-release-5.0__periodics.yaml`:
- Around line 159-175: This periodic variant is missing required top-level
sections copied from the main config; add the base_images, build_root, images,
promotion, and releases sections from the main config into this variant file and
keep only tests entries (like the existing e2e-aws-ovn-conformance-ccm and
e2e-aws-ovn-conformance-ccm-techpreview blocks) with their cron/interval
scheduling; ensure the inserted sections match the main config exactly
(including any base_images keys and build_root settings) so the variant
structure follows the guideline that only tests are varied while
build/images/promotion/releases remain identical.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 3c47be1d-c435-4d22-a084-1aa181d0879d

📥 Commits

Reviewing files that changed from the base of the PR and between 501cf43 and 1ad913a.

⛔ Files ignored due to path filters (3)
  • ci-operator/jobs/openshift/hypershift/openshift-hypershift-main-presubmits.yaml is excluded by !ci-operator/jobs/**
  • ci-operator/jobs/openshift/hypershift/openshift-hypershift-release-4.23-periodics.yaml is excluded by !ci-operator/jobs/**
  • ci-operator/jobs/openshift/hypershift/openshift-hypershift-release-5.0-periodics.yaml is excluded by !ci-operator/jobs/**
📒 Files selected for processing (4)
  • ci-operator/config/openshift/hypershift/openshift-hypershift-main.yaml
  • ci-operator/config/openshift/hypershift/openshift-hypershift-release-4.23__periodics.yaml
  • ci-operator/config/openshift/hypershift/openshift-hypershift-release-5.0.yaml
  • ci-operator/config/openshift/hypershift/openshift-hypershift-release-5.0__periodics.yaml
💤 Files with no reviewable changes (1)
  • ci-operator/config/openshift/hypershift/openshift-hypershift-release-5.0.yaml

@openshift-merge-bot
Copy link
Copy Markdown
Contributor

[REHEARSALNOTIFIER]
@mtulio: the pj-rehearse plugin accommodates running rehearsal tests for the changes in this PR. Expand 'Interacting with pj-rehearse' for usage details. The following rehearsable tests have been affected by this change:

Test name Repo Type Reason
pull-ci-openshift-hypershift-main-e2e-aws-ovn-conformance-ccm openshift/hypershift presubmit Presubmit changed
pull-ci-openshift-hypershift-main-e2e-aws-ovn-conformance-ccm-techpreview openshift/hypershift presubmit Presubmit changed
periodic-ci-openshift-hypershift-release-5.0-periodics-e2e-aws-ovn-conformance-ccm N/A periodic Periodic changed
periodic-ci-openshift-hypershift-release-5.0-periodics-e2e-aws-ovn-conformance-ccm-techpreview N/A periodic Periodic changed
periodic-ci-openshift-hypershift-release-4.23-periodics-e2e-aws-ovn-conformance-ccm-techpreview N/A periodic Ci-operator config changed
periodic-ci-openshift-hypershift-release-4.23-periodics-e2e-aws-ovn-conformance-ccm N/A periodic Ci-operator config changed
Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci Bot commented May 21, 2026

@mtulio: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants