Skip to content

OCPBUGS-86879: pre-install skopeo for OCL e2e tests#79990

Open
umohnani8 wants to merge 1 commit into
openshift:mainfrom
umohnani8:mco-fix-ocl-skopeo-main
Open

OCPBUGS-86879: pre-install skopeo for OCL e2e tests#79990
umohnani8 wants to merge 1 commit into
openshift:mainfrom
umohnani8:mco-fix-ocl-skopeo-main

Conversation

@umohnani8
Copy link
Copy Markdown
Contributor

@umohnani8 umohnani8 commented Jun 2, 2026

Skopeo v1.23.0 (released 2026-05-26) requires Go 1.25, but the CI build root image only has Go 1.24. The OCL tests install skopeo from source at runtime via hack/install-skopeo.sh, which always fetches the latest release. This causes all OCL test suites to fail.

Add a src-with-skopeo overlay image that installs skopeo from RHEL repos via dnf at image build time, and use it for the OCL test steps. This avoids the GitHub API dependency and Go version mismatch entirely.

Summary by CodeRabbit

This PR updates the OpenShift CI configuration for the machine-config-operator to fix OCL (on-cluster layering) e2e test failures caused by a skopeo/Go version mismatch and runtime installation from GitHub.

Background: Skopeo v1.23.0 requires Go 1.25 while the CI build root image provides Go 1.24. OCL tests previously built and installed the latest skopeo from source at runtime (hack/install-skopeo.sh), which failed due to the Go mismatch and created a dependency on GitHub releases.

What changed:

  • Adds a new build image variant, src-with-skopeo, produced from the existing src image and installing skopeo via RHEL dnf packages at image build time.
  • Marks src-with-skopeo as excluded from promotion (alongside existing excluded images).
  • Updates two OCL e2e test workflows (e2e-gcp-op-ocl-part1 and e2e-gcp-op-ocl-part2) so their test containers use from: src-with-skopeo instead of from: src, removing the runtime skopeo build and the GitHub/Go-version dependency.

Impact: OCL tests no longer install skopeo from source at runtime, avoiding Go toolchain incompatibilities and reliance on external GitHub releases.

@openshift-ci-robot openshift-ci-robot added jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. labels Jun 2, 2026
@openshift-ci-robot
Copy link
Copy Markdown
Contributor

@umohnani8: This pull request references Jira Issue OCPBUGS-86879, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (5.0.0) matches configured target version for branch (5.0.0)
  • bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Skopeo v1.23.0 (released 2026-05-26) requires Go 1.25, but the CI build root image only has Go 1.24. The OCL tests install skopeo from source at runtime via hack/install-skopeo.sh, which always fetches the latest release. This causes all OCL test suites to fail.

Add a src-with-skopeo overlay image that installs skopeo from RHEL repos via dnf at image build time, and use it for the OCL test steps. This avoids the GitHub API dependency and Go version mismatch entirely.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@umohnani8
Copy link
Copy Markdown
Contributor Author

/pj-rehearse

@openshift-merge-bot
Copy link
Copy Markdown
Contributor

@umohnani8: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented Jun 2, 2026

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 91f8c2c3-e52b-4fa8-b2fa-48e5b5c6c659

📥 Commits

Reviewing files that changed from the base of the PR and between dc789d0 and b368c1b.

📒 Files selected for processing (1)
  • ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-main.yaml
🚧 Files skipped from review as they are similar to previous changes (1)
  • ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-main.yaml

Walkthrough

Adds a CI build image src-with-skopeo (based on src with skopeo installed), excludes it from promotion, and switches two e2e test steps to use this image instead of src.

Changes

src-with-skopeo image and test updates

Layer / File(s) Summary
Build image definition and promotion configuration
ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-main.yaml
Adds src-with-skopeo built from src with skopeo installed and appends it to promotion.to[0].excluded_images.
Update test containers to use src-with-skopeo
ci-operator/config/openshift/machine-config-operator/openshift-machine-config-operator-main.yaml
Updates e2e-gcp-op-ocl-part1 and e2e-gcp-op-ocl-part2 test steps to set their test container from: field to src-with-skopeo.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~8 minutes

Possibly related PRs

  • openshift/release#79933: Related changes affecting the split/adjustment of e2e-gcp-op-ocl jobs and their configuration.

Suggested labels

rehearsals-ack

Suggested reviewers

  • deepsm007
  • danilo-gemoli
🚥 Pre-merge checks | ✅ 15
✅ Passed checks (15 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The PR title clearly describes the main change: pre-installing skopeo for OCL e2e tests as referenced in OCPBUGS-86879.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed PR only modifies CI configuration YAML files; no Ginkgo test code or test names are introduced, so stability requirement is not applicable.
Test Structure And Quality ✅ Passed PR contains only CI configuration YAML changes, no Ginkgo test code is present. The custom check for test structure and quality is not applicable.
Microshift Test Compatibility ✅ Passed This PR only modifies CI configuration (YAML) in the release repository and does not add any new Ginkgo e2e tests. The check is not applicable as it requires new test code to be added.
Single Node Openshift (Sno) Test Compatibility ✅ Passed PR modifies only CI YAML configuration, not Ginkgo e2e test code. No new tests (It(), Describe(), etc.) are added, so SNO test compatibility check does not apply.
Topology-Aware Scheduling Compatibility ✅ Passed This is a CI configuration file that adjusts test image sources for OpenStack CI tests. It does not modify deployment manifests, operator code, or pod scheduling constraints.
Ote Binary Stdout Contract ✅ Passed PR modifies only CI configuration YAML files; no Go code or test code is changed, so OTE Binary Stdout Contract check is not applicable.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed PR only modifies CI configuration (container image selection), not adding new Ginkgo e2e test code. Check is not applicable.
No-Weak-Crypto ✅ Passed PR modifies only YAML CI config to add skopeo image; no weak crypto patterns (MD5, SHA1, DES, RC4, 3DES, Blowfish, ECB) or custom implementations detected.
Container-Privileges ✅ Passed No privileged container settings (privileged, hostPID/Network/IPC, SYS_ADMIN, allowPrivilegeEscalation, or root without justification) found in the YAML CI configuration changes.
No-Sensitive-Data-In-Logs ✅ Passed PR adds skopeo installation via dnf in Dockerfile and updates test configs; no passwords, tokens, API keys, PII, session IDs, hostnames, or sensitive data are exposed in logging.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@openshift-ci openshift-ci Bot requested review from cheesesashimi and djoshy June 2, 2026 15:51
@openshift-ci openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 2, 2026
@openshift-ci-robot
Copy link
Copy Markdown
Contributor

@umohnani8: This pull request references Jira Issue OCPBUGS-86879, which is valid.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (5.0.0) matches configured target version for branch (5.0.0)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
Details

In response to this:

Skopeo v1.23.0 (released 2026-05-26) requires Go 1.25, but the CI build root image only has Go 1.24. The OCL tests install skopeo from source at runtime via hack/install-skopeo.sh, which always fetches the latest release. This causes all OCL test suites to fail.

Add a src-with-skopeo overlay image that installs skopeo from RHEL repos via dnf at image build time, and use it for the OCL test steps. This avoids the GitHub API dependency and Go version mismatch entirely.

Summary by CodeRabbit

This PR updates the CI configuration for the OpenShift machine-config-operator to address a Go version compatibility issue with the skopeo tool used by on-cluster layering (OCL) e2e tests.

Background: Skopeo v1.23.0 (released 2026-05-26) requires Go 1.25, but the CI build root image contains Go 1.24. Previously, OCL tests installed skopeo from source at runtime via a script, which fetched the latest release from GitHub. This caused test failures due to the Go version mismatch and created a dependency on GitHub API availability.

Changes made:

  • Adds a new src-with-skopeo build image that installs skopeo from RHEL repositories using dnf at image build time, avoiding the runtime installation and Go version dependency
  • Excludes src-with-skopeo from promotion (alongside custom-os-image and zacks-toolbox)
  • Updates two OCL test workflows (e2e-gcp-op-ocl-part1 and e2e-gcp-op-ocl-part2) to use the src-with-skopeo image instead of the base src image for their test containers

This approach eliminates the GitHub API dependency and sidesteps the Go version compatibility issue by pre-installing skopeo from stable RHEL packages at image build time.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@dkhater-redhat
Copy link
Copy Markdown
Contributor

/lgtm

@openshift-ci openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Jun 2, 2026
@umohnani8
Copy link
Copy Markdown
Contributor Author

/pj-rehearse cancel

@openshift-merge-bot
Copy link
Copy Markdown
Contributor

@umohnani8: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@umohnani8
Copy link
Copy Markdown
Contributor Author

/pj-rehearse pull-ci-openshift-machine-config-operator-main-e2e-gcp-op-ocl-part1
/pj-rehearse pull-ci-openshift-machine-config-operator-main-e2e-gcp-op-ocl-part2

@openshift-merge-bot
Copy link
Copy Markdown
Contributor

@umohnani8: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@openshift-merge-bot
Copy link
Copy Markdown
Contributor

@umohnani8: job(s): cancel either don't exist or were not found to be affected, and cannot be rehearsed

@openshift-merge-bot
Copy link
Copy Markdown
Contributor

@umohnani8: requesting more than one rehearsal in one comment is not supported. If you would like to rehearse multiple specific jobs, please separate the job names by a space in a single command.

Skopeo v1.23.0 (released 2026-05-26) requires Go 1.25, but the CI
build root image only has Go 1.24. The OCL tests install skopeo from
source at runtime via hack/install-skopeo.sh, which always fetches the
latest release. This causes all OCL test suites to fail.

Add a src-with-skopeo overlay image that installs skopeo from RHEL
repos via dnf at image build time, and use it for the OCL test steps.
This avoids the GitHub API dependency and Go version mismatch entirely.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Urvashi <umohnani@redhat.com>
@umohnani8 umohnani8 force-pushed the mco-fix-ocl-skopeo-main branch from dc789d0 to b368c1b Compare June 2, 2026 17:07
@openshift-ci openshift-ci Bot removed the lgtm Indicates that a PR is ready to be merged. label Jun 2, 2026
@umohnani8
Copy link
Copy Markdown
Contributor Author

/pj-rehearse pull-ci-openshift-machine-config-operator-main-e2e-gcp-op-ocl-part1

@openshift-merge-bot
Copy link
Copy Markdown
Contributor

@umohnani8: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@openshift-merge-bot
Copy link
Copy Markdown
Contributor

[REHEARSALNOTIFIER]
@umohnani8: the pj-rehearse plugin accommodates running rehearsal tests for the changes in this PR. Expand 'Interacting with pj-rehearse' for usage details. The following rehearsable tests have been affected by this change:

Test name Repo Type Reason
pull-ci-openshift-machine-config-operator-main-bootstrap-unit openshift/machine-config-operator presubmit Ci-operator config changed
pull-ci-openshift-machine-config-operator-main-e2e-agent-compact-ipv4 openshift/machine-config-operator presubmit Ci-operator config changed
pull-ci-openshift-machine-config-operator-main-e2e-agent-compact-ipv4-iso-no-registry openshift/machine-config-operator presubmit Ci-operator config changed
pull-ci-openshift-machine-config-operator-main-e2e-aws-disruptive openshift/machine-config-operator presubmit Ci-operator config changed
pull-ci-openshift-machine-config-operator-main-e2e-aws-mco-disruptive openshift/machine-config-operator presubmit Ci-operator config changed
pull-ci-openshift-machine-config-operator-main-e2e-aws-ovn openshift/machine-config-operator presubmit Ci-operator config changed
pull-ci-openshift-machine-config-operator-main-e2e-aws-ovn-fips openshift/machine-config-operator presubmit Ci-operator config changed
pull-ci-openshift-machine-config-operator-main-e2e-aws-ovn-fips-op openshift/machine-config-operator presubmit Ci-operator config changed
pull-ci-openshift-machine-config-operator-main-e2e-aws-ovn-ocb-techpreview openshift/machine-config-operator presubmit Ci-operator config changed
pull-ci-openshift-machine-config-operator-main-e2e-aws-ovn-serial-ipsec openshift/machine-config-operator presubmit Ci-operator config changed
pull-ci-openshift-machine-config-operator-main-e2e-aws-ovn-upgrade openshift/machine-config-operator presubmit Ci-operator config changed
pull-ci-openshift-machine-config-operator-main-e2e-aws-ovn-upgrade-ipsec openshift/machine-config-operator presubmit Ci-operator config changed
pull-ci-openshift-machine-config-operator-main-e2e-aws-ovn-upgrade-ocb-techpreview openshift/machine-config-operator presubmit Ci-operator config changed
pull-ci-openshift-machine-config-operator-main-e2e-aws-ovn-upgrade-out-of-change openshift/machine-config-operator presubmit Ci-operator config changed
pull-ci-openshift-machine-config-operator-main-e2e-aws-ovn-windows openshift/machine-config-operator presubmit Ci-operator config changed
pull-ci-openshift-machine-config-operator-main-e2e-aws-ovn-workers-rhel8 openshift/machine-config-operator presubmit Ci-operator config changed
pull-ci-openshift-machine-config-operator-main-e2e-aws-proxy openshift/machine-config-operator presubmit Ci-operator config changed
pull-ci-openshift-machine-config-operator-main-e2e-aws-serial openshift/machine-config-operator presubmit Ci-operator config changed
pull-ci-openshift-machine-config-operator-main-e2e-aws-single-node openshift/machine-config-operator presubmit Ci-operator config changed
pull-ci-openshift-machine-config-operator-main-e2e-aws-upgrade-single-node openshift/machine-config-operator presubmit Ci-operator config changed
pull-ci-openshift-machine-config-operator-main-e2e-aws-workers-rhel8 openshift/machine-config-operator presubmit Ci-operator config changed
pull-ci-openshift-machine-config-operator-main-e2e-azure openshift/machine-config-operator presubmit Ci-operator config changed
pull-ci-openshift-machine-config-operator-main-e2e-azure-ovn-multidisk-techpreview openshift/machine-config-operator presubmit Ci-operator config changed
pull-ci-openshift-machine-config-operator-main-e2e-azure-ovn-upgrade openshift/machine-config-operator presubmit Ci-operator config changed
pull-ci-openshift-machine-config-operator-main-e2e-azure-ovn-upgrade-out-of-change openshift/machine-config-operator presubmit Ci-operator config changed

A total of 68 jobs have been affected by this change. The above listing is non-exhaustive and limited to 25 jobs.

A full list of affected jobs can be found here

Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

@umohnani8
Copy link
Copy Markdown
Contributor Author

/pj-rehearse pull-ci-openshift-machine-config-operator-main-e2e-gcp-op-ocl-part2

@openshift-merge-bot
Copy link
Copy Markdown
Contributor

@umohnani8: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci Bot commented Jun 2, 2026

@umohnani8: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/rehearse/openshift/machine-config-operator/main/e2e-agent-compact-ipv4-iso-no-registry dc789d0 link unknown /pj-rehearse pull-ci-openshift-machine-config-operator-main-e2e-agent-compact-ipv4-iso-no-registry
ci/rehearse/openshift/machine-config-operator/main/e2e-aws-mco-disruptive dc789d0 link unknown /pj-rehearse pull-ci-openshift-machine-config-operator-main-e2e-aws-mco-disruptive
ci/rehearse/openshift/machine-config-operator/main/e2e-aws-disruptive dc789d0 link unknown /pj-rehearse pull-ci-openshift-machine-config-operator-main-e2e-aws-disruptive
ci/rehearse/openshift/machine-config-operator/main/e2e-agent-compact-ipv4 dc789d0 link unknown /pj-rehearse pull-ci-openshift-machine-config-operator-main-e2e-agent-compact-ipv4

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@eggfoobar
Copy link
Copy Markdown
Contributor

/lgtm

@openshift-ci openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Jun 3, 2026
@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci Bot commented Jun 3, 2026

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dkhater-redhat, eggfoobar, umohnani8

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@eggfoobar
Copy link
Copy Markdown
Contributor

@umohnani8 Feel free to /pj-rehearse ack if you're happy with this change, from the successful runs this looks good to me.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants