Job for Hypershift with KubeVirt in disconnected env#78382
Conversation
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughAdds a new intranet periodic disconnected conformance job and updates CI step scripts and a workflow to change registry discovery/auth flow, skopeo usage, oc-mirror/oc adm mirroring flags (including a 4.21/4.22 release-mirroring workaround), TLS/HTTP transport overrides, and pre-deploy/configure Konflux. Changes
Sequence Diagram(s)sequenceDiagram
participant CI as CI Job
participant Steps as CI Step Scripts
participant Auth as Pull-Secret/Auth
participant Skopeo as Skopeo
participant OcMirror as oc-mirror / oc adm
participant MirrorReg as Mirror Registry
participant Cluster as Target Cluster
participant Konflux as Konflux Operator
participant MCE as hypershift-mce-install
CI->>Steps: start job
Steps->>Auth: extract pull-secret -> /home/pull-secret
Steps->>Skopeo: skopeo login (mirror) & skopeo login (registry.redhat.io)
Steps->>Skopeo: skopeo copy --authfile /home/pull-secret ...
Steps->>OcMirror: oc-mirror (--v1, --source-use-http, --*_skip-tls, --remove-signatures, --ignore-release-signature)
OcMirror->>MirrorReg: push mirrored images
Steps->>OcMirror: oc adm release mirror (select digest for 4.21/4.22) -> mirror release locally
Steps->>Cluster: apply ImageDigestMirrorSet (redirect quay release -> mirrored image)
CI->>Konflux: pre-deploy Konflux (targets restricted, deploy disabled, registry proxy on 6004)
Konflux->>Cluster: configure operator catalogs/targets
CI->>MCE: run hypershift-mce-install
MCE->>Cluster: install MCE and run conformance
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes 🚥 Pre-merge checks | ✅ 11 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (11 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
|
/pj-rehearse periodic-ci-openshift-hypershift-release-4.22-periodics-mce-e2e-kubevirt-metal-ovn-disconnected |
|
@mgencur: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In
`@ci-operator/step-registry/hypershift/kubevirt/create/disconnected/workarounds/hypershift-kubevirt-create-disconnected-workarounds-commands.sh`:
- Around line 104-134: The ImageContentSourcePolicy apply block (the oc apply -f
- <<END that creates kind: ImageContentSourcePolicy named
mirror-config-capi-specific-release) is unconditionally applied even when
CLUSTER_VERSION isn't 4.21/4.22, causing invalid redirects; gate that oc apply
behind the same CLUSTER_VERSION check used for the oc adm release mirror blocks
(or use a combined conditional for 4.21 OR 4.22) and ensure you emit the mirror
configuration using the IDMS path or re-run the existing ICSP→IDMS
migration/wait sequence after creating it so the policy is migrated consistently
with the earlier workaround.
In
`@ci-operator/step-registry/hypershift/mce/kubevirt/baremetalds/disconnected/conformance/hypershift-mce-kubevirt-baremetalds-disconnected-conformance-workflow.yaml`:
- Around line 25-27: The post step currently lists "wait" first which applies
its defaults (TIMEOUT="+3 hours", PR_ONLY="false") and will delay teardown;
update the post sequence so "wait" is either removed or moved to the end (after
"hypershift-kubevirt-baremetalds-disconnected-highlightmissingimages") or
explicitly override its defaults by adding TIMEOUT and PR_ONLY variables for the
"wait" ref; modify the "post" block containing refs "wait" and
"hypershift-kubevirt-baremetalds-disconnected-highlightmissingimages"
accordingly to ensure teardown is not deferred for three hours on every run.
- Around line 39-40: The pre-step ordering is wrong: move the ref
"deploy-konflux-operator" so it appears after "hypershift-mce-install" in the
workflow refs list; specifically, reorder the two ref entries so that
"hypershift-mce-install" runs first (creating ${SHARED_DIR}/mirror_registry_url
in hypershift-mce-install-disconnected-commands.sh) and then
"deploy-konflux-operator" executes and can read that file.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Enterprise
Run ID: a38d580a-23d6-49ed-8cb3-eda10bb58cb2
⛔ Files ignored due to path filters (1)
ci-operator/jobs/openshift/hypershift/openshift-hypershift-release-4.22-periodics.yamlis excluded by!ci-operator/jobs/**
📒 Files selected for processing (5)
ci-operator/config/openshift/hypershift/openshift-hypershift-release-4.22__periodics-mce.yamlci-operator/step-registry/hypershift/kubevirt/baremetalds/redhat-operators/hypershift-kubevirt-baremetalds-redhat-operators-commands.shci-operator/step-registry/hypershift/kubevirt/create/disconnected/workarounds/hypershift-kubevirt-create-disconnected-workarounds-commands.shci-operator/step-registry/hypershift/mce/install/disconnected/hypershift-mce-install-disconnected-commands.shci-operator/step-registry/hypershift/mce/kubevirt/baremetalds/disconnected/conformance/hypershift-mce-kubevirt-baremetalds-disconnected-conformance-workflow.yaml
| - ref: deploy-konflux-operator | ||
| - ref: hypershift-mce-install |
There was a problem hiding this comment.
Move deploy-konflux-operator after hypershift-mce-install.
In disconnected mode, deploy-konflux-operator reads ${SHARED_DIR}/mirror_registry_url, but this workflow only creates that file inside hypershift-mce-install-disconnected-commands.sh at the very end. With the current order, the new pre-step can fail before it can resolve the internal registry.
Suggested fix
- - ref: deploy-konflux-operator
- - ref: hypershift-mce-install
+ - ref: hypershift-mce-install
+ - ref: deploy-konflux-operator📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| - ref: deploy-konflux-operator | |
| - ref: hypershift-mce-install | |
| - ref: hypershift-mce-install | |
| - ref: deploy-konflux-operator |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In
`@ci-operator/step-registry/hypershift/mce/kubevirt/baremetalds/disconnected/conformance/hypershift-mce-kubevirt-baremetalds-disconnected-conformance-workflow.yaml`
around lines 39 - 40, The pre-step ordering is wrong: move the ref
"deploy-konflux-operator" so it appears after "hypershift-mce-install" in the
workflow refs list; specifically, reorder the two ref entries so that
"hypershift-mce-install" runs first (creating ${SHARED_DIR}/mirror_registry_url
in hypershift-mce-install-disconnected-commands.sh) and then
"deploy-konflux-operator" executes and can read that file.
There was a problem hiding this comment.
I don't use hypershift-mce-install-disconnected step. The mirror_registry_url is set in step enable-qe-catalogsource-baremetalds-proxy-server which runs before deploy-konflux-operator.
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Find enable-qe-catalogsource-baremetalds-proxy-server step script
fd -i 'enable-qe-catalogsource-baremetalds-proxy-server' --type f
# Find deploy-konflux-operator step script
fd -i 'deploy-konflux-operator' --type f
Length of output: 672
🏁 Script executed:
#!/bin/bash
echo "=== enable-qe-catalogsource-baremetalds-proxy-server-commands.sh ==="
cat ci-operator/step-registry/enable-qe-catalogsource/baremetalds/proxy-server/enable-qe-catalogsource-baremetalds-proxy-server-commands.sh
echo ""
echo "=== deploy-konflux-operator-commands.sh ==="
cat ci-operator/step-registry/deploy-konflux-operator/deploy-konflux-operator-commands.sh
Length of output: 8525
@mgencur, you're absolutely right — I apologize for the incorrect review. The script confirms it: enable-qe-catalogsource-baremetalds-proxy-server-commands.sh writes ${SHARED_DIR}/mirror_registry_url (via oc get imagecontentsourcepolicy) and runs before deploy-konflux-operator, so the dependency is already satisfied by the time deploy-konflux-operator reads that file. The current step ordering is correct.
📝 Learnings were identified and not saved (knowledge base disabled). Enable
|
/pj-rehearse periodic-ci-openshift-hypershift-release-4.22-periodics-mce-e2e-kubevirt-metal-ovn-disconnected |
|
@mgencur: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse periodic-ci-openshift-hypershift-release-4.22-periodics-mce-e2e-kubevirt-metal-ovn-disconnected |
|
@mgencur: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
There was a problem hiding this comment.
Actionable comments posted: 1
♻️ Duplicate comments (1)
ci-operator/step-registry/hypershift/mce/kubevirt/baremetalds/disconnected/conformance/hypershift-mce-kubevirt-baremetalds-disconnected-conformance-workflow.yaml (1)
26-26:⚠️ Potential issue | 🟠 MajorRemove or gate the
waitpost step before merge.At Line 26,
waitruns with defaults (+3 hours,PR_ONLY=false), so every periodic run is delayed before teardown/gather steps.Suggested change
post: - - ref: wait - ref: hypershift-kubevirt-baremetalds-disconnected-highlightmissingimages - ref: hypershift-mce-dump - chain: gather-core-dump - chain: hypershift-mce-kubevirt-destroy - chain: baremetalds-ofcir-post🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@ci-operator/step-registry/hypershift/mce/kubevirt/baremetalds/disconnected/conformance/hypershift-mce-kubevirt-baremetalds-disconnected-conformance-workflow.yaml` at line 26, The post-step referencing "ref: wait" should be removed or guarded so periodic jobs aren't blocked; change the step that contains "ref: wait" to either delete that post step entirely or add a conditional guard (e.g., check PR_ONLY or run only on pull-request runs) so it only executes for PRs (for example by testing PR_ONLY == "true" or using the workflow's pull-request condition), ensuring teardown/gather steps run immediately for periodic jobs.
🧹 Nitpick comments (1)
ci-operator/step-registry/hypershift/kubevirt/create/disconnected/workarounds/hypershift-kubevirt-create-disconnected-workarounds-commands.sh (1)
108-114: Consider consolidating version checks into a single if/elif block.The separate
ifstatements work but leavemirror_imageuninitialized if the version is neither 4.21 nor 4.22. While the outer guard at line 116 prevents usage in that case, consolidating improves clarity:♻️ Suggested consolidation
-if [[ "${CLUSTER_VERSION}" == "4.22" ]]; then - mirror_image="quay.io/openshift-release-dev/ocp-release@sha256:7f183e9b5610a2c9f9aabfd5906b418adfbe659f441b019933426a19bf6a5962" -fi - -if [[ "${CLUSTER_VERSION}" == "4.21" ]]; then - mirror_image="quay.io/openshift-release-dev/ocp-release@sha256:1f2c28ac126453a3b9e83b349822b9f1fb7662973a212f936b90fdc40e06eb58" -fi - -if [[ "${CLUSTER_VERSION}" == "4.21" || "${CLUSTER_VERSION}" == "4.22" ]]; then +case "${CLUSTER_VERSION}" in + "4.22") + mirror_image="quay.io/openshift-release-dev/ocp-release@sha256:7f183e9b5610a2c9f9aabfd5906b418adfbe659f441b019933426a19bf6a5962" + ;; + "4.21") + mirror_image="quay.io/openshift-release-dev/ocp-release@sha256:1f2c28ac126453a3b9e83b349822b9f1fb7662973a212f936b90fdc40e06eb58" + ;; + *) + mirror_image="" + ;; +esac + +if [[ -n "${mirror_image}" ]]; then🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@ci-operator/step-registry/hypershift/kubevirt/create/disconnected/workarounds/hypershift-kubevirt-create-disconnected-workarounds-commands.sh` around lines 108 - 114, Consolidate the two separate version checks into a single if/elif block so CLUSTER_VERSION is evaluated once and mirror_image is deterministically set for supported versions; replace the two `if [[ "${CLUSTER_VERSION}" == "4.22" ]]` and `if [[ "${CLUSTER_VERSION}" == "4.21" ]]` blocks with `if [[ "${CLUSTER_VERSION}" == "4.22" ]]; then ... elif [[ "${CLUSTER_VERSION}" == "4.21" ]]; then ... fi` (reference: CLUSTER_VERSION and mirror_image) and optionally add an `else` to explicitly unset or log an error for unsupported versions.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In
`@ci-operator/step-registry/hypershift/kubevirt/create/disconnected/workarounds/hypershift-kubevirt-create-disconnected-workarounds-commands.sh`:
- Around line 62-65: Remove the unsupported --ignore-release-signature flag from
the two oc-mirror invocations that currently include it (the lines invoking
./oc-mirror with file://"${LOCALPATH}" and the second with
docker://${mirror_registry}); keep --remove-signatures as-is (or replace/add
--secure-policy if you need explicit signature verification behavior) so the
oc-mirror commands use only supported v2 flags.
---
Duplicate comments:
In
`@ci-operator/step-registry/hypershift/mce/kubevirt/baremetalds/disconnected/conformance/hypershift-mce-kubevirt-baremetalds-disconnected-conformance-workflow.yaml`:
- Line 26: The post-step referencing "ref: wait" should be removed or guarded so
periodic jobs aren't blocked; change the step that contains "ref: wait" to
either delete that post step entirely or add a conditional guard (e.g., check
PR_ONLY or run only on pull-request runs) so it only executes for PRs (for
example by testing PR_ONLY == "true" or using the workflow's pull-request
condition), ensuring teardown/gather steps run immediately for periodic jobs.
---
Nitpick comments:
In
`@ci-operator/step-registry/hypershift/kubevirt/create/disconnected/workarounds/hypershift-kubevirt-create-disconnected-workarounds-commands.sh`:
- Around line 108-114: Consolidate the two separate version checks into a single
if/elif block so CLUSTER_VERSION is evaluated once and mirror_image is
deterministically set for supported versions; replace the two `if [[
"${CLUSTER_VERSION}" == "4.22" ]]` and `if [[ "${CLUSTER_VERSION}" == "4.21" ]]`
blocks with `if [[ "${CLUSTER_VERSION}" == "4.22" ]]; then ... elif [[
"${CLUSTER_VERSION}" == "4.21" ]]; then ... fi` (reference: CLUSTER_VERSION and
mirror_image) and optionally add an `else` to explicitly unset or log an error
for unsupported versions.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Enterprise
Run ID: 97fb1c72-d19e-4ecb-92b6-0c560ba68de3
📒 Files selected for processing (2)
ci-operator/step-registry/hypershift/kubevirt/create/disconnected/workarounds/hypershift-kubevirt-create-disconnected-workarounds-commands.shci-operator/step-registry/hypershift/mce/kubevirt/baremetalds/disconnected/conformance/hypershift-mce-kubevirt-baremetalds-disconnected-conformance-workflow.yaml
|
/pj-rehearse periodic-ci-openshift-hypershift-release-4.22-periodics-mce-e2e-kubevirt-metal-ovn-disconnected |
|
@mgencur: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse periodic-ci-openshift-hypershift-release-4.22-periodics-mce-e2e-kubevirt-metal-ovn-disconnected |
|
@mgencur: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse periodic-ci-openshift-hypershift-release-4.22-periodics-mce-e2e-kubevirt-metal-ovn-disconnected |
|
@mgencur: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse periodic-ci-openshift-hypershift-release-4.22-periodics-mce-e2e-kubevirt-metal-ovn-disconnected |
|
@mgencur: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse periodic-ci-openshift-hypershift-release-4.22-periodics-mce-e2e-kubevirt-metal-ovn-disconnected |
|
@mgencur: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse periodic-ci-openshift-hypershift-release-4.22-periodics-mce-e2e-kubevirt-metal-ovn-disconnected |
|
@mgencur: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse periodic-ci-openshift-hypershift-release-4.22-periodics-mce-e2e-kubevirt-metal-ovn-disconnected |
|
@mgencur: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/uncc |
|
/pj-rehearse periodic-ci-openshift-hypershift-release-4.22-periodics-mce-e2e-kubevirt-metal-ovn-disconnected |
|
@mgencur: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse periodic-ci-openshift-hypershift-release-4.22-periodics-mce-e2e-kubevirt-metal-ovn-disconnected |
|
@mgencur: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse periodic-ci-openshift-hypershift-release-4.22-periodics-mce-e2e-kubevirt-metal-ovn-disconnected |
|
@mgencur: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
openshift-ci
Bot
added
the
needs-rebase
Add e2e-kubevirt-metal-ovn-disconnected to the release-4.22 periodics-mce config using the existing MCE disconnected workflow. The workflow is updated to include deploy-konflux-operator for Konflux-based operator deployment in disconnected environments. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
mgencur
force-pushed
the
kubevirt_disconnected
branch
from
0f5148d to
4b69812
Compare
openshift-ci
Bot
removed
the
needs-rebase
|
/pj-rehearse periodic-ci-openshift-hypershift-release-4.22-periodics-mce-e2e-kubevirt-metal-ovn-disconnected |
|
@mgencur: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse periodic-ci-openshift-hypershift-release-4.22-periodics-mce-e2e-kubevirt-metal-ovn-disconnected |
|
@mgencur: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
[REHEARSALNOTIFIER]
A total of 742 jobs have been affected by this change. The above listing is non-exhaustive and limited to 25 jobs. A full list of affected jobs can be found here Interacting with pj-rehearseComment: Once you are satisfied with the results of the rehearsals, comment: |
|
@mgencur: The following test failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
jparrill
left a comment
jparrill
left a comment
There was a problem hiding this comment.
/approve
Dropped a question just to know more.
| /home/oc-mirror --config "/home/imageset-config.yaml" docker://${mirror_registry} --oci-registries-config="/home/registry.conf" --continue-on-error --skip-missing | ||
| for i in 1 2 3; do | ||
| echo "oc-mirror attempt ${i}/3" | ||
| /home/oc-mirror --v1 --config "/home/imageset-config.yaml" docker://${mirror_registry} --oci-registries-config="/home/registry.conf" --continue-on-error --skip-missing || true |
There was a problem hiding this comment.
Where this file is composed from (imageset-config.yaml)?
There was a problem hiding this comment.
It is simply defined on line 112 as this:
cat <<END |tee "/home/imageset-config.yaml"
kind: ImageSetConfiguration
apiVersion: mirror.openshift.io/v1alpha2
storageConfig:
local:
path: mirror
mirror:
operators:
- catalog: "oci:///home/cnv-local-catalog"
targetCatalog: ${catalog_image}
targetTag: "${CNV_PRERELEASE_VERSION}"
packages:
- name: kubevirt-hyperconverged
channels:
- name: ${CNV_CHANNEL}
END
Does it answer your question?
openshift-ci
Bot
added
the
approved
| mirror_registry=$(oc get imagecontentsourcepolicy -o json | jq -r '.items[].spec.repositoryDigestMirrors[0].mirrors[0]') | ||
| mirror_registry="" | ||
| set +e | ||
| for attempt in 1 2 3; do |
There was a problem hiding this comment.
why is this failing? is cluster not ready to read a resource?
There was a problem hiding this comment.
Good question. I thought is was brought from #75258 but it was not. I actually can't find the reason so I think it was done by Claude (as a side effect), when replacing calling the same command multiple times with a loop (in other files). I can put back the original line if you want.
There was a problem hiding this comment.
nah, it's nicer this way, but indeed looks totally useless:)
it's a repeated pattern later on too. It's fine for a test code, but eventually may make sense to clean up, move to oc mirror -v2, etc
There was a problem hiding this comment.
move to oc mirror -v2
Yeah.
|
/lgtm |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: jiezhao16, jparrill, mgencur The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
/pj-rehearse ack |
|
@mgencur: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
openshift-merge-bot
Bot
added
the
rehearsals-ack
|
@openshift-ci[bot]: your |
|
@jiezhao16: your |
* ci(hypershift): add MCE KubeVirt disconnected conformance periodic test Add e2e-kubevirt-metal-ovn-disconnected to the release-4.22 periodics-mce config using the existing MCE disconnected workflow. The workflow is updated to include deploy-konflux-operator for Konflux-based operator deployment in disconnected environments. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Adjust node count and CPU per node --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
5 participants
Related to https://redhat.atlassian.net/browse/OCPBUGS-84113
Summary by CodeRabbit
Tests
Bug Fixes
Chores