feat(quay): add OMR v2 to v3 migration periodic#82188
Conversation
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughAdds a scheduled AWS disconnected CI workflow for Quay OMR v2-to-v3 upgrades, including dedicated host provisioning, OMR v2 installation and validation, OMR v3 migration, and rootless-host-aware mirroring and diagnostics. ChangesOMR v2-to-v3 disconnected upgrade
Estimated code review effort: 5 (Critical) | ~120 minutes Sequence Diagram(s)sequenceDiagram
participant CIWorkflow
participant AWS
participant OMRv2Host
participant OpenShift
participant OMRv3Migrator
CIWorkflow->>AWS: provision disconnected test environment
AWS-->>CIWorkflow: publish host connection artifacts
CIWorkflow->>OMRv2Host: install OMR v2 mirror registry
OMRv2Host-->>OpenShift: provide registry and mirror metadata
OpenShift-->>CIWorkflow: validate cluster, IDMS, and mirrored CLI
CIWorkflow->>OMRv3Migrator: run v2-to-v3 migration
OMRv3Migrator->>OMRv2Host: migrate content and remove v2 services
OMRv2Host-->>CIWorkflow: return migration and health status
Possibly related PRs
Suggested labels: Suggested reviewers: Important Pre-merge checks failedPlease resolve all errors before merging. Addressing warnings is optional. ❌ Failed checks (1 error, 2 warnings)
✅ Passed checks (12 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
openshift-ci
Bot
added
the
approved
|
/pj-rehearse periodic-ci-quay-quay-master-omr-v2-upgrade-v3-disconnected-install-upgrade |
|
@jbpratt: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse max |
|
@jbpratt: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (1)
ci-operator/step-registry/quay/omr-v2/host-provision/quay-omr-v2-host-provision-commands.sh (1)
152-165: 🔒 Security & Privacy | 🔵 Trivial | 💤 Low valueSSH ingress open to 0.0.0.0/0.
The security group allows tcp/22 from anywhere, unlike port 8443 which is correctly scoped to
VpcCidr. Access still requires the cluster-profile SSH key, but if Prow's egress IP ranges are known/stable, scoping SSH to a known CIDR would reduce the exposed attack surface for this 12-hour-lived host.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@ci-operator/step-registry/quay/omr-v2/host-provision/quay-omr-v2-host-provision-commands.sh` around lines 152 - 165, Update the OMRSecurityGroup SSH ingress rule to restrict tcp/22 from 0.0.0.0/0 to the approved stable Prow egress CIDR or CIDRs, reusing the existing configuration parameter or pattern for those ranges where available; leave the port 8443 rule scoped to VpcCidr.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@ci-operator/step-registry/quay/omr-v3/upgrade-from-v2/quay-omr-v3-upgrade-from-v2-commands.sh`:
- Around line 99-103: Remove the local checksum round-trip involving
runtime-material.sha256, including its later verification, because it only
hashes unchanged SHARED_DIR files and cannot validate migration integrity.
Preserve the existing remote continuity and health check in the migrate-to-v3
heredoc as the migration verification.
- Line 89: Add an explicit validation failure message around the image_archive
size check, matching the diagnostic behavior of the extracted binary check at
lines 93-96. Ensure an empty or missing archive logs a clear error before
returning a nonzero status, rather than relying solely on errexit.
- Around line 131-137: Sanitize the migration output before it is published:
update the quay migrate logging and artifact-copy flow around migration.log so
credentials and tokens are redacted before the file reaches ARTIFACT_DIR.
Preserve the existing failure handling and sanitized-artifact message, and
ensure both successful and failed migration logs follow the redaction path.
---
Nitpick comments:
In
`@ci-operator/step-registry/quay/omr-v2/host-provision/quay-omr-v2-host-provision-commands.sh`:
- Around line 152-165: Update the OMRSecurityGroup SSH ingress rule to restrict
tcp/22 from 0.0.0.0/0 to the approved stable Prow egress CIDR or CIDRs, reusing
the existing configuration parameter or pattern for those ranges where
available; leave the port 8443 rule scoped to VpcCidr.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Enterprise
Run ID: d3a1bdd1-7617-4e52-aaef-c65b224fbcee
⛔ Files ignored due to path filters (2)
ci-operator/jobs/quay/quay/quay-quay-master-periodics.yamlis excluded by!ci-operator/jobs/**ci-operator/jobs/quay/quay/quay-quay-master-presubmits.yamlis excluded by!ci-operator/jobs/**
📒 Files selected for processing (27)
ci-operator/config/quay/quay/quay-quay-master__omr-v2-upgrade-v3.yamlci-operator/step-registry/quay/aws-ipi-disconnected-omr-v2-upgrade-v3-provision/OWNERSci-operator/step-registry/quay/aws-ipi-disconnected-omr-v2-upgrade-v3-provision/quay-aws-ipi-disconnected-omr-v2-upgrade-v3-provision-chain.metadata.jsonci-operator/step-registry/quay/aws-ipi-disconnected-omr-v2-upgrade-v3-provision/quay-aws-ipi-disconnected-omr-v2-upgrade-v3-provision-chain.yamlci-operator/step-registry/quay/omr-v2/OWNERSci-operator/step-registry/quay/omr-v2/host-provision/OWNERSci-operator/step-registry/quay/omr-v2/host-provision/quay-omr-v2-host-provision-commands.shci-operator/step-registry/quay/omr-v2/host-provision/quay-omr-v2-host-provision-ref.metadata.jsonci-operator/step-registry/quay/omr-v2/host-provision/quay-omr-v2-host-provision-ref.yamlci-operator/step-registry/quay/omr-v2/install/OWNERSci-operator/step-registry/quay/omr-v2/install/quay-omr-v2-install-commands.shci-operator/step-registry/quay/omr-v2/install/quay-omr-v2-install-ref.metadata.jsonci-operator/step-registry/quay/omr-v2/install/quay-omr-v2-install-ref.yamlci-operator/step-registry/quay/omr-v2/validate/OWNERSci-operator/step-registry/quay/omr-v2/validate/quay-omr-v2-validate-commands.shci-operator/step-registry/quay/omr-v2/validate/quay-omr-v2-validate-ref.metadata.jsonci-operator/step-registry/quay/omr-v2/validate/quay-omr-v2-validate-ref.yamlci-operator/step-registry/quay/omr-v3/gather/quay-omr-v3-gather-commands.shci-operator/step-registry/quay/omr-v3/gather/quay-omr-v3-gather-ref.yamlci-operator/step-registry/quay/omr-v3/mirror/quay-omr-v3-mirror-commands.shci-operator/step-registry/quay/omr-v3/mirror/quay-omr-v3-mirror-ref.yamlci-operator/step-registry/quay/omr-v3/upgrade-from-v2/OWNERSci-operator/step-registry/quay/omr-v3/upgrade-from-v2/quay-omr-v3-upgrade-from-v2-commands.shci-operator/step-registry/quay/omr-v3/upgrade-from-v2/quay-omr-v3-upgrade-from-v2-ref.metadata.jsonci-operator/step-registry/quay/omr-v3/upgrade-from-v2/quay-omr-v3-upgrade-from-v2-ref.yamlci-operator/step-registry/quay/omr-v3/validate/quay-omr-v3-validate-commands.shci-operator/step-registry/quay/omr-v3/validate/quay-omr-v3-validate-ref.yaml
| skopeo copy --retry-times=3 --src-authfile="${auth_file}" \ | ||
| "docker://${OMR_IMAGE}" \ | ||
| "docker-archive:${image_archive}:quay-mirror:ci" | ||
| [[ -s "${image_archive}" ]] |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Silent failure on empty image archive.
This relies solely on errexit to abort if the skopeo copy produced an empty/missing archive, with no diagnostic message — unlike the equivalent check at lines 93-96 for the extracted binary.
🔧 Suggested fix
-[[ -s "${image_archive}" ]]
+if [[ ! -s "${image_archive}" ]]; then
+ echo "The exported OMR v3 image archive is missing or empty." >&2
+ exit 1
+fi📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| [[ -s "${image_archive}" ]] | |
| if [[ ! -s "${image_archive}" ]]; then | |
| echo "The exported OMR v3 image archive is missing or empty." >&2 | |
| exit 1 | |
| fi |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@ci-operator/step-registry/quay/omr-v3/upgrade-from-v2/quay-omr-v3-upgrade-from-v2-commands.sh`
at line 89, Add an explicit validation failure message around the image_archive
size check, matching the diagnostic behavior of the extracted binary check at
lines 93-96. Ensure an empty or missing archive logs a clear error before
returning a nonzero status, rather than relying solely on errexit.
|
/pj-rehearse periodic-ci-quay-quay-master-omr-v2-upgrade-v3-disconnected-install-upgrade |
|
@jbpratt: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse periodic-ci-quay-quay-master-omr-v2-upgrade-v3-disconnected-install-upgrade |
|
@jbpratt: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse periodic-ci-quay-quay-master-omr-v2-upgrade-v3-disconnected-install-upgrade |
|
@jbpratt: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse periodic-ci-quay-quay-master-omr-v2-upgrade-v3-disconnected-install-upgrade |
|
@jbpratt: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse periodic-ci-quay-quay-master-omr-v2-upgrade-v3-disconnected-install-upgrade |
|
@jbpratt: your |
|
/pj-rehearse periodic-ci-quay-quay-master-omr-v2-upgrade-v3-disconnected-install-upgrade |
|
@jbpratt: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/retest |
|
/pj-rehearse periodic-ci-quay-quay-master-omr-v2-upgrade-v3-disconnected-install-upgrade |
|
@jbpratt: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse periodic-ci-quay-quay-master-omr-v2-upgrade-v3-disconnected-install-upgrade |
|
@jbpratt: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse periodic-ci-quay-quay-master-omr-v2-upgrade-v3-disconnected-install-upgrade |
|
@jbpratt: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse periodic-ci-quay-quay-master-omr-v2-upgrade-v3-disconnected-install |
|
@jbpratt: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse periodic-ci-quay-quay-master-omr-v2-upgrade-v3-disconnected-install |
|
@jbpratt: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse periodic-ci-quay-quay-master-omr-v2-upgrade-v3-disconnected-install |
|
@jbpratt: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
[REHEARSALNOTIFIER]
Interacting with pj-rehearseComment: Once you are satisfied with the results of the rehearsals, comment: |
|
/pj-rehearse periodic-ci-quay-quay-master-omr-v2-upgrade-v3-disconnected-install |
|
@jbpratt: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse periodic-ci-quay-quay-master-omr-v2-upgrade-v3-disconnected-install |
|
@jbpratt: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/lgtm |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: jbpratt, LiZhang19817 The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
/pj-rehearse ack |
|
@jbpratt: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
openshift-merge-bot
Bot
added
the
rehearsals-ack
|
@jbpratt: all tests passed! Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
* feat(quay): add OMR v2 to v3 migration periodic * fix(quay): use installer image for OMR host provisioning * fix(quay): own rootless Podman config directory * fix(quay): use supported OMR v2 install flags * fix(quay): preserve OMR host SSH access * fix(quay): use OMR v2 health endpoint * fix(quay): use journalctl timestamp format * fix(quay): make OMR v2 journal check diagnostic * fix(quay): validate migrated OMR registry * chore(quay): clarify OMR periodic name * test(quay): validate OMR migration persistence
2 participants
Summary
Add a weekly clean-room CI workflow that installs a disconnected OpenShift cluster through released OMR v2, migrates the populated registry to the pipeline-built OMR v3, and proves the cluster can still pull the preserved content.
Related Issue
NO-ISSUE: adds migration and disconnected-install coverage for OMR v3.
Changes
Testing
bash -n <changed command scripts>passesshellcheck --severity=warning <changed command scripts>passesmake registry-metadata SKIP_PULL=truepassesmake ci-operator-checkconfig SKIP_PULL=truepassesmake jobs WHAT=quay/quay SKIP_PULL=truepassescheckconfigfor the generated periodic passesmake validate-step-registry SKIP_PULL=true: the current image no longer accepts the Makefile's-prow-configflag; a direct resolver run loaded the config and registry before requiring in-cluster credentialsNotes
Target Version: not applicable. No backport is expected.
Summary by CodeRabbit
Adds a weekly disconnected Quay “clean-room” CI workflow on the
masterbranch to validate upgrading a populated mirrored registry from OMR v2 → OMR v3 while ensuring the registry contents remain pullable.The workflow runs as a Sunday 03:00 UTC periodic job and provisions an AWS disconnected environment using the existing FCOS proxy/bastion path plus a dedicated large RHEL 9 host for rootless OMR v2 (via a CloudFormation-backed EC2 host-provision step). It then:
Supporting CI infrastructure includes:
ci-operator/config/quay/.../quay-quay-master__omr-v2-upgrade-v3.yaml) wiring the pre provisioning chain, post gather, and test refs (quay-omr-v2-validate,quay-omr-v3-upgrade-from-v2,quay-omr-v3-validate), with images pinned to OpenShift release 4.22.Prow validation passes for syntax/ShellCheck/metadata/configuration/job generation and targeted Prow checks; Makefile step-registry validation remains incomplete because the current configresolver image no longer supports the
-prow-configflag.