Skip to content

Resolve FBC image from Quay; exclude upgrade tests from FAR presubmit - #82774

Open
ugreener wants to merge 1 commit into
openshift:mainfrom
ugreener:fix/medik8s-lib-quay-first-fbc
Open

Resolve FBC image from Quay; exclude upgrade tests from FAR presubmit#82774
ugreener wants to merge 1 commit into
openshift:mainfrom
ugreener:fix/medik8s-lib-quay-first-fbc

Conversation

@ugreener

@ugreener ugreener commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

Problem

1. medik8s-lib Quay resolution: The resolve_commit_sha() function in medik8s-lib.sh resolved the FBC (File-Based Catalog) commit SHA by querying the GitLab API for the HEAD commit of dragonfly/rhwa-fbc. This approach had three problems:

  • Wrong image selected ~50% of the time. GitLab HEAD points to whichever OCP version was built last (e.g., 4.22), but the CI job may need a different version (e.g., 4.18). When they don't match, the Quay manifest check fails with a 404, triggering a noisy fallback path.
  • Unnecessary GitLab dependency. gitlab.cee.redhat.com regularly returns 503 errors lasting 30+ seconds.
  • Redundant API calls. The unpinned path made 3 API calls when 1 is sufficient.

2. FAR upgrade test leak: The FAR upgrade test (upgrade.go, added in commit 43fbac9c on Jul 29) runs inside the regular e2e-far-aws presubmit and e2e-far-weekly-aws periodic, but it has its own dedicated presubmit (e2e-far-upgrade-presubmit-aws). Running it in the regular FAR job causes a 409 Conflict failure because the upgrade test's BeforeAll creates a Secret that already exists from the destructive tests.

Changes

medik8s-lib-commands.sh

  • resolve_commit_sha(): Skip GitLab API entirely. Query Quay directly for the latest active SHA tag for the specific OCP version image. Tags are filtered to 40-character hex SHA names and sorted by start_ts for deterministic selection.
  • Separate curl error handling from jq parsing for better error messages during Quay API failures.
  • Add OCP_VERSION format validation (^[0-9]{2,4}$) as defense-in-depth.
  • Set FBC_SHA_PINNED=false explicitly in the Quay resolution path.
  • verify_fbc_image(): Only verify manifest for explicitly pinned SHAs.

medik8s-system-tests CI config (4.22-konflux)

  • Add ECO_TEST_LABELS: '!tier:upgrade' to e2e-far-aws presubmit and e2e-far-weekly-aws periodic to exclude upgrade tests. Upgrade tests have their own dedicated job in the 4.22-upgrade variant with ECO_TEST_LABELS: tier:upgrade.

Summary by CodeRabbit

  • Updates Medik8s OpenShift 4.22 CI to resolve unpinned FBC commit SHAs from Quay active tags for the requested OCP_VERSION.
  • Selects the newest 40-character SHA tag by start_ts with one API call.
  • Preserves manifest verification for explicitly pinned FBC_SHA_PINNED values.
  • Continues to use the resolved SHA for IDMS fetching.
  • Excludes upgrade tests from FAR AWS regular and weekly runs with ECO_TEST_LABELS: '!tier:upgrade'.
  • Documentation now uses OCP_VERSION and FBC_SHA_PINNED.
  • The author requested two rehearsal runs for pull-ci-medik8s-system-tests-main-4.22-konflux-e2e-far-aws.

@openshift-ci
openshift-ci Bot requested review from jmontleon and weshayutin August 2, 2026 06:49
@openshift-ci openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 2, 2026
@coderabbitai

coderabbitai Bot commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

Walkthrough

Medik8s now selects active Quay SHA tags using OCP_VERSION, tracks pinned SHAs, and verifies pinned manifests without fallback tags. Regular and weekly FAR AWS tests exclude upgrade-tier tests.

Changes

Medik8s CI behavior

Layer / File(s) Summary
Quay SHA resolution and manifest verification
ci-operator/step-registry/medik8s/lib/medik8s-lib-commands.sh
The caller provides OCP_VERSION and FBC_SHA_PINNED. The script resolves active 40-character Quay SHA tags when needed and verifies manifests only for pinned SHAs.
FAR AWS test label propagation
ci-operator/config/medik8s/system-tests/medik8s-system-tests-main__4.22-konflux.yaml
Regular and weekly FAR AWS environments set ECO_TEST_LABELS to exclude upgrade-tier tests. Their test steps forward the value.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Caller
  participant resolve_commit_sha
  participant QuayAPI
  participant verify_fbc_image
  Caller->>resolve_commit_sha: provide OCP_VERSION and FBC_SHA_PINNED
  alt SHA supplied
    resolve_commit_sha->>Caller: mark SHA as pinned
  else SHA not supplied
    resolve_commit_sha->>QuayAPI: query active 40-character SHA tags
    QuayAPI-->>resolve_commit_sha: return latest active SHA tag
    resolve_commit_sha->>Caller: return resolved SHA
  end
  opt SHA is pinned
    Caller->>verify_fbc_image: provide pinned SHA
    verify_fbc_image->>QuayAPI: request Quay manifest
    QuayAPI-->>verify_fbc_image: return HTTP status
    verify_fbc_image-->>Caller: pass or fail verification
  end
Loading

Suggested labels: rehearsals-ack

Suggested reviewers: jmontleon, weshayutin

🚥 Pre-merge checks | ✅ 15
✅ Passed checks (15 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed The PR changes only shell logic and CI environment propagation; it adds or modifies no Ginkgo test titles or It/Describe/Context/When constructs.
Test Structure And Quality ✅ Passed The patch changes only a shell library and YAML configuration; it adds no Ginkgo test code or It blocks to review.
Microshift Test Compatibility ✅ Passed The PR changes only a shell library and CI YAML; it adds no Ginkgo tests or references to MicroShift-unsupported APIs or features.
Single Node Openshift (Sno) Test Compatibility ✅ Passed The patch changes only shell library logic and CI YAML; it adds or modifies no Ginkgo e2e tests that could assume a multi-node cluster.
Topology-Aware Scheduling Compatibility ✅ Passed The PR changes a shell library and CI test environment variables only; no deployment, operator, controller, or topology-related scheduling constraints were added.
Ote Binary Stdout Contract ✅ Passed The PR changes only medik8s shell and YAML files; it adds no OTE binary or process-level Go setup, so the OTE stdout contract is not implicated.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed The PR adds no new Ginkgo tests. It changes CI configuration and a shell library; the Quay calls are not test implementations covered by this check.
No-Weak-Crypto ✅ Passed The changed files add Quay tag selection and HTTP manifest checks only; no MD5, SHA1 algorithm, DES, RC4, 3DES, Blowfish, ECB, custom crypto, or secret comparisons are introduced.
Container-Privileges ✅ Passed The PR adds only ECO_TEST_LABELS and FBC resolution logic; scans found no privileged, host namespace, SYS_ADMIN, allowPrivilegeEscalation, or root settings.
No-Sensitive-Data-In-Logs ✅ Passed Changed logs contain only OCP version, public Quay image paths, HTTP status, and 40-character FBC commit SHAs; no passwords, tokens, API keys, PII, or customer data are logged.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes both main changes: Quay-based FBC image resolution and exclusion of upgrade tests from FAR presubmit.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@ci-operator/step-registry/medik8s/lib/medik8s-lib-commands.sh`:
- Around line 70-84: The resolve_commit_sha() function currently evaluates only
the first 50 Quay tags, so it can miss the latest active SHA tag. Update the
Quay-fetch logic to iterate through all result pages (or otherwise retrieve all
active tags), accumulate the matching 40-character SHA tags, and then select the
newest using last_modified descending before assigning FBC_COMMIT_SHA.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Pro Plus

Run ID: 58d7ef8d-9493-4b2b-80fd-da402fe27009

📥 Commits

Reviewing files that changed from the base of the PR and between 08d932a and d707376.

📒 Files selected for processing (1)
  • ci-operator/step-registry/medik8s/lib/medik8s-lib-commands.sh

Comment thread ci-operator/step-registry/medik8s/lib/medik8s-lib-commands.sh
@ugreener ugreener changed the title medik8s-lib: resolve FBC image from Quay instead of GitLab HEAD Resolve FBC commit SHA from Quay active tags instead of GitLab API Aug 2, 2026
@ugreener
ugreener force-pushed the fix/medik8s-lib-quay-first-fbc branch 2 times, most recently from c55c132 to e94b86e Compare August 2, 2026 07:17
@ugreener

ugreener commented Aug 2, 2026

Copy link
Copy Markdown
Contributor Author

/pj-rehearse pull-ci-medik8s-system-tests-main-4.22-konflux-e2e-far-aws

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@ugreener: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@ugreener

ugreener commented Aug 2, 2026

Copy link
Copy Markdown
Contributor Author

/pj-rehearse pull-ci-medik8s-system-tests-main-4.22-konflux-e2e-far-aws

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@ugreener: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

The dragonfly/rhwa-fbc GitLab repo is a monorepo containing FBC
definitions for multiple OCP versions (4.21, 4.22, etc.). Konflux
builds separate images per version. When HEAD is a 4.21-only commit,
checking rhwa-fbc-422:<HEAD_SHA> on Quay returns 404, forcing a
noisy fallback path.

Fix by resolving the FBC commit SHA directly from Quay's active tags
for the specific OCP version image. This eliminates the GitLab API
call, the manifest 404, and the fallback entirely for the unpinned
path. Pinned SHA verification (via FBC_COMMIT_SHA env var) is
preserved unchanged.

Co-Authored-By: Claude <noreply@anthropic.com>
@ugreener
ugreener force-pushed the fix/medik8s-lib-quay-first-fbc branch from e94b86e to da5a7ea Compare August 2, 2026 12:06
@ugreener ugreener changed the title Resolve FBC commit SHA from Quay active tags instead of GitLab API medik8s-lib: resolve FBC image from Quay; exclude upgrade tests from FAR presubmit Aug 2, 2026
@ugreener

ugreener commented Aug 2, 2026

Copy link
Copy Markdown
Contributor Author

/pj-rehearse pull-ci-medik8s-system-tests-main-4.22-konflux-e2e-far-aws

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@ugreener: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

[REHEARSALNOTIFIER]
@ugreener: the pj-rehearse plugin accommodates running rehearsal tests for the changes in this PR. Expand 'Interacting with pj-rehearse' for usage details. The following rehearsable tests have been affected by this change:

Test name Repo Type Reason
pull-ci-medik8s-system-tests-main-4.22-konflux-e2e-far-aws medik8s/system-tests presubmit Ci-operator config changed
pull-ci-medik8s-system-tests-main-4.22-disconnected-e2e-far-aws-disconnected medik8s/system-tests presubmit Registry content changed
pull-ci-medik8s-system-tests-main-4.22-upgrade-e2e-far-upgrade-presubmit-aws medik8s/system-tests presubmit Registry content changed
pull-ci-medik8s-system-tests-main-4.22-konflux-e2e-mdr-aws medik8s/system-tests presubmit Registry content changed
pull-ci-medik8s-system-tests-main-4.22-konflux-e2e-nhc-aws medik8s/system-tests presubmit Registry content changed
pull-ci-medik8s-system-tests-main-4.22-konflux-e2e-nmo-aws medik8s/system-tests presubmit Registry content changed
pull-ci-medik8s-system-tests-main-4.22-konflux-e2e-sbr-aws-odf medik8s/system-tests presubmit Registry content changed
pull-ci-medik8s-system-tests-main-4.22-konflux-e2e-sbr-aws-efs medik8s/system-tests presubmit Registry content changed
pull-ci-medik8s-system-tests-main-4.22-konflux-e2e-sbr-hypershift-aws-odf medik8s/system-tests presubmit Registry content changed
pull-ci-medik8s-system-tests-main-4.22-konflux-e2e-sbr-hypershift-persistent-aws-odf medik8s/system-tests presubmit Registry content changed
pull-ci-medik8s-system-tests-main-4.22-konflux-e2e-snr-aws medik8s/system-tests presubmit Registry content changed
periodic-ci-medik8s-system-tests-main-4.22-konflux-e2e-snr-weekly-aws N/A periodic Registry content changed
periodic-ci-medik8s-system-tests-main-4.22-konflux-e2e-nhc-weekly-aws N/A periodic Registry content changed
periodic-ci-medik8s-system-tests-main-4.22-konflux-e2e-far-weekly-aws N/A periodic Ci-operator config changed
periodic-ci-medik8s-system-tests-main-4.22-konflux-e2e-nmo-weekly-aws N/A periodic Registry content changed
periodic-ci-medik8s-system-tests-main-4.22-disconnected-e2e-far-weekly-aws-disconnected N/A periodic Registry content changed
periodic-ci-medik8s-system-tests-main-4.22-konflux-e2e-sbr-weekly-aws-odf N/A periodic Registry content changed
periodic-ci-medik8s-system-tests-main-4.22-konflux-e2e-mdr-weekly-aws N/A periodic Registry content changed
periodic-ci-medik8s-system-tests-main-4.22-upgrade-e2e-far-upgrade-aws N/A periodic Registry content changed
periodic-ci-medik8s-system-tests-main-4.22-konflux-e2e-sbr-weekly-aws-efs N/A periodic Registry content changed
periodic-ci-medik8s-system-tests-main-4.21-disconnected-e2e-sbr-weekly-aws-disconnected-nfs N/A periodic Registry content changed
Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

@openshift-ci

openshift-ci Bot commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ugreener

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci

openshift-ci Bot commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

@ugreener: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@ugreener ugreener changed the title medik8s-lib: resolve FBC image from Quay; exclude upgrade tests from FAR presubmit Resolve FBC image from Quay in medik8s-lib; exclude upgrade tests from FAR presubmit Aug 2, 2026
@ugreener ugreener changed the title Resolve FBC image from Quay in medik8s-lib; exclude upgrade tests from FAR presubmit Resolve FBC image from Quay; exclude upgrade tests from FAR presubmit Aug 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant