feat(ci): add disconnected OCP smoke test for Helm and Operator - #5268
Merged
openshift-merge-bot[bot] merged 26 commits intoAug 27, 2026
Merged
Conversation
|
Skipping CI for Draft Pull Request. |
Member
Author
|
/agentic_review |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #5268 +/- ##
==========================================
- Coverage 63.68% 59.69% -3.99%
==========================================
Files 121 109 -12
Lines 2288 2062 -226
Branches 527 500 -27
==========================================
- Hits 1457 1231 -226
Misses 829 829
Partials 2 2
Continue to review full report in Codecov by Harness.
🚀 New features to boost your workflow:
|
Code Review by Qodo
1.
|
Member
Author
|
/test e2e-ocp-disconnected-helm-nightly |
Member
Author
|
/test e2e-ocp-disconnected-operator-nightly |
zdrapela
force-pushed
the
refactor/disconnected-ci-modular
branch
from
August 19, 2026 07:43
d4a33ef to
cbcef51
Compare
Member
Author
|
/test e2e-ocp-disconnected-helm-nightly |
Member
Author
|
/test e2e-ocp-disconnected-operator-nightly |
zdrapela
force-pushed
the
refactor/disconnected-ci-modular
branch
from
August 19, 2026 08:27
cbcef51 to
5513416
Compare
Member
Author
|
/test e2e-ocp-disconnected-helm-nightly |
Member
Author
|
/test e2e-ocp-disconnected-operator-nightly |
Member
Author
|
/test e2e-ocp-disconnected-helm-nightly |
Member
Author
|
/test e2e-ocp-disconnected-helm-nightly |
Member
Author
|
/test e2e-ocp-disconnected-helm-nightly |
Contributor
Contributor
Contributor
Member
Author
|
/agentic_review |
Member
Author
|
/test e2e-ocp-disconnected-helm-nightly /test e2e-ocp-helm |
|
Code review by qodo was updated up to the latest commit c56d5ef |
zdrapela
force-pushed
the
refactor/disconnected-ci-modular
branch
from
August 21, 2026 08:13
c56d5ef to
0839c7a
Compare
Member
Author
|
/test e2e-ocp-disconnected-operator-nightly |
zdrapela
marked this pull request as ready for review
August 21, 2026 08:17
The hub recovery in ensure_helm_hub_after_postgres deleted the hub pod after a 30s grace. On AWS the pod delete coincided with the PostgreSQL StatefulSet pod being rescheduled, triggering a ~5 min EBS CSI volume detach/re-attach (FailedAttachVolume) that cascaded into a smoke-test healthcheck timeout. Replace the destructive oc delete pod with a rolling oc rollout restart deployment, which only touches the hub Deployment and never disturbs the PostgreSQL StatefulSet. Raise the pre-restart grace 30s->180s so the hub usually goes Available on its own (making the restart a no-op), and the rollout timeout 300s->420s to absorb slow disconnected image pulls. Assisted-by: OpenCode
resolve_homepage_plugin_package grepped the mirroring summary and tried two hardcoded candidate names (dynamic-home-page, homepage) to cope with the RHIDP-14515 frontend rename, plus ->/-> separator handling. Read the homepage frontend package straight from the plugin catalog index that CATALOG_INDEX_IMAGE already pins instead. This works unmodified whether the index is pinned (e.g. :1.10, which pins dynamic-home-page) or tracks :next (which pins homepage), and survives any future rename: the package name, digest and frontend id are all taken from whichever homepage frontend the index actually references. Add a shared _catalog_index_source_ref helper (used by mirror_plugins, the local digest-list build, and homepage resolution) and resolve homepage before resolve_catalog_index_image, which rewrites CATALOG_INDEX_IMAGE into its mirror-consumption form. Assisted-by: OpenCode
Move the volumes/volumeMounts injected by the disconnected Helm
post-renderer into helm-post-renderer-patch.yaml, rendered with envsubst
for ${MIRROR_REGISTRY_URL} and merged into the rendered Deployment with
yq. The patch is now readable/reviewable as plain YAML instead of an
inline yq expression.
yq is still used for the merge (a structured array-append into the
install-dynamic-plugins initContainer by name), preserving the chart's
default volumes and avoiding the Helm array-clobber pitfall. Behavior is
unchanged; verified the rendered output matches the previous inline
version for single- and multi-document manifest streams.
Assisted-by: OpenCode
…ed mirror The integrated OCP registry does not auto-create projects on push. On a fresh cluster, skopeo/oc-mirror manifest writes to a missing namespace upload blobs but fail the manifest write with "denied", breaking the local disconnected smoke at the ImageStream-tagging stage. Pre-create the known push-target projects in setup_local_ocp_mirror and lazily create any data-driven namespace (derived from plugin source paths, e.g. rhdh-plugin-export-overlays) inside the ImageStream-tag loop. Grant image-puller across the same shared project list so the workload can pull mirrored plugins cross-namespace. Scoped entirely to the LOCAL_DISCONNECTED path; no effect on Prow CI. Assisted-by: OpenCode
The app-config-rhdh-disconnected-smoke.yaml ConfigMap is disconnected-only and belongs alongside the other disconnected resources rather than in the shared config_map directory. Assisted-by: OpenCode
Move the aarch64 skopeo shim heredoc out of local.sh into
resources/disconnected/skopeo-amd64-shim.sh.tpl and render it with
envsubst, substituting only ${REAL_SKOPEO} so the shim's own runtime
$#/$1/$@/${cmd} expansions are preserved. Keeps the generated script
out of the shell library and easier to read/lint.
Assisted-by: OpenCode
…esql-15
The community Helm chart's default PostgreSQL image
(registry.redhat.io/rhel9/postgresql-15) is not pullable without a Red Hat
pull secret. Align the disconnected fallback with the showcase value files
(quay.io/fedora/postgresql-15) and hoist it into a shared
POSTGRESQL_IMAGE_{REGISTRY,REPO,TAG} constant in env_variables.sh.
Also coalesce yq's literal "null" (missing key) to empty so the default
fallback is actually applied.
Assisted-by: OpenCode
redhat-developer#5263 made CATALOG_INDEX_IMAGE always set (via CATALOG_INDEX_IMAGE_OVERRIDE falling back to :RELEASE_VERSION), so the chart-value catalog-index pinning is now redundant dead code and a second source of truth. Delete it and consume the shared env contract everywhere: - Remove the CI_* chart-derived catalog block in ocp-disconnected-helm.sh; the index is mirrored from CATALOG_INDEX_IMAGE and re-pinned to the mirrored digest by resolve_catalog_index_image. - mirror.sh additionalImages now mirrors CATALOG_INDEX_IMAGE directly so CI and LOCAL_DISCONNECTED mirror and consume the same index. - Delete disconnected::pin_local_catalog_index_from_chart (local.sh), its CI-safe stub (plugins.sh), its export -f (disconnected.sh) and its call in ocp-disconnected-operator.sh. This also removes the duplicated @sha256 digest-separator normalization the catalog blocks carried (self-review #4/#5/#6); the remaining PG separator block is a single inline caller and stays as-is. Assisted-by: OpenCode
The disconnected Helm job duplicated the image --set flag construction with three divergences from helm::get_image_params: backstage registry points at the in-cluster mirror host, the catalog index registry points at MIRROR_REGISTRY_URL, and LOCAL_DISCONNECTED omits the hub image (chart + IDMS resolve it). Parameterize the shared helper with optional --backstage-registry, --catalog-registry and --omit-backstage-image; all default to the existing connected-install behavior so aks/eks/gke callers are unchanged. The disconnected job now builds its image flags through the helper instead of a bespoke block. Assisted-by: OpenCode
env.sh mixed environment/auth concerns with fetching helper scripts from the rhdh-operator repo. Move that helper into a dedicated lib/disconnected/scripts.sh and rename it disconnected::fetch_operator_repo_script to say what it fetches. Source the new module in disconnected.sh, update the export -f, and update the three call sites (plugins.sh, local.sh, ocp-disconnected-operator.sh). No behavior change. Assisted-by: OpenCode
local.sh had grown to ~700 lines mixing five distinct concerns. Split it into
lib/disconnected/local/ modules behind a thin facade that preserves the
sourced-once guard and the LOCAL_DISCONNECTED-only source contract:
hooks.sh cluster_mirror_host + _hook_* overrides for mirror.sh/plugins.sh
registry.sh integrated-registry bring-up, retry-on-503, MIRROR_* bootstrap,
amd64 skopeo shim
access.sh mirror push-target projects, workload + OLM pull access
mirror-host.sh IDMS/ITMS push-route -> in-cluster registry service rewrite
plugins.sh full mirror_plugins override + digest list + imagestream tags
Pure code movement: the set of defined functions and the
DISCONNECTED_LOCAL_MIRROR_PROJECTS array are byte-identical to the original, and
all 21 functions load through the facade. No behavior change.
Assisted-by: OpenCode
fetch_operator_repo_script had sha/pull-request/branch ref detection and an
optional $3 ref override, but no call site ever passed $3 and CI always sets
RELEASE_BRANCH_NAME from JOB_SPEC (defaulting to main locally). Drop the dead
branching and always fetch from refs/heads/${RELEASE_BRANCH_NAME}, matching the
existing convention in install-methods/operator.sh.
Assisted-by: OpenCode
Local disconnected runs leave pull secrets and CA files in mktemp dirs; extend the existing openshift-ci-tests.sh cleanup trap to rm -rf DISCONNECTED_TMPDIR when set.
Operator requires a separate rhdh-plugin-mirror-policy ConfigMap for extraFiles volume keys; Helm and Operator now both mount the same resources/disconnected/policy.json instead of duplicating inline JSON.
Disconnected Helm was splitting chart repository@sha256 fields inline. Move that into common::normalize_chart_image_ref so digest vs tag handling is not duplicated if other jobs need the same chart encoding.
A sourced RETURN trap clobbers later cleanup in the job shell, and tar -xzf fails on uncompressed OCI layers. Extract with tar -xf and remove the temp dir on an explicit return path instead.
…d smoke ref:// is resolved through the mirrored catalog index, so constructing oci:// package refs (and the CI vs local hook) is unnecessary.
--plugin-index enumerated every catalog plugin, including unpublished :tag refs that fail to pull. Smoke only needs homepage plus the catalog index so ref:// still resolves.
ref:// cannot resolve with includes: []; envsubst the catalog digest so install-dynamic-plugins does not CrashLoop. Tighten the homepage filter and override skopeo os/arch on catalog extract.
Integrated registry only serves ImageStreams by tag, so local disconnected needs :sha256-<digest> for the catalog index. Retry skopeo copies on registry Recreate EOF.
Smoke only needs the current stream; mirroring * on next pulled 43 historical images and aborted on quay CDN EOF.
zdrapela
force-pushed
the
refactor/disconnected-ci-modular
branch
from
August 27, 2026 06:15
a68ac85 to
372a443
Compare
Contributor
|
The container image build workflow finished with status: |
|
openshift-merge-bot
Bot
merged commit Aug 27, 2026
8b343e2
into
redhat-developer:main
30 checks passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Jira: https://redhat.atlassian.net/browse/RHIDP-13974
Human written
prepare-restricted-enviroment.shscript withoc-mirrorand avoids using Podman for rebuilding the index. To allow using nested Podman in CI, we would need to use a special image instead of thee2e-test-runnerimage, or adjust thee2e-test-runnerto include all the specifics of the special image. This will also complicate running it locally, so I settled on theoc-mirrorpath.AI Generated
Mirrors RHDH images, chart, operator, and dynamic plugins into an isolated mirror registry on a disconnected OCP cluster, deploys RHDH, and runs a Playwright smoke test (guest login → homepage). Covers both Helm and Operator installs, each runnable in CI (Prow bastion mirror) and Local (
LOCAL_DISCONNECTED=1, cluster integrated registry).Path selection
flowchart TD CI["CI: openshift-ci-tests.sh<br/>*disconnected*{helm,operator}*nightly*"] --> F L["Local: local-run.sh<br/>detects *disconnected*, requires oc login<br/>sets DISCONNECTED=true, LOCAL_DISCONNECTED=1"] --> F F["lib/disconnected.sh (facade)<br/>sources env/mirror/plugins/namespace/operator/helm"] --> D{LOCAL_DISCONNECTED=1?} D -- no --> CIH["CI no-op hooks<br/>external MIRROR_REGISTRY_URL"] D -- yes --> LO["source lib/disconnected/local.sh<br/>overrides _hook_* + integrated registry"] CIH --> H["handle_ocp_disconnected_helm"] CIH --> O["handle_ocp_disconnected_operator"] LO --> H LO --> OPaths
Helm. Validate env + auth → pull chart (GA from charts.openshift.io, CI from OCI) →
oc-mirrorthe hub, PostgreSQL, and catalog-index images → apply IDMS/ITMS and wait for MachineConfigPools → mirror plugins viamirror-plugins.shand re-pin homepage/catalog-index digests → create mirror CA + registry-auth secret and plugin-mirror ConfigMap →helm upgrade -iwith a post-renderer that appends disconnected volumes (registries.conf, mirror CA) to the rendered Deployment → EBS-safe hub recovery → Playwright smoke.Operator. Validate env + auth → fetch and run
prepare-restricted-environment.shwith--use-oc-mirror(mirrors operator/operand images, installs the OLM v1 CatalogSource) → wait for ClusterCatalogServing=Trueand the Backstage CRD → mirror plugins viamirror-plugins.shand re-pin digests → create mirror CA + auth +policy.jsonand homepage/plugin-mirror ConfigMaps → render the Backstage CR from anenvsubsttemplate (extraFiles/extraEnvs for disconnected mounts) →deploy_rhdh_operator→ Playwright smoke.CI vs Local. CI uses an external bastion mirror (
MIRROR_REGISTRY_URL) andprepare's default flags plus external CA/pull-secret helpers. Local uses the cluster's integrated registry vialib/disconnected/local/:registry.shbrings it up (MIRROR_* bootstrap, amd64 skopeo shim),mirror-host.shrewrites the push-route IDMS to the in-cluster registry service,access.shcreates push-target projects and workload/OLM pull grants,plugins.shfully overrides plugin mirroring (digest list + imagestream tags), andhooks.shsupplies the_hook_*/cluster_mirror_hostoverrides.Key decisions
lib/disconnected.shsources concern-based modules; CI defines no-op_hook_*,local.shoverrides them for integrated-registry behaviour. Keeps CI functions clean.newuidmap/VFS chown fail underhostUsers: false), so the operator path forces--use-oc-mirror.extraVolumes[]; the post-renderer appends disconnected volumes to the rendered Deployment instead.envsubsttemplates (Backstage CR, homepage dynamic-plugins, post-renderer patch, skopeo shim).CATALOG_INDEX_IMAGEcontract — CI and Local mirror and consume the same index; homepage/catalog-index re-pinned to the actually-mirrored digest.oc rollout restart(notoc delete pod) so the PostgreSQL StatefulSet is never disturbed (avoids ~5 min EBS re-attach).policy.jsonfor unsigned mirrored images.rhdh-operator scripts
Both are fetched at runtime by
disconnected::fetch_operator_repo_scriptfromredhat-developer/rhdh-operator.rhdh/scripts/onrefs/heads/${RELEASE_BRANCH_NAME}(defaults tomainlocally):mirror-plugins.shprepare-restricted-environment.shCompanion PR