Skip to content

test(ci): exercise Helm PostgreSQL 15-16-18 upgrade path#5139

Draft
zdrapela wants to merge 10 commits into
redhat-developer:mainfrom
zdrapela:test/pg15-to-pg18-helm-upgrade
Draft

test(ci): exercise Helm PostgreSQL 15-16-18 upgrade path#5139
zdrapela wants to merge 10 commits into
redhat-developer:mainfrom
zdrapela:test/pg15-to-pg18-helm-upgrade

Conversation

@zdrapela

@zdrapela zdrapela commented Jul 22, 2026

Copy link
Copy Markdown
Member

Summary

  • CI evidence for RHIDP-14594: chart-managed PostgreSQL upgrade using product rhel9/postgresql-* images
  • Flow: PG15 → Playwright → PG16 → Playwright → PG18 → Playwright
  • Unique ARTIFACT_DIR subdirs: showcase-pg15, showcase-pg16, showcase-pg18 (Playwright + pod logs + postgres diagnostics; no overwrite)
  • Two-hop POSTGRESQL_UPGRADE=copy: 15→16→18

Companion Fedora PR: #5141

Verified (prior run, single Playwright at end)

Test plan

  • /test e2e-ocp-helm with Playwright after each major
  • Artifacts under showcase-pg15|pg16|pg18

Validate the sclorg-supported two-hop path (POSTGRESQL_UPGRADE=copy)
for RHIDP-14594 evidence on e2e-ocp-helm.
@openshift-ci

openshift-ci Bot commented Jul 22, 2026

Copy link
Copy Markdown

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@zdrapela

Copy link
Copy Markdown
Member Author

/test e2e-ocp-helm

@zdrapela zdrapela changed the title test(ci): Helm internal PostgreSQL 15→16→18 upgrade (RHIDP-14594) test(ci): Helm internal PostgreSQL 15 to 18 upgrade (RHIDP-14594) Jul 22, 2026
Avoid racing Playwright against a recreating postgresql pod after
removing POSTGRESQL_UPGRADE. Log server_version for upgrade evidence.
@zdrapela zdrapela changed the title test(ci): Helm internal PostgreSQL 15 to 18 upgrade (RHIDP-14594) test(ci): exercise Helm PostgreSQL 15-16-18 upgrade path Jul 22, 2026
@zdrapela

Copy link
Copy Markdown
Member Author

/test e2e-ocp-helm

@github-actions

Copy link
Copy Markdown
Contributor

The container image build workflow finished with status: cancelled.

@github-actions

Copy link
Copy Markdown
Contributor

Image was built and published successfully. It is available at:

@codecov

codecov Bot commented Jul 22, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 59.96%. Comparing base (7d73af5) to head (7a93c77).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #5139      +/-   ##
==========================================
- Coverage   63.69%   59.96%   -3.74%     
==========================================
  Files         123      111      -12     
  Lines        2424     2198     -226     
  Branches      573      545      -28     
==========================================
- Hits         1544     1318     -226     
  Misses        878      878              
  Partials        2        2              
Flag Coverage Δ
rhdh 59.96% <ø> (-3.74%) ⬇️
Components Coverage Δ
Backend plugins ∅ <ø> (∅)
Backend app 66.66% <ø> (ø)
Frontend app 58.89% <ø> (ø)
Plugin utils ∅ <ø> (∅)
Dynamic plugins utils ∅ <ø> (∅)

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7d73af5...7a93c77. Read the comment docs.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Start from fedora/postgresql-15 (not rhel9 default), require the
expected image before Ready, raise upgrade timeouts, enlarge PVC for
copy mode, and dump pod logs when a hop stalls.
@zdrapela

Copy link
Copy Markdown
Member Author

/test e2e-ocp-helm

@github-actions

Copy link
Copy Markdown
Contributor

Image was built and published successfully. It is available at:

Command substitution was swallowing oc describe/logs on hop failure, and
psql warnings were being treated as database names during collation refresh.
@zdrapela

Copy link
Copy Markdown
Member Author

/test e2e-ocp-helm

fedora/postgresql-18 advertises PREV_VERSION=16 but only ships
postgresql-17 binaries, so POSTGRESQL_UPGRADE=copy from 16 never becomes Ready.
@zdrapela

Copy link
Copy Markdown
Member Author

Root cause of Hop B failure (from gather-extra logs):

==========  $PGDATA upgrade: 16 -> 18  ==========
===>  Starting old postgresql once again for a clean shutdown...
/usr/lib64/pgsql/postgresql-16/bin/pg_ctl: No such file or directory

quay.io/fedora/postgresql-18 sets POSTGRESQL_PREV_VERSION=16 but only packages /usr/lib64/pgsql/postgresql-17/. Switched the CI path to registry.redhat.io/rhel9/postgresql-{15,16,18} (product chart images), which correctly ship the previous-major binaries.

/test e2e-ocp-helm

@github-actions

Copy link
Copy Markdown
Contributor

The container image build workflow finished with status: cancelled.

@zdrapela

Copy link
Copy Markdown
Member Author

e2e-ocp-helm green on e61a9a512

Evidence (job):

  • rhel9/postgresql-1515.18
  • Hop A POSTGRESQL_UPGRADE=copy16.14
  • Hop B POSTGRESQL_UPGRADE=copy18.4
  • Showcase Playwright: 36 passed
  • Log: PostgreSQL 15 -> 16 -> 18 Helm upgrade sequence completed

@github-actions

Copy link
Copy Markdown
Contributor

Image was built and published successfully. It is available at:

Exercise PG15 → tests → PG16 → tests → PG18 → tests, and always persist
pod logs plus Playwright output under unique ARTIFACT_DIR subdirs per hop.
@zdrapela

Copy link
Copy Markdown
Member Author

Updated flow: Playwright after each major (PG15 / PG16 / PG18), with unique ARTIFACT_DIR subdirs (showcase-pg15, showcase-pg16, showcase-pg18) so pod logs and Playwright artifacts are not overwritten.

/test e2e-ocp-helm

@github-actions

Copy link
Copy Markdown
Contributor

Image was built and published successfully. It is available at:

Multi-phase PG upgrade CI left kubectl redis port-forwards bound on 16379
because stop() only signaled the shell parent. Spawn detached, kill the
process group, use direct kubectl args, and fail fast when OVERALL_RESULT
is already non-zero after a phase.
@zdrapela

Copy link
Copy Markdown
Member Author

Fix for multi-phase Playwright: tear down redis port-forward process groups + fail fast when a phase already set OVERALL_RESULT=1.

/test e2e-ocp-helm

@zdrapela

Copy link
Copy Markdown
Member Author

/test e2e-ocp-helm

@zdrapela
zdrapela marked this pull request as ready for review July 23, 2026 08:18
@zdrapela

Copy link
Copy Markdown
Member Author

/test e2e-ocp-helm

@rhdh-qodo-merge

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (1) 📘 Rule violations (0) 🔗 Cross-repo conflicts (1) 📜 Skill insights (1)

Context used
⚠️ Tickets: not configured — ticket URL found in PR but could not be fetched — check ticket provider credentials
✅ Compliance rules (platform): 46 rules
✅ Cross-repo context
  Explored: repo: redhat-developer/rhdh-chart (sha: e476f298)
  Not relevant to this PR: redhat-developer/rhdh-plugins
  Not relevant to this PR: redhat-developer/rhdh-operator
  Not relevant to this PR: redhat-developer/rhdh-local

Grey Divider


Action required

1. Hardcoded externalAccess token 📜 Skill insight ⛨ Security
Description
New Helm values files hardcode a static backend externalAccess token/subject
(test-token/test-subject), which are credential-like values committed to the repo. This violates
the no-secrets-in-code requirement and risks accidental reuse outside CI or leakage via
logs/artifacts.
Code

.ci/pipelines/value_files/values_showcase_15.yaml[R65-70]

+          dangerouslyDisableDefaultAuthPolicy: true
+          externalAccess:
+            - type: static
+              options:
+                token: test-token
+                subject: test-subject
Relevance

⭐⭐⭐ High

Team previously removed hardcoded static API token from CI values and sourced it from secrets/env
vars.

PR-#2565

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The checklist forbids committing secrets/credentials in code. The added Helm values configure
backend externalAccess using a hardcoded token/subject rather than sourcing from a Secret or
environment variable.

Rule 2137: Secrets must not be hardcoded in source files
.ci/pipelines/value_files/values_showcase_15.yaml[63-70]
.ci/pipelines/value_files/values_showcase_16.yaml[63-70]
.ci/pipelines/value_files/values_showcase_18.yaml[63-70]
Skill: e2e-verify-fix

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
New Helm values files commit a static auth credential (`externalAccess.options.token` / `subject`) directly in YAML.

## Issue Context
These values are used to configure Backstage backend auth; even if intended for CI, committed tokens/credentials violate the repository's secrets-in-code requirements.

## Fix Focus Areas
- .ci/pipelines/value_files/values_showcase_15.yaml[63-70]
- .ci/pipelines/value_files/values_showcase_16.yaml[63-70]
- .ci/pipelines/value_files/values_showcase_18.yaml[63-70]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools



Remediation recommended

2. RBAC tests skipped 🐞 Bug ≡ Correctness
Description
handle_ocp_pull now deploys/tests only the base (showcase) Helm release and never deploys or runs
Playwright against the RBAC release, reducing CI coverage and allowing RBAC regressions to merge
undetected. This is a behavior change from the standard Helm CI flow which validates both showcase
and showcase-rbac.
Code

.ci/pipelines/jobs/ocp-pull.sh[R91-104]

  cluster_setup_ocp_helm
-  initiate_deployments "${PW_PROJECT_SHOWCASE}" "${PW_PROJECT_SHOWCASE_RBAC}"
+
+  local original_value_file="${HELM_CHART_VALUE_FILE_NAME}"
+  HELM_CHART_VALUE_FILE_NAME="values_showcase_15.yaml"
+  base_deployment "${PW_PROJECT_SHOWCASE}"
+  HELM_CHART_VALUE_FILE_NAME="${original_value_file}"
+
  deploy_test_backstage_customization_provider "${NAME_SPACE}"
  local url="https://${RELEASE_NAME}-developer-hub-${NAME_SPACE}.${K8S_CLUSTER_ROUTER_BASE}"
-  testing::check_and_test "${RELEASE_NAME}" "${NAME_SPACE}" "${PW_PROJECT_SHOWCASE}" "${url}"
-  local rbac_url="https://${RELEASE_NAME_RBAC}-developer-hub-${NAME_SPACE_RBAC}.${K8S_CLUSTER_ROUTER_BASE}"
-  testing::check_and_test "${RELEASE_NAME_RBAC}" "${NAME_SPACE_RBAC}" "${PW_PROJECT_SHOWCASE_RBAC}" "${rbac_url}"
+  # Unique artifact roots so Playwright/junit/pod logs from later hops cannot overwrite earlier ones.
+  local art_pg15="${PW_PROJECT_SHOWCASE}-pg15"
+  local art_pg16="${PW_PROJECT_SHOWCASE}-pg16"
+  local art_pg18="${PW_PROJECT_SHOWCASE}-pg18"
+
Relevance

⭐⭐ Medium

No direct prior reviews on ocp-pull requiring RBAC; repo does invest in RBAC CI fixes though.

PR-#4526
PR-#4075

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The updated pull job shows only base_deployment and the PG upgrade phases for the base release,
with no RBAC URL/test invocation. The repo’s deployment utilities and nightly Helm job demonstrate
that RBAC is normally deployed and tested as part of the Helm CI signal.

.ci/pipelines/jobs/ocp-pull.sh[80-127]
.ci/pipelines/utils.sh[443-550]
.ci/pipelines/jobs/ocp-nightly.sh[16-47]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
The OCP pull Helm job (`handle_ocp_pull`) no longer deploys the RBAC instance or runs its Playwright suite. This reduces the CI signal compared to the standard Helm CI flow.

### Issue Context
`initiate_deployments` exists to run base + RBAC deployments together (in disjoint namespaces). Nightly Helm CI runs both showcase and showcase-rbac tests.

### Fix Focus Areas
- Reintroduce RBAC deployment in the pull job while keeping the new PG15→PG16→PG18 upgrade sequence for the base release.
- Ensure RBAC Playwright runs at least once (e.g., after initial deployments), with its own artifact subdir override to prevent clobbering.

### Fix Focus Areas (code references)
- .ci/pipelines/jobs/ocp-pull.sh[80-127]
- .ci/pipelines/utils.sh[443-550]
- .ci/pipelines/jobs/ocp-nightly.sh[16-47]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools



Informational

3. Hardcoded Helm release name 🔗 Cross-repo conflict ☼ Reliability
Description
The new PostgreSQL diagnostics helper hardcodes helm get values rhdh ..., assuming the Helm
release is always named rhdh, but rhdh-chart installs use an arbitrary, user-defined release name
and resource naming depends on .Release.Name. As a result, when RELEASE_NAME is overridden,
diagnostics may query the wrong release and even suppress the resulting error, producing
wrong/misleading or empty output and making upgrade failures harder to debug.
Code

.ci/pipelines/lib/postgres.sh[R42-43]

+    log::info "Helm values snapshot (postgresql image):"
+    helm get values rhdh -n "${namespace}" 2>&1 | grep -A20 -E 'postgresql:|^  image:' || true
Relevance

⭐⭐⭐ High

Team has accepted avoiding hardcoded deployment identifiers; similar accepted change to avoid
hardcoded service names.

PR-#2939

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The cited pipeline change adds dump_postgres_diagnostics() that explicitly runs `helm get values
rhdh, which fixes the queried release name to rhdh` regardless of how the chart was installed. In
contrast, the pipeline and chart workflows allow a configurable release name (via env/config and a
release_name argument used during Helm install), and the chart’s naming helpers derive
resource/secret names from .Release.Name rather than a constant; therefore, when the release name
is not rhdh, helm get values rhdh will target a different/nonexistent release, and with the
error suppressed, the collected Helm values snapshot can be missing or misleading.

.ci/pipelines/lib/postgres.sh[21-44]
.ci/pipelines/lib/postgres.sh[18-45]
.ci/pipelines/env_variables.sh[71-83]
.ci/pipelines/lib/helm.sh[241-268]
External repo: redhat-developer/rhdh-chart, charts/backstage/README.md [26-33]
External repo: redhat-developer/rhdh-chart, charts/backstage/templates/_helpers.tpl [25-36]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`.ci/pipelines/lib/postgres.sh` hardcodes `helm get values rhdh`, which assumes the chart is always installed as release `rhdh`. This conflicts with rhdh-chart usage and the pipeline’s configurable `RELEASE_NAME`/`release_name`, causing PostgreSQL diagnostics to query the wrong release and potentially suppress errors, leading to missing or misleading diagnostic output.

## Issue Context
- `dump_postgres_diagnostics()` is intended to help debug chart-managed PostgreSQL upgrades by capturing relevant state, including a Helm values snapshot.
- rhdh-chart installations commonly use an arbitrary, user-provided release name (e.g., `helm install my-backstage ...`), and chart resource naming relies on `.Release.Name`.
- The CI/pipeline supports configurable release naming (via env vars) and existing Helm install paths already use a provided `release_name`, so diagnostics must align with that configured release.

## Fix Focus Areas
- .ci/pipelines/lib/postgres.sh[18-45]
- .ci/pipelines/env_variables.sh[71-83]
- .ci/pipelines/lib/helm.sh[241-268]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


Grey Divider

Qodo Logo

@rhdh-qodo-merge

Copy link
Copy Markdown

PR Summary by Qodo

CI: verify Helm internal PostgreSQL 15→16→18 upgrades with Playwright per hop

🧪 Tests ✨ Enhancement 🕐 40+ Minutes

Grey Divider

AI Description

• Add a CI upgrade flow for chart-managed PostgreSQL: 15 → 16 → 18 using rhel9 images.
• Run Playwright after each major and store artifacts under per-hop directories.
• Improve upgrade reliability with explicit Postgres readiness/image checks and richer diagnostics.
Diagram

graph TD
  A["ocp-pull.sh"] --> B["postgres.sh helpers"] --> C[("PostgreSQL Pod/PVC")]
  A --> D["Helm install/upgrade"] --> E[/"values_showcase_15/16/18*.yaml"/]
  D --> C
  A --> F(["Backstage Deployment"]) --> G["Playwright e2e"]
  G --> H["kubectl port-forward (Redis)"]
  subgraph Legend
    direction LR
    _s["Script"] ~~~ _k(["K8s workload"]) ~~~ _c[/"Helm values"/] ~~~ _d[("Database")]
  end
Loading
High-Level Assessment

The following are alternative approaches to this PR:

1. Use `oc wait`/statefulset rollout checks instead of custom polling
  • ➕ Less bespoke logic; relies on standard Kubernetes readiness primitives
  • ➕ Potentially simpler logs/maintenance over time
  • ➖ Harder to enforce 'expected image substring' gating during upgrades
  • ➖ Often provides less actionable diagnostics when pods churn during copy-mode upgrades
2. Single Playwright run at the end of all upgrade hops
  • ➕ Shorter CI runtime; fewer transient failures from repeated port-forwards
  • ➕ Less artifact volume
  • ➖ Loses attribution: failures may be unclear which hop introduced them
  • ➖ Misses intermediate breakages that recover later but still matter operationally

Recommendation: The PR’s approach (gate each hop on Postgres readiness + expected image, run Playwright per major, and partition artifacts per phase) is appropriate for producing upgrade evidence and making failures diagnosable. The custom wait/diagnostic helpers are justified here because they enforce image correctness and preserve failure context, which standard rollout-only checks typically don’t capture well during POSTGRESQL_UPGRADE=copy.

Files changed (10) +1128 / -17

Enhancement (1) +160 / -0
postgres.shAdd Postgres readiness, diagnostics, and collation-refresh helpers for CI +160/-0

Add Postgres readiness, diagnostics, and collation-refresh helpers for CI

• Introduces a new shell library to (1) wait for a Postgres pod to be Ready while validating the running image, (2) dump actionable diagnostics without being swallowed by command substitution, (3) log server_version as upgrade evidence, and (4) refresh collation versions across user databases with sanitized DB name parsing.

.ci/pipelines/lib/postgres.sh

Bug fix (2) +44 / -13
verify-redis-cache.spec.tsStop using shell-wrapped login for Redis port-forward in e2e +14/-5

Stop using shell-wrapped login for Redis port-forward in e2e

• Switches Redis port-forward setup to spawn kubectl directly (using the current CI context) and requires NAME_SPACE to be set. This supports clean teardown/restart of port-forwards across multiple Playwright phases in one CI job.

e2e-tests/playwright/e2e/verify-redis-cache.spec.ts

port-forward.tsKill port-forward process groups to prevent leaks between Playwright phases +30/-8

Kill port-forward process groups to prevent leaks between Playwright phases

• Runs port-forward subprocesses as detached process-group leaders and stops them by killing the whole process group (SIGTERM then SIGKILL). This prevents leftover shell/kubectl grandchildren from contaminating subsequent Playwright runs.

e2e-tests/playwright/utils/port-forward.ts

Tests (1) +101 / -4
ocp-pull.shOrchestrate PG15→PG16→PG18 Helm upgrades with per-hop Playwright + artifacts +101/-4

Orchestrate PG15→PG16→PG18 Helm upgrades with per-hop Playwright + artifacts

• Adds a multi-phase CI flow that deploys with PostgreSQL 15, upgrades to 16 and then 18 using copy-mode values, and runs Playwright after each major. Introduces helper functions to wait for expected Postgres images/Readiness, log server_version evidence, and save per-phase diagnostics and pod logs under unique artifact subdirectories.

.ci/pipelines/jobs/ocp-pull.sh

Other (6) +823 / -0
utils.shSource the new Postgres CI helper library +2/-0

Source the new Postgres CI helper library

• Wires the new postgres.sh library into the shared CI utilities so other scripts can use the readiness/diagnostics helpers.

.ci/pipelines/utils.sh

values_showcase_15.yamlAdd Helm values for showcase baseline on rhel9/postgresql-15 +165/-0

Add Helm values for showcase baseline on rhel9/postgresql-15

• Adds a dedicated values file selecting the product rhel9 PostgreSQL 15 image and baseline persistence/env configuration for the internal DB used in CI.

.ci/pipelines/value_files/values_showcase_15.yaml

values_showcase_16.yamlAdd Helm values for showcase running on rhel9/postgresql-16 +163/-0

Add Helm values for showcase running on rhel9/postgresql-16

• Adds the steady-state (non-upgrade) values file for PostgreSQL 16, used after completing the 15→16 copy-mode hop.

.ci/pipelines/value_files/values_showcase_16.yaml

values_showcase_16_upgrade.yamlAdd Helm values enabling POSTGRESQL_UPGRADE=copy for 15→16 hop +165/-0

Add Helm values enabling POSTGRESQL_UPGRADE=copy for 15→16 hop

• Adds an upgrade-only values file that switches the image to PostgreSQL 16 and sets POSTGRESQL_UPGRADE=copy to perform the chart-managed major upgrade.

.ci/pipelines/value_files/values_showcase_16_upgrade.yaml

values_showcase_18.yamlAdd Helm values for showcase running on rhel9/postgresql-18 +163/-0

Add Helm values for showcase running on rhel9/postgresql-18

• Adds the steady-state (non-upgrade) values file for PostgreSQL 18, used after completing the 16→18 copy-mode hop.

.ci/pipelines/value_files/values_showcase_18.yaml

values_showcase_18_upgrade.yamlAdd Helm values enabling POSTGRESQL_UPGRADE=copy for 16→18 hop +165/-0

Add Helm values enabling POSTGRESQL_UPGRADE=copy for 16→18 hop

• Adds an upgrade-only values file that switches the image to PostgreSQL 18 and sets POSTGRESQL_UPGRADE=copy to run the chart-managed major upgrade.

.ci/pipelines/value_files/values_showcase_18_upgrade.yaml

@github-actions

Copy link
Copy Markdown
Contributor

Image was built and published successfully. It is available at:

…grades

Capture the Backstage pod UID before each major hop, restart if needed, and
wait until the previous UID is gone before Playwright. Seed a unique catalog
Component on PG15 and verify it via API + UI after each upgrade.
@zdrapela

Copy link
Copy Markdown
Member Author

/test e2e-ocp-helm

@github-actions

Copy link
Copy Markdown
Contributor

Image was built and published successfully. It is available at:

In-cluster http://*.svc targets return 201 for location create but never
ingest entities under Backstage URL reader / SSRF restrictions. Use a
GitHub blob URL at PULL_PULL_SHA instead.
@zdrapela

Copy link
Copy Markdown
Member Author

/test e2e-ocp-helm

@github-actions

Copy link
Copy Markdown
Contributor

Image was built and published successfully. It is available at:

The Catalog table renders metadata.title ("PG Upgrade Data Proof"), so
exact-match on metadata.name never appears after search.
@zdrapela

Copy link
Copy Markdown
Member Author

/test e2e-ocp-helm

@sonarqubecloud

Copy link
Copy Markdown

@github-actions

Copy link
Copy Markdown
Contributor

Image was built and published successfully. It is available at:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant