Skip to content

chore(ci): Read clickhouse image from versions json for CI - #70905

Merged
tpgilmore merged 2 commits into
masterfrom
chore_ci-use-var-for-clickhouse-version
Jul 15, 2026
Merged

chore(ci): Read clickhouse image from versions json for CI#70905
tpgilmore merged 2 commits into
masterfrom
chore_ci-use-var-for-clickhouse-version

Conversation

@tpgilmore

Copy link
Copy Markdown
Contributor

Problem

The turbo-tests product suite in ci-backend.yml hardcoded its ClickHouse server image (clickhouse/clickhouse-server:26.6.1.1193). Everywhere else in backend CI already derives the image from .github/clickhouse-versions.json, but these two pins were hand-maintained copies. Whenever local dev and hobby were bumped to a new ClickHouse version, the product tests kept running the old image until someone remembered to edit both the canonical workflow and its depot shadow. That is the silent drift the versions file exists to prevent.

Changes

Claude made these changes at my direction. turbo-tests now reads the image from the versions file instead of a literal:

  • Added a local_image output to the get_clickhouse_versions job, read from the JSON's local key with the same null guard the existing oldest_supported read uses.
  • Pointed both turbo-tests image references at ${{ needs.get_clickhouse_versions.outputs.local_image }} and added get_clickhouse_versions to that job's needs.
  • Mirrored the same change into the depot shadow (.depot/workflows/ci-backend.yml) so the shadow-drift check stays green.
-                  CLICKHOUSE_SERVER_IMAGE: clickhouse/clickhouse-server:26.6.1.1193
+                  CLICKHOUSE_SERVER_IMAGE: ${{ needs.get_clickhouse_versions.outputs.local_image }}

Note

local in the versions file is the 3-part tag clickhouse/clickhouse-server:26.6.1, not the 4-part 26.6.1.1193. That is a real rolling tag pointing at the same build today, and the compat matrix already pulls it, so there is no functional change now. From here on the product tests track whatever the versions file says.

How did you test this code?

No new automated tests, since this is CI config. I (Claude) could not run the actual turbo-tests job or a full CI run locally. What I did run:

  • bin/hogli lint:workflows (after syncing the flox venv): 6/6 checks pass across 100 workflows.
  • YAML parse of both files, plus a grep confirming no hardcoded image pins remain.
  • jq -r '.local' .github/clickhouse-versions.json returns clickhouse/clickhouse-server:26.6.1, and I confirmed that tag is real and pullable on Docker Hub.
  • Traced the skip path: when get_clickhouse_versions is skipped, the output is empty and docker-compose.dev.yml (which extends base) falls back to base's ${CLICKHOUSE_SERVER_IMAGE:-...} default. turbo-tests runs with always(), so the new needs edge cannot cause it to skip. This matches how the existing oldest_supported_image consumers already behave.

actionlint is not installed locally, so CI owns that check. I verified the needs reference and the output declaration by hand.

Automatic notifications

  • Publish to changelog?
  • Alert Sales and Marketing teams?

Docs update

No user-facing docs affected.

🤖 Agent context

Autonomy: Human-driven by @tpgilmore (agent-assisted)

Tool: Claude Code (Claude Opus 4.8). I followed the repo's authoring-ci-workflows skill (read its SKILL.md, per the CLAUDE.md requirement for any .github/workflows edit). No other skills applied.

Decisions:

  • Read from the JSON's local key rather than adding a new 4-part key. The versions file is already the source of truth and the plan was to move these pins onto its 3-part identifiers, so reusing local keeps one lever instead of two.
  • Exposed the value as a get_clickhouse_versions output rather than an inline jq in turbo-tests, to match how oldest_supported_image is already consumed and keep the JSON read in one place.
  • Left the skip-path fallback to docker-compose's existing default instead of hardcoding one, since the base default is itself kept current by the version-bump automation, so both paths stay on the promoted version.

This closes a lag where the product-test image had to be updated by hand after each ClickHouse bump. The versions file is populated by the existing infra sync, so pulling from it removes the manual step.

The turbo-tests hardcoded the ClickHouse server image, so it drifted from
the version local dev and hobby run whenever docker-compose was bumped.
Read it from clickhouse-versions.json's `local` key via a new
get_clickhouse_versions output, making the JSON the single source of
truth for the product-test image. Mirrored into the depot shadow to
keep the drift check green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@tpgilmore
tpgilmore requested a review from a team July 14, 2026 21:17
@pr-assigner-resolver-posthog
pr-assigner-resolver-posthog Bot requested a review from a team July 14, 2026 21:18
@greptile-apps

greptile-apps Bot commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

Comments Outside Diff (1)

  1. .github/workflows/ci-backend.yml, line 618-624 (link)

    P2 Failed Version Read Still Runs Tests

    When get_clickhouse_versions fails, this job can still run because always() is not paired with a success check for the new dependency. In that state local_image is empty, so the Docker Compose fallback runs product tests against the old default image while the version-read job is already red, which makes the test result misleading and burns CI time.

Reviews (1): Last reviewed commit: "chore(ci): Read clickhouse image from ve..." | Re-trigger Greptile

@trunk-io

trunk-io Bot commented Jul 14, 2026

Copy link
Copy Markdown

Static BadgeStatic BadgeStatic BadgeStatic Badge

View Full Report ↗︎Docs

@tpgilmore

Copy link
Copy Markdown
Contributor Author

Comments Outside Diff (1)

  1. .github/workflows/ci-backend.yml, line 618-624 (link)
    P2 Failed Version Read Still Runs Tests
    When get_clickhouse_versions fails, this job can still run because always() is not paired with a success check for the new dependency. In that state local_image is empty, so the Docker Compose fallback runs product tests against the old default image while the version-read job is already red, which makes the test result misleading and burns CI time.

Reviews (1): Last reviewed commit: "chore(ci): Read clickhouse image from ve..." | Re-trigger Greptile

The version in the docker-compose file should match this version at all times, in the unlikely event the fallback happens it should still be a valid test scenario.

@tpgilmore
tpgilmore merged commit 7fa80de into master Jul 15, 2026
235 checks passed
@tpgilmore
tpgilmore deleted the chore_ci-use-var-for-clickhouse-version branch July 15, 2026 14:55
@deployment-status-posthog

deployment-status-posthog Bot commented Jul 15, 2026

Copy link
Copy Markdown

Deploy status

Environment Status Deployed At Workflow
dev ✅ Deployed 2026-07-15 15:19 UTC Run
prod-us ✅ Deployed 2026-07-15 15:46 UTC Run
prod-eu ✅ Deployed 2026-07-15 15:48 UTC Run

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants