Skip to content

feat(promote-release): use crane tag so per-arch moving tags stay signed - #197

Merged
sydorovdmytro merged 4 commits into
mainfrom
dmytrosydorov/devops-1083-promote-release-crane
Jul 23, 2026
Merged

feat(promote-release): use crane tag so per-arch moving tags stay signed#197
sydorovdmytro merged 4 commits into
mainfrom
dmytrosydorov/devops-1083-promote-release-crane

Conversation

@sydorovdmytro

@sydorovdmytro sydorovdmytro commented Jul 23, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

What

Switches the promote-release retag primitive from docker buildx imagetools create to crane tag.

Why

docker buildx imagetools create is digest-preserving only when the source is already a multi-arch index. On a bare single-platform manifest (the per-arch tags, e.g. :vX.Y.Z-amd64) it wraps the manifest in a new index, changing its digest and orphaning the digest-scoped cosign signature.

That blocks promoting the per-arch moving tags (latest-amd64, latest-fips-arm64v8, {major}-amd64, ...). GHCR pull stats show these are heavily used (the -amd64 group ~1,637 pulls vs ~1,849 for the multi-arch :latest on vcluster-pro), so they can't be silently dropped — see DEVOPS-1083.

crane tag re-points a tag at the exact same manifest digest for both single-platform manifests and multi-arch indexes. One uniform path now covers the whole moving-tag matrix, per-arch included, with signatures intact.

Changes

  • src/action.sh: retag imagetools create → crane tag; pre-flight imagetools inspect → crane digest.
  • action.yml: install crane (imjasonh/setup-crane, pinned); crane reads the docker config the existing GHCR docker/login-action writes, so no auth change. Per-arch usage documented on the images input.
  • Tests: docker_mock.bash → crane_mock.bash; added a per-arch-suffix retag test. 46 bats tests, actionlint + zizmor clean, docs regenerated.

Validation

The imagetools-vs-crane claim and the per-arch fix are verified live against a throwaway experiments repo (real GHCR + real cosign keyless), and this PR is exercised end-to-end via a caller in vClusterLabs-Experiments. Exact test matrix documented in a PR comment below.

DEVOPS-1083

@sydorovdmytro
feat(promote-release): use crane tag so per-arch moving tags stay signed
7d4afb1 
docker buildx imagetools create is digest-preserving only for an already
multi-arch index. On a bare single-platform manifest (per-arch tags like
:vX.Y.Z-amd64) it wraps the manifest in a NEW index, changing the digest and
orphaning the digest-scoped cosign signature. That blocks promoting the
per-arch moving tags (latest-amd64, latest-fips-arm64v8, ...), which GHCR
pull stats show are heavily used.

Switch the retag primitive to `crane tag`, which re-points a tag at the exact
same manifest digest for both single-platform manifests and multi-arch
indexes -- one uniform path covering the whole moving-tag matrix, per-arch
included, with signatures intact. Pre-flight existence check moves from
`imagetools inspect` to `crane digest`. action.yml installs crane
(imjasonh/setup-crane) and crane reads the docker config the existing GHCR
login writes, so no auth change is needed.

Verified live (real GHCR + cosign): crane preserves per-arch and index
digests and cosign verify passes on the retagged per-arch tag, while
imagetools changes the per-arch digest and cosign verify then fails.

DEVOPS-1083
@sydorovdmytro

sydorovdmytro commented Jul 23, 2026

Copy link
Copy Markdown
Contributor Author

Validation — exactly what was tested

Two layers: unit (mocked, in-repo CI) and end-to-end (real GHCR + real cosign, driving this branch's action).

1. Unit (bats, runs in PR CI — ✅ green)

  • docker_mock.bash → crane_mock.bash; the mock now records crane tag <src> <newtag> and crane digest <ref>.
  • All 47 tests pass, including a new per-arch test asserting suffix entries -amd64 / -fips-arm64v8 retag <image>:<version><suffix> → latest<suffix> / <major><suffix> / <major>.<minor><suffix>.
  • Dry-run assertion updated to the crane command form.
  • actionlint + zizmor clean; docs regenerated (check-docs green).

2. End-to-end (real, not mocked)

Repo: vClusterLabs-Experiments/devops-1083-promote-crane-e2e. A caller workflow (workflow_dispatch) that:

  1. Builds a production-shaped fixture at v1.0.0 with crane: four bare single-platform manifests (-amd64, -arm64v8, -fips-amd64, -fips-arm64v8) plus two multi-arch indexes (v1.0.0, v1.0.0-fips). Media types asserted (image.manifest vs image.index).
  2. Signs with cosign keyless (--recursive, so each index digest and its per-arch child digests are signed) — matching prod's digest-scoped OCI-referrer signatures.
  3. Calls promote-release@dmytrosydorov/devops-1083-promote-release-crane for real (dry-run: false) with the full vcluster-pro-shaped matrix: multi-arch + fips index + all four per-arch tags (6 images entries).
  4. Asserts results.

Results (run 30006092627):

  • All 18 moving tags (6 entries × latest/1/1.0) resolve to the exact digest of their v1.0.0 source — per-arch and index alike. E.g. latest-amd64 == v1.0.0-amd64 (sha256:c64c687…), latest-fips-arm64v8 == v1.0.0-fips-arm64v8 (sha256:5cd72f3…), latest == v1.0.0 (sha256:9e6f619…).
  • cosign verify passes on every crane-retagged tag checked: latest, latest-fips, latest-amd64, latest-arm64v8, latest-fips-amd64, latest-fips-arm64v8, 1, 1.0-fips — i.e. signatures survived the retag for both per-arch manifests and indexes.

Why this is the fix (contrast, verified separately)

In vClusterLabs-Experiments/devops-1083-crane-perarch the same per-arch source was retagged with docker buildx imagetools create: digest changed (c64c687… → 91bed47…, it wrapped the bare manifest in a new index) and cosign verify then FAILED — the regression this PR avoids. imagetools on an existing index preserved the digest (why the old code worked for multi-arch but would have silently broken per-arch).

Not exercised here (unchanged by this PR)

oss-repo release promotion and Homebrew tap patching were left out of the e2e — this PR only changes the docker retag primitive; that logic is untouched and covered by the bats suite.

Cleanup owed (need delete_repo)

devops-1083-promote-crane-e2e and devops-1083-crane-perarch (plus the three earlier repos).

loft-bot
loft-bot reviewed Jul 23, 2026
View reviewed changes
Comment thread .github/actions/promote-release/test/action.bats
@sydorovdmytro
chore(promote-release): skip crane install on dry-run
94603d2 
Match the guard on the GHCR login step (if: inputs.dry-run != 'true').
crane is only invoked on a real run (DRY_RUN != "true"), so installing it
during an exact dry-run is wasted work; the guard keeps the two setup
steps consistent.
@sydorovdmytro
test(promote-release): assert dry-run prints the full retag matrix
842d894 
The dry-run test spot-checked only the first planned `crane tag` line, so an
early loop exit that only manifested under dry-run would still pass. Pin the
second entry's line too, so the plural "prints planned retags" claim holds
across entries.
@sydorovdmytro
sydorovdmytro marked this pull request as ready for review July 23, 2026 13:29
loft-bot
loft-bot reviewed Jul 23, 2026
View reviewed changes
Comment thread .github/actions/promote-release/action.yml
Comment thread .github/actions/promote-release/README.md
@sydorovdmytro
docs(promote-release): fix stale test stub reference (docker -> crane)
f074336 
The Testing section still said the bats suite runs with a stubbed `docker`
on PATH; this branch deletes docker_mock.bash and stubs `crane` instead.
@sydorovdmytro
sydorovdmytro merged commit b63990f into main Jul 23, 2026
9 checks passed
@loft-bot loft-bot mentioned this pull request Aug 4, 2026
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@loft-bot loft-bot loft-bot left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@sydorovdmytro @loft-bot