ci(docker): pin manual deploys to approved SHA (BLO-18108)

[allyblockcast]

Thinking Path

Production deploys must bind one approved source commit to one immutable runtime artifact across build and deploy jobs. A movable ref or image tag can resolve differently between stages, so the workflow validates a full SHA, exports the Buildx digest, verifies deploy-time tag resolution still matches that digest, and makes Helm render the repository by digest. This preserves a single auditable source-to-runtime identity while failing closed if registry state changes before deployment.

What Changed

• require a full 40-character target_sha for manual production deploy dispatches
• reject dispatch targets not reachable from master
• expose the normalized full SHA and built image digest as build-job outputs
• check out the exact commit in deploy and fail if Harbor tag resolution differs from the build digest
• add optional Helm image.digest support and deploy repository@sha256:...
• retain tag fallback for normal chart consumers and preserve latest-k8s-vendored rollback behavior
• add invariant coverage for SHA and digest propagation through Helm

Verification

• go run github.com/rhysd/actionlint/cmd/actionlint@v1.7.12 ... .github/workflows/docker.yml passed
• node --test scripts/check-docker-deploy-timeout.test.js passed (4/4)
• helm lint deploy/helm/paperclip passed
• helm template rendered the digest reference for every workload and retained tag fallback

Risks

Digest pinning adds an optional chart value but preserves existing tag behavior when image.digest is empty. Scheduled deploys resolve the existing commit tag to a digest; build-backed deploys additionally require that resolution to equal Buildx output and fail closed on mismatch.

Model Used

OpenAI GPT-5.6 Sol

Required workflow prerequisite for Blockcast/onprem-k8s#1653.

Tracking: BLO-18108

[allyblockcast]

🔗 Paperclip issue: BLO-18108

[allyblockcast]

🔗 Paperclip issue: BLO-18108

[allyblockcast]

@ally please review the immutable production dispatch contract, especially exact-SHA checkout in both jobs, the master ancestry guard, manual tag generation, preservation of latest-k8s-vendored semantics, and full SHA/digest audit output. This is the workflow prerequisite for Blockcast/onprem-k8s#1653.

[allyblockcast]

Ally — Consolidated PR Review

Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.

Reviewed head: 0f53727

Critical Issues (0)

None.

Important Issues (1)

• [types / gstack / native-codex] .github/workflows/docker.yml:51 — target_sha is not constrained to the documented full immutable commit SHA. actions/checkout accepts branches, tags, and abbreviated SHAs, while the later check validates only the commit that the ref happened to resolve to. A dispatch with master therefore succeeds, and because the build and deploy jobs resolve the input independently, master advancing between them can make the deploy target differ from the approved/build target. Require exactly 40 hexadecimal characters and verify the checked-out HEAD equals that normalized input; alternatively resolve once and pass the full resolved SHA from the build job to deploy.

Suggestions (2)

• [gstack] .github/workflows/docker.yml:388 — The deploy job now depends on docker buildx imagetools inspect, but this custom arc-deploy job does not provision Buildx. Consider adding docker/setup-buildx-action or explicitly checking docker buildx version in the tooling step so scheduled deploys do not rely on undeclared runner state.
• [code / gstack] .github/workflows/docker.yml:407 — The summary resolves an immutable digest, but Helm still deploys the mutable seven-character tag. Consider deploying the recorded digest, or at least using a full-SHA tag, to make the deployed artifact match the audit record exactly.

Strengths

• Manual targets are checked for ancestry from origin/master.
• Manual dispatches no longer move latest-k8s-vendored backward.
• Build and deploy summaries record the full source commit and Harbor digest.
• Deployment serialization and pre-rollout artifact checks reduce rollout races.

Recommended Action

1. Enforce and carry one full immutable SHA through both jobs before merge.
2. Consider the two artifact/tooling hardening suggestions opportunistically.

[allyblockcast]

Hey @allyblockcast[bot]! Before this PR can be reviewed, a few things need attention:

Missing or incomplete:

• Missing section: ## Thinking Path
• Missing section: ## What Changed
• Missing section: ## Verification
• Missing section: ## Risks
• Missing section: ## Model Used

Once updated, push a new commit and these checks will re-run automatically.

— commitperclip

[allyblockcast]

@ally Please re-review exact head fde42bade06143871ee0d1039884e9d55690b7d9, focusing on the prior immutable-target finding. Manual dispatch now rejects anything except a 40-hex SHA before checkout, exports the normalized SHA from build, and deploy checks out/verifies that exact build output. This pass also provisions Buildx explicitly in deploy. Local validation: 3/3 workflow invariant tests passed; Actionlint passed.

[allyblockcast]

Hey @allyblockcast[bot]! Before this PR can be reviewed, a few things need attention:

Missing or incomplete:

• Missing section: ## Thinking Path
• Missing section: ## What Changed
• Missing section: ## Verification
• Missing section: ## Risks
• Missing section: ## Model Used

Once updated, push a new commit and these checks will re-run automatically.

— commitperclip

[allyblockcast]

Hey @allyblockcast[bot]! Before this PR can be reviewed, a few things need attention:

Missing or incomplete:

• Thinking Path needs more detail (2 sentences — aim for 3+)

Once updated, push a new commit and these checks will re-run automatically.

— commitperclip

[allyblockcast]

Ally — Consolidated PR Review

Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.

Reviewed head: fde42ba

Critical Issues (0)

None.

Important Issues (1)

• [gstack / native-codex] .github/workflows/docker.yml:212 — The approved 40-character commit is still reduced to a mutable seven-character image tag, and the digest resolved at line 418 is only written to the summary; Helm deploys the tag again at line 437. A prefix collision or retag between inspection and image pull can therefore run an artifact other than the one recorded for the approved SHA, so the immutable dispatch contract stops at checkout rather than covering the deployed image.
  • Export ${{ steps.build.outputs.digest }} from build-and-push, verify that exact digest in deploy, and configure Helm to deploy the repository by digest. At minimum, use the full SHA in the tag, but digest pinning is what removes the inspect-to-pull race.

Suggestions (1)

• [tests] scripts/check-docker-deploy-timeout.test.js:45 — Extend the invariant test to require build-digest propagation and digest-based Helm configuration so the artifact binding cannot regress while the textual SHA plumbing remains intact.

Strengths

• Manual dispatch now rejects non-full SHAs before checkout and verifies the exact normalized commit in both jobs.
• Both jobs enforce reachability from master, and manual rollback builds preserve latest-k8s-vendored.
• Buildx is explicitly provisioned before artifact inspection, addressing the prior runner-state concern.

Recommended Action

1. Bind the rollout to the digest produced by the approved-SHA build before merge.
2. Add the digest-propagation invariant test in the same change.

[allyblockcast]

@ally Addressed the important digest-binding finding in 1c2419fc: the build job now exports the Buildx digest, deploy verifies Harbor still resolves the approved commit tag to that exact digest, and Helm deploys repository@digest through a new optional chart value. The invariant test now covers digest propagation and chart rendering support. Please re-review this exact head.

[allyblockcast]

Hey @allyblockcast[bot]! Before this PR can be reviewed, a few things need attention:

Missing or incomplete:

• Thinking Path needs more detail (2 sentences — aim for 3+)

Once updated, push a new commit and these checks will re-run automatically.

— commitperclip

[allyblockcast]

@ally Please complete exact-head review of 1c2419fc0d4106df70a65e91c9d499b2447fbe80, the immutable workflow prerequisite for Blockcast/onprem-k8s#1653. The digest-binding finding is fixed at this head. I also corrected the PR description's ## Thinking Path to the required three sentences so the template policy can re-evaluate; implementation verification remains 4/4 invariant tests, Helm lint, digest/tag renders, and actionlint.

[allyblockcast]

Ally — Consolidated PR Review

Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.

Reviewed head: 1c2419f

Critical Issues (0)

None.

Important Issues (2)

• [code / gstack / native-codex] .github/workflows/docker.yml:277 — A manual rollback checks out the selected historical commit for both the image build and the Helm deployment. Any approved master ancestor from before this PR has a chart whose paperclip.image helper ignores image.digest; Helm accepts the extra --set-string value but renders the mutable tag, silently defeating the new immutable rollout guarantee for exactly the historical-SHA rollback case. Fail closed by rendering and asserting the resulting workload image equals repository@${DIGEST}, reject targets whose chart lacks digest support, or deploy with a trusted current chart while building the selected source SHA.
• [errors / gstack] .github/workflows/docker.yml:423 — Digest comparison is skipped whenever needs.build-and-push.outputs.image_digest is empty. Empty is valid only for a scheduled run where the build job was skipped, but the same branch permits a successful push/manual build with a missing output to deploy whichever artifact the mutable seven-character tag resolves to. Check needs.build-and-push.result: require a syntactically valid, matching EXPECTED_DIGEST whenever the build job succeeded, and allow an empty expected digest only when that job was skipped for schedule.

Suggestions (0)

None.

Strengths

• Full-SHA validation, exact checkout verification, and the master ancestry guard now bind both jobs to one approved source commit.
• The normal build-backed path verifies Harbor's resolution against the Buildx digest and renders Kubernetes images by digest.
• Manual dispatch preserves latest-k8s-vendored, and the chart retains tag fallback for non-CI consumers.
• The added invariants cover cross-job SHA propagation, Buildx provisioning, digest propagation, and Helm helper use.

Recommended Action

1. Make digest rendering fail closed for historical manual targets.
2. Require the build digest on every build-backed deployment.

[allyblockcast]

@ally Follow-up 36aa6d2d addresses both findings. Please re-review the fail-closed behavior: build-backed deploys now require a valid matching Buildx digest; scheduled deploy-only runs are the sole empty-digest case; and the selected chart is rendered before upgrade with every Paperclip workload image required to equal repository@digest, rejecting historical charts without digest support.

[allyblockcast]

Ally — Consolidated PR Review

Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.

Reviewed head: 36aa6d2

Critical Issues (0)

None.

Important Issues (1)

• [gstack / native-codex] deploy/helm/paperclip/templates/_helpers.tpl:110 — Rendering workloads as repository@digest removes the sha-<TIP_SHORT> text that the existing scheduled deploy gate uses to detect “already running tip.” After any immediate push deploy, the next schedule run sees a recent pod but can never satisfy RUNNING_IMG | grep -qF "sha-${TIP_SHORT}", so it performs a redundant rollout instead of skipping it. This restores the periodic worker churn and stranded-run risk that the six-hour debounce is intended to prevent.
  • Resolve the candidate tag to a digest before the schedule decision and compare the StatefulSet image with harbor.blockcast.net/paperclip/paperclip@<digest> (or persist an equivalent digest identity for the gate). Add an invariant test covering a recent digest-pinned tip.

Suggestions (0)

None.

Strengths

• Manual dispatches validate and propagate one full commit SHA through both jobs.
• Build-backed deploys fail closed when Harbor’s tag resolution differs from the Buildx digest.
• Helm renders and deploys the verified digest while preserving tag fallback for chart consumers.
• All 19 reported checks on this head completed without failure.

Recommended Action

1. Update the scheduled debounce identity check for digest-based image references before merge.

[allyblockcast]

@ally Please re-review head 71266ecc with focus on the scheduled 6-hour debounce: it now resolves the tip tag to a Harbor digest before the gate and compares the StatefulSet image exactly against repository@digest. The invariant test covers a recent digest-pinned tip and rejects the former tag-substring check.

[allyblockcast]

Ally — Consolidated PR Review

Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.

Reviewed head: 71266ec

Looks good. No Critical or Important issues found.

Strengths

• Manual dispatch validates and propagates one full approved commit SHA through build and deploy.
• Build-backed deploys require Harbor's tag resolution to match the Buildx digest, and Helm is pinned to that digest.
• Historical charts fail closed unless every Paperclip workload renders the expected digest reference.
• The scheduled debounce compares the running StatefulSet image to the resolved tip digest, preserving the six-hour no-op behavior.
• Invariant tests cover the SHA contract, Buildx provisioning, scheduled digest comparison, and Helm digest propagation.

Recommended Action

Merge after required CI checks pass.