Execution: rollout smoke-test — in-build fail-closed SHA gate + operator verify one-liner

[Kenny-Heitritter]

Parent

Map #16 · implements the smoke-test decision locked in #22.

What to build

Give operators a way to answer "did this codeq change reach all users on env X?" without shelling into pods, and prevent a mis-baked image from ever going live. The smoke-test asserts one claim: lab-base:latest carries the expected codeq SHA (bucket manifest.json sha == the codeq --version SHA read from inside the image).

Two identical checks, both envs:

• (A) Fail-closed in-build gate — an assert-codeq-sha step in build-lab-base[-prod], after the build/tag steps but before the deferred images: …:latest push. It compares the manifest SHA against codeq --version from the freshly built image and exits non-zero on mismatch. Because the :latest push is deferred until every step passes, a failed assertion aborts the build and :latest is never poisoned — the fail-closed guard on the staging auto-rebuild. Extend the lab-base download-artifacts step to also fetch latest/manifest.json.
• (B) Operator one-liner — verify-codeq-rollout.sh <env>: pull the live lab-base:latest from Artifact Registry, run codeq --version, SHA-compare to the bucket manifest. Registry-read auth only — no kubectl exec, no pod access. The prod go-live confirmation step and universal spot-check.

Lands in qbraid-infrastructure (cloud-build-qbraid-dstacks.tf) for (A); (B) as a small script + documented one-liner in CODEQ_DEPLOYMENT.md, with the "image-correct ⟹ live for all users" definition and pass/fail criterion. Full spec: #22.

Acceptance criteria

• build-lab-base and build-lab-base-prod each run an assert-codeq-sha step before the images: push; a SHA mismatch aborts the build with :latest untouched.
• The lab-base build fetches latest/manifest.json (not just the binary) to source the expected SHA.
• verify-codeq-rollout.sh <env> reports pass/fail for staging and prod using registry-read only (no pod shell).
• CODEQ_DEPLOYMENT.md documents check B's one-liner, the pass/fail criterion, and that already-running-pod convergence is by-construction on restart (per Decide running-pod refresh policy (layer 2) #21) — not measured or gated.

Blocked by

• Execution: publish the CODEQ_DEPLOYMENT.md anchor doc to dev #23 — publish the anchor doc (doc home for check B).
• Execution: stamp git SHA / build-id into codeq --version + manifest.json (both envs) #24 — SHA/build-id stamping (needs the stamped sha in manifest.json + codeq --version).

[Kenny-Heitritter]

Implemented across two PRs — open for review (not merged).

• Infra: qBraid/qbraid-infrastructure#620 (commit 253f41e) — fail-closed assert-codeq-sha step on build-lab-base[-prod] before the deferred images: push (SHA mismatch aborts the build; :latest never poisoned), download-artifacts also fetches manifest.json, and scripts/verify-codeq-rollout.sh <env> (registry-read only, no pod access).
• Doc: docs(codeq): execution-era CODEQ_DEPLOYMENT.md sections (#25–#29) #42 — check-B one-liner + pass/fail criterion + the by-construction passive-convergence note.

terraform validate + shellcheck clean; the codeq --version SHA parse verified against the (silent) codeq wrapper.