Recover fail-closed ECR manifest verification for active release capture

[alexeygrigorev]

Parent: #110
Observed on CI run: https://github.com/DataTalksClub/website/actions/runs/31329237384

Incident

All source, quality, selected Django, Playwright, container, and aggregate CI gates passed for main 77167686cee6ce9c23e542527de639625799f913. Automatic deployment then stopped before publication or service mutation in auto-capture-prior with:

release failed safely: active image manifest is missing from development ECR

The live service remains healthy on accepted source ffc0f37b8631710546c825e2ebd8de85d40c478f, VERSION 20260809-164941-ffc0f37, digest sha256:eb091aec0f7edc66f6493acf9b1b14a23d6ee8fac5c0888888789713daae5730. The same capture already proved the full-SHA tag, digest description, and VERSION tag resolve to that digest; only the subsequent BatchGetImage manifest lookup failed.

Scope

Make active-image manifest verification tolerate the observed ECR lookup form while retaining exact, fail-closed proof. Do not skip prior capture, publish before capture, accept a mutable/wrong tag, or weaken compensation eligibility.

Acceptance criteria

• Try the exact digest lookup first. If ECR does not return a usable manifest for that lookup, retry using the already-proven exact 40-character source-SHA tag.
• The fallback succeeds only when there are no ECR failures, exactly one image is returned, its imageId.imageDigest equals the expected immutable digest, and its manifest is nonempty.
• Wrong digest, missing/empty manifest, multiple/no images, malformed responses, and ECR failure results still stop before publication or service mutation.
• Unit/contract tests cover direct success, fallback success, fallback wrong digest, fallback failure, and absent manifest.
• Existing release identity, prior-pair, compensation, OIDC/current-main, and deployment contract tests stay green.
• After independent acceptance, deploy the exact main commit through CI and verify live health/source/VERSION/digest plus historical event-total behavior.

[alexeygrigorev]

Engineer report — ready for independent QA

Implementation is uncommitted in /home/alexey/git/dtc-website/.tmp/website-117-ecr-manifest-fallback, branch issue-117-ecr-manifest-fallback, base 77167686cee6ce9c23e542527de639625799f913.

Change

verify_image_digest_exists retains the existing exact full-SHA tag, exact digest, and VERSION-tag DescribeImages proofs. It tries BatchGetImage by exact digest first. Only when that result is unusable does it query the already-proven full 40-character source-SHA tag. Either result is accepted only if it is a dictionary with no failures, exactly one image, the returned imageId.imageDigest exactly equals the expected digest, and imageManifest is a nonblank string. Otherwise prior capture still fails before publication or service mutation.

No AWS permission, workflow trigger, OIDC role, publication, deployment, or service-mutation logic changed.

Verification

• Focused direct/fallback/fail-closed tests: 3 passed, including 5 adversarial fallback subtests.
• Deployment/release/identity tests: 213 passed, 351 subtests.
• Repository lint, formatting, mypy, OpenAPI, management parity, Django check, migrations check, and production deployment check: passed.
• git diff --check: passed.
• Candidate file hashes:
  • deploy/aws_gateway.py: 8fa424c47352dd30e7b10b84743437cb4753097c5c91b275f993e52c59580685
  • core/tests/test_deployment_workflow.py: 76e79e1ce2553e3e2e9175408df47e5058ef13b0163a38ae45c7d1d73d2e4af6

No commit, push, merge, deployment, or live mutation was performed. The failed run left the live service on accepted ffc0f37....

[alexeygrigorev]

Independent technical QA — FAIL

I reviewed the uncommitted candidate on branch issue-117-ecr-manifest-fallback at HEAD 77167686cee6ce9c23e542527de639625799f913. I did not edit/fix, commit, push, merge, deploy, accept, close, or change issue checkboxes.

Blocking finding: malformed failures shapes are accepted

AwsReleaseGateway._contains_exact_image_manifest() rejects response.get("failures") only when that value is truthy. A present but structurally invalid falsey value therefore passes validation. The ECR BatchGetImage service model defines both images and failures as lists, so {}, "", 0, false, and null are malformed failures values, not valid empty failure lists.

I reproduced the issue through the complete verify_image_digest_exists() path:

1. All three prior DescribeImages proofs returned the exact expected immutable digest.
2. The direct digest BatchGetImage returned the observed unusable ImageNotFound response.
3. The exact full-SHA fallback returned one image bound to the expected digest with a nonblank manifest, but used malformed "failures": {}.
4. verify_image_digest_exists() returned success instead of raising ReleaseContractError.

The helper independently returned True for each of these malformed response fields:

NoneType None True
dict {} True
str '' True
int 0 True
bool False True
full_method_accepted_malformed_failures_dict True

This violates the explicit criterion that malformed responses fail closed. The added invalid-fallback table covers wrong digest, a truthy failure list, missing/blank manifest, and multiple images, but it does not cover a malformed falsey failures shape.

Positive findings

• The code tries the exact digest first and calls the fallback only when the digest response is unusable.
• The fallback lookup key is the already-validated lowercase 40-character source SHA.
• Ordinary fallback success remains bound to the expected returned imageId.imageDigest and a nonblank string manifest.
• Wrong digest, truthy ECR failure, no/multiple images, missing manifest, whitespace-only manifest, malformed top-level response, malformed images, malformed image, and malformed imageId fail closed.
• The diff changes only the existing BatchGetImage read path and tests. It adds no AWS action or write permission and does not alter publication, ECS mutation, prior-pair, recovery, or compensation flows. The initial prior proof and pre-mutation recaptures continue to invoke exact image verification.

Verification

• Focused deployment/release/identity slice: 213 passed, 351 subtests passed.
• OIDC probe contract tests: 35 passed.
• make lint: passed.
• make format-check: 357 files passed.
• make typecheck: 279 files passed.
• make django-check, make database-portability-check, and git diff --check: passed.
• No UI changed, so browser screenshots are not applicable.

Frozen candidate evidence remained unchanged through QA: two tracked modified files, zero untracked files, tracked git diff --binary SHA-256 9a63d8db0f3161cd43f468127bafe859bcc6e2dc33c52774b40159b4b771b96e.

Verdict: FAIL. The candidate returns to engineering to make malformed falsey failures values fail closed and add a regression. No deployment is authorized from this verdict.

[alexeygrigorev]

Engineer remediation after QA FAIL

Fixed the malformed-response gap without changing lookup or AWS behavior. _contains_exact_image_manifest now requires failures to be present as an actual list and requires that list to be empty. Falsey malformed values ({}, null, "", 0, and false) are rejected.

Added each reported shape to the fail-closed fallback regression matrix. Focused result: 3 tests passed with 10 subtests. Ruff, formatting, and git diff --check pass.

Updated candidate hashes:

• deploy/aws_gateway.py: f45c20d655189fc0a26714f58c45bd73b88b85bb90ebafc005a7d05bf968bc41
• core/tests/test_deployment_workflow.py: ed1ed8b8e9a0d9cf70cf7fbd8e7eb6ccb8c0774115a063ad5abc62040838c400

Candidate remains uncommitted and undeployed, ready for independent re-QA.

[alexeygrigorev]

Independent remediation re-QA — FAIL

I re-tested the uncommitted remediation reported in #117 (comment) on branch issue-117-ecr-manifest-fallback, HEAD 77167686cee6ce9c23e542527de639625799f913. I did not edit/fix, commit, push, merge, deploy, accept, close, or change issue checkboxes.

Prior functional blocker: closed

I independently exercised verify_image_digest_exists() rather than relying only on the added unit table:

• Direct digest success made exactly one digest-keyed BatchGetImage call and did not fall back.
• An unusable digest response made exactly one fallback using the validated full 40-character source-SHA tag.
• A valid fallback succeeded only with an empty list in failures, exactly one image, the expected returned imageId.imageDigest, and a nonblank string manifest.
• All five reported malformed falsey failures values — {}, null, "", 0, and false — raised the safe ReleaseContractError through the full method.
• Fourteen additional wrong-digest, provider-failure, no-image, multiple-image, missing/empty/whitespace manifest, and malformed top-level/images/image/imageId/manifest shapes also failed closed.
• A malformed direct response was treated as unusable and could succeed only through a separately valid exact-tag fallback.

Independent probe summary:

direct_success=PASS
exact_sha_fallback=PASS
invalid_fallbacks_rejected=19
malformed_falsey_failures_rejected=5
malformed_direct_requires_fallback=PASS

The implementation remains read-only with respect to AWS: it adds only a second use of the already-allowed BatchGetImage action. Prior capture, exact prior-pair proof, pre-mutation recapture, recovery, and compensation control flow remain unchanged.

Blocking quality finding

make typecheck fails in the newly added malformed-response regression:

core/tests/test_deployment_workflow.py:5833: error: Need type annotation for "malformed_failures"  [var-annotated]
Found 1 error in 1 file (checked 279 source files)
make: *** [Makefile:40: typecheck] Error 1

The heterogeneous generator at line 5833 does not provide mypy enough type information under the repository settings. The issue requires existing quality and deployment contracts to remain green, so the candidate cannot pass technical QA while this gate is red.

Other verification

• Focused deployment/release/identity slice: 213 passed, 356 subtests passed.
• OIDC probe contracts: 35 passed.
• make lint: passed.
• make format-check: 357 files passed.
• make django-check, make database-portability-check, and git diff --check: passed.
• No UI changed; browser screenshots are not applicable.

The engineer-supplied file hashes reproduce exactly:

• deploy/aws_gateway.py: f45c20d655189fc0a26714f58c45bd73b88b85bb90ebafc005a7d05bf968bc41
• core/tests/test_deployment_workflow.py: ed1ed8b8e9a0d9cf70cf7fbd8e7eb6ccb8c0774115a063ad5abc62040838c400

Frozen candidate: two tracked modified files, zero untracked files, tracked git diff --binary SHA-256 2b37c86affc9e25610885ad81a947ac497a516d0e3e98ee56ff916a55dd878e5.

Verdict: FAIL. The security/functionality blocker is remediated, but the new regression must pass the project typecheck before the candidate can proceed. No deployment is authorized from this verdict.

[alexeygrigorev]

Engineer remediation after second QA FAIL

Added an explicit tuple[object, ...] annotation for the heterogeneous malformed-failure test inputs. Runtime production logic is unchanged.

Verification:

• focused active-image matrix: 3 passed, 10 subtests;
• full make typecheck: 279 source files, no issues;
• Ruff, formatting, and git diff --check: passed.

Updated test-file hash: 39fbc17e5ef36a631b0b0eb34b4efe74e78ecd86dff8f656cfd85c79d169aa64. Production-file hash remains f45c20d655189fc0a26714f58c45bd73b88b85bb90ebafc005a7d05bf968bc41.

Candidate remains uncommitted and undeployed, ready for independent re-QA.

[alexeygrigorev]

Final independent remediation re-QA — PASS

I re-tested the final uncommitted candidate on branch issue-117-ecr-manifest-fallback, HEAD 77167686cee6ce9c23e542527de639625799f913, after the type-annotation remediation in #117 (comment). I did not edit/fix, commit, push, merge, deploy, accept, close, or change issue checkboxes.

Diff and frozen evidence

The production implementation is byte-for-byte unchanged from the functionally accepted remediation. The only change since the previous QA candidate is the explicit test-only tuple[object, ...] annotation for the heterogeneous malformed-failure values.

The supplied hashes reproduce exactly:

• deploy/aws_gateway.py: f45c20d655189fc0a26714f58c45bd73b88b85bb90ebafc005a7d05bf968bc41
• core/tests/test_deployment_workflow.py: 39fbc17e5ef36a631b0b0eb34b4efe74e78ecd86dff8f656cfd85c79d169aa64
• tracked git diff --binary: 38d043540ca4f7c541082d8c648cb0f793a1b50bdd3aa9213ced313f6b6c96f7

The candidate remains exactly two tracked modified files with zero untracked files.

Acceptance verification

• Direct digest manifest success uses one digest-keyed lookup and does not invoke the tag fallback.
• An unusable digest response invokes exactly one fallback using the already-proven exact 40-character source-SHA tag.
• Fallback success requires an actual empty failures list, exactly one image, exact returned immutable digest binding, and a nonblank string manifest.
• Wrong digest, provider failure, absent/empty manifest, no/multiple images, malformed top-level/nested response shapes, and all five falsey malformed failures values ({}, null, "", 0, false) fail closed.
• The production diff only adds an ECR BatchGetImage read. It adds no AWS action or write permission and does not alter prior capture, publication ordering, ECS mutation, recovery, or compensation control flow.

Verification results

• Focused active-image matrix: 3 passed, 10 subtests passed.
• Deployment/release/identity slice: 213 passed, 356 subtests passed.
• make typecheck: 279 source files, no issues.
• make lint: passed.
• make format-check: 357 files passed.
• git diff --check: passed.
• The prior OIDC, Django/parity, and portability checks ran green against the same unchanged production-file hash. No UI changed, so screenshots are not applicable.

Verdict: PASS for independent technical QA. This verdict does not authorize or perform commit, merge, push, deployment, product acceptance, issue closure, or checklist changes.

[alexeygrigorev]

Product-manager acceptance — accepted

Accepted based on the final independent PASS: #117 (comment)

The change preserves exact tag/digest/VERSION proof, attempts digest-form manifest lookup first, uses only the already-proven full-SHA tag as a fallback, and remains fail-closed for all malformed, failed, empty, multiple, blank-manifest, and wrong-digest responses. It changes no permission, OIDC, publication, or service-mutation behavior. Full deployment/release and quality gates are green.

Authorized for the exact reviewed candidate to be committed, merged to main, and exercised by automatic CI/CD. Live acceptance still requires exact-SHA health and deployment evidence after CI.

[alexeygrigorev]

CI remediation

The first post-merge run 31330078036 stopped in the non-AWS quality stage because the frozen terminology inventory still contained the previous whole-file digest for core/tests/test_deployment_workflow.py. No publication or AWS mutation was reached.

Updated only that existing allowlist entry to the independently accepted test-file SHA-256 39fbc17e5ef36a631b0b0eb34b4efe74e78ecd86dff8f656cfd85c79d169aa64; class, reason, follow-up, and expected occurrence count remain unchanged.

Verification: terminology checker passed; 6 terminology tests + 4 subtests passed; git diff --check passed. This one-file follow-up is uncommitted and ready for independent confirmation.

[alexeygrigorev]

Independent CI-remediation QA — PASS

I independently reviewed the one-file uncommitted follow-up reported in #117 (comment) on branch issue-117-ecr-manifest-fallback, HEAD 1cf2471278f231ad8bbcc63fbbfbd2e16f350cec. I did not edit/fix, commit, push, merge, deploy, accept, close, or change issue checkboxes.

Exact binding and scope

• The diff changes exactly one line in _docs/compatibility/development-terminology-allowlist.json: the existing core/tests/test_deployment_workflow.py whole-file SHA-256.
• The new allowlist digest is 39fbc17e5ef36a631b0b0eb34b4efe74e78ecd86dff8f656cfd85c79d169aa64, which independently matches the current and previously accepted test-file bytes.
• The entry remains unique. Its class is still legacy_contract_test, reason text is unchanged, follow-up remains #94, and expected occurrence count remains 27.
• No path, finding class, reason, follow-up, or count was added, removed, or broadened.

The already accepted code hashes remain exact:

• production deploy/aws_gateway.py: f45c20d655189fc0a26714f58c45bd73b88b85bb90ebafc005a7d05bf968bc41
• test core/tests/test_deployment_workflow.py: 39fbc17e5ef36a631b0b0eb34b4efe74e78ecd86dff8f656cfd85c79d169aa64

Verification

• make terminology-check: passed (Development terminology inventory is classified).
• core/tests/test_development_terminology.py: 6 passed, 4 subtests passed.
• Focused active-image matrix: 3 passed, 10 subtests passed.
• JSON parse check: passed.
• git diff --check: passed.
• Candidate state: exactly one tracked modified file, zero untracked files.
• Current tracked git diff --binary SHA-256: ea50a0c26c7fd7b729427c845197eef3bdfc72a0c948b11ba8117adf4694f64b.

The production and test code are byte-for-byte identical to the prior accepted candidate, so the previously green deployment/release and typecheck results remain bound to the same code hashes; this follow-up only restores the frozen checker binding.

Verdict: PASS for this one-file CI remediation. This verdict does not perform or authorize commit, push, merge, deployment, product acceptance, issue closure, or checklist changes.

[alexeygrigorev]

Product-manager acceptance addendum

Accepted the one-file frozen-inventory update based on independent PASS: #117 (comment)

It binds the existing classification entry to the already accepted test file without changing its class, reason, follow-up, or count. Production and test code hashes remain exact. Authorized for commit and merge, followed by a fresh automatic run.

[alexeygrigorev]

Exact-main CI/CD and live acceptance

The accepted remediation is now proven end to end.

• Merged exact main source: b1edf2fb0049da6fb451699eac7caa4740b96c4b
• CI/deployment run: https://github.com/DataTalksClub/website/actions/runs/31330262019 — success
• resolve-release, classification, selective Django, Playwright, container, quality, aggregate gate, prior capture, publication, and deployment all passed.
• The previously failing auto-capture-prior job passed against the live ECR state using the fail-closed manifest logic.
• Live https://web.dtcdev.click/health/ready returns ready with source b1edf2fb0049da6fb451699eac7caa4740b96c4b, version 20260809-185529-b1edf2f, and immutable image digest sha256:f61dbfc25df7f2f2c813410e80f29786edb4e651cf6fc2e698c419c6ed3f9f81.
• A representative live event detail returns 200 under the development noindex, nofollow policy. It correctly omits registration copy because the protected historical sources have not been reconciled/activated; that separate HUMAN gate remains on Import historical registration totals safely and show aggregate event counts #112 and was not bypassed.

All #117 criteria are satisfied. Closing this bounded CI/CD regression; #112 remains open for its protected-source lifecycle gate.