Checkpoint and cache identity omit the staged frame: reform-vector cache ignores seed, constraint matrix unfingerprinted, manual materializer bumps

[MaxGhenis]

Summary

Several build-identity keys omit inputs that change results, and the code already works around the gaps by hand. This is the input-side complement to #326 (which is specifically the missing solver_config_sha256 — not duplicated here) and #314 (crosswalk pinning, since fixed in the checkpoint key).

The gaps (each verified on main, 8264c49)

1. The target-frame checkpoint hashes the raw H5, not the staged frame it actually materializes from.
_target_frame_checkpoint_identity (tools/build_us_fiscal_refresh_release.py:1291-1315) keys on: base H5 SHA, pe-us version, seed, period, registry version, crosswalk SHA. But materialization runs on base_frame after immigration, take-up, hours, SNAP, eligibility, pregnancy, SCF wealth, ACA, and Medicaid stages have rewritten it. The code acknowledges this hole explicitly — tools/build_us_fiscal_refresh_release.py:161-167:

# 2: the medicaid_take_up stage (populace #331) changed base_frame's
# takes_up_medicaid_if_eligible before target-frame materialization, ...
# the checkpoint identity hashes the on-disk base dataset, not the staged frame,
# and would otherwise silently reuse pre-stage frames.
TARGET_FRAME_CHECKPOINT_MATERIALIZER_VERSION = 2

Every future stage edit must remember to bump this integer by hand; forgetting reuses a stale checkpoint silently.

2. The reform-vector cache excludes seed while the vectors are computed on a seed-dependent frame.
REFORM_VECTOR_CACHE_CONTEXT_KEYS (:155-160) is only {base sha, pe-us version, period, crosswalk}; commit/seed/registry are intentionally excluded (:1184-1187) so calibration-only reruns reuse the cache. But the per-household reform vectors are evaluated on the post-source-stage frame (:6124), and the take-up / SCF-wealth / pregnancy stages draw seeded Bernoullis — so two runs with the same base and row count but different --seed share cache entries computed under the other seed's frame, pairing stale reform vectors with a freshly computed baseline. The materializer-version constant guards the target-frame checkpoint but there is no equivalent key on the reform cache (only TARGET_MATERIALIZATION_CACHE_SCHEMA_VERSION).

3. The constraint matrix is not fingerprinted, and incumbent comparison keys on a hash that excludes it.
_target_surface_payload (packages/populace-calibrate/src/populace/calibrate/diagnostics.py:130-149) hashes target rows (names/periods/measure selectors/values); the matrix contributes only rows/columns/nnz. _assert_incumbent_target_surface_matches (:1822-1850) accepts incumbent diagnostics on target_surface.sha256 equality alone — an engine change that moves coefficients (A) while preserving the target rows (b) is treated as the same surface.

4. CLI-overridable donor inputs skip identity and provenance.
The default SCF extract is SHA-pinned in scf_wealth.py, but a --scf-summary-extract path is loaded directly (:5950-5961); its digest enters neither the checkpoint identity nor the build/release manifests. Same family: _download_base_h5 (:1596-1604) fetches the mutable root with no revision (the observed SHA is recorded after the fact, so the identity is sound, but the input is not re-fetchable at that identity), and the Ledger --ledger-facts-sha256 / --ledger-manifest-sha256 pins are optional with bare-JSONL feeds accepted (:593-617).

5. The exact selected support set is not propagated into release lineage.
SelectionReport.as_manifest() (packages/populace-build/src/populace/build/us_runtime/warm_start_selection.py:124-147) drops identities_sha256 (the selection-source manifest embeds and verifies it on load, but the release manifest's selection_source payload carries only mode/join key/counts/source provenance). The US country package (us/country_package.json) commits no selection resource. A release therefore does not record which records were selected in a directly comparable form.

Fix sketch

One canonical build-run identity (the umbrella #326 also feeds into):

• add a canonical post-source-stage frame hash and derive both the target-frame checkpoint key and the reform-vector cache key from it (this retires the manual materializer integer and closes the seed hole at once);
• include CSR indptr/indices/data (or a content hash of them) plus stable record IDs in target_surface, and require engine identity in incumbent comparisons;
• record CLI-supplied donor/extract SHAs in the build manifest and require them in the checkpoint identity;
• carry identities_sha256 through as_manifest() into the release manifest.

Acceptance criteria

• Two runs differing only in --seed cannot share reform-vector cache entries.
• Editing any source stage invalidates the target-frame checkpoint without a manual constant bump (post-source frame hash in the key).
• target_surface distinguishes two calibrations whose targets match but whose constraint coefficients differ.
• A CLI-supplied SCF extract's SHA-256 appears in the build manifest.
• The release manifest records the selection identities_sha256 when a selection source is used.

Cross-links

• Calibration-stage cache/checkpoint identity omits solver config (no solver_config_sha256) #326 (solver-config identity — complementary, deliberately not duplicated here)
• CD vintage crosswalk is an unpinned build input; July-1 release surface not reproducible #314 (crosswalk was the same class of gap; its fix pattern applies)
• Make target materialization checkpoints reusable across calibration-only runs #217 (checkpoint reuse across calibration-only runs — reuse safety depends on these keys)
• Informed-init L0 selection: warm-start reselection for pool changes (ACS/ASEC ingest forcing function) #346 (informed-init reselection will add more seed-dependent stages ahead of materialization)
• Stale TANF degenerate-input exclusion on main aborts every default US build before calibration (Build I fix never ported) #377 (cross-register consistency — same lesson: local guards, no cross-artifact identity)

---

Surfaced by an external architecture review (Codex/gpt-5.6-sol, 2026-07-09); citations independently verified.

[MaxGhenis]

Identity spec v1 (conductor-authored; implementation routes to Codex after this is agreed)

The design principle the review's field list misses: identity maximalism recreates volatility as permanent cold builds. Every field below is classified as either IDENTITY-KEYED (changes bust caches and mint a new build identity) or PROVENANCE-RECORDED (logged in manifests for audit, never keyed). The test for keying: can this field change the numbers in the artifact? If it only changes how we describe the run, it is recorded, not keyed.

BuildRunIdentity — keyed fields

• base input: repo id + revision + full artifact sha256
• facts feed: file sha256 + declared supersession version
• target registry: version hash (already exists) AND the compiled spec-count
• country-spec fingerprint: hash of the resolved spec resources actually consumed (not the whole repo tree)
• donor artifacts (SCF/SIPP/etc.): each sha256
• selection: manifest sha + identity-set sha (when sparse)
• seeds + stage configuration that reaches math (batching does NOT key — see below)
• engine identity: policyengine-us + policyengine-core exact versions
• solver config: optimizer, lr, epochs, λ, mass mode, ratio caps (Calibration-stage cache/checkpoint identity omits solver config (no solver_config_sha256) #326)
• canonical post-source-frame hash: one content hash of the staged frame (the review is right that raw-H5 hashing is insufficient)

Provenance-recorded, NOT keyed

• populace commit (recorded; keying it would cold-start every build on comment-only commits — the frame hash + solver config already capture anything that matters; if a code change alters results, the frame hash or outputs change and parity tests catch it)
• batch sizes, worker counts, memory knobs (must be result-invariant BY CONTRACT — enforced by a batch-invariance test, not by keying)
• hostnames, timestamps, wall-clock, pressure telemetry

Constraint-system fingerprint (binds selection/weights/registry)

sha256 over: CSR indptr ‖ indices ‖ data (canonical dtype/order), target vector, stable record-ID array (not positions), registry version, engine identity, materializer implementation version. Warm-start loads REQUIRE matching fingerprint or an explicit --rebase-warm-start that re-joins by stable IDs and logs the delta. Certified sparse builds pin exact selected identities.

Solver-stage tolerance policy

Downstream comparisons against solver outputs (weights, diagnostics) use declared tolerance bands derived from the measured seed-noise floor (#307), recorded in the gates report; data/export stages compare byte-identical. A parity claim must state which regime it used.

Acceptance: identity object serialized into every checkpoint, cache key, release manifest, and gates report; changing any keyed field provably busts the target-frame and reform caches (test); changing any recorded-only field provably does NOT (test); batch-invariance test on a fixture build.

[MaxGhenis]

Empirical support for the canonical-frame-hash field (from the Build J re-cert, 2026-07-09): a byte-identical base builder rerun on identical inputs produced data-identical output (summary byte-identical except self-sha; base_household_weight_total exact to 134690323.34024483) but a DIFFERENT file sha256 (0b50660a… vs 18833fb6…) — HDF5 serialization nondeterminism. Raw-file hashing would have needlessly minted a new build identity and cold-started every cache; the canonical content hash is the right keyed field, with the file sha recorded as provenance only.