Build E: certification-candidate staging build from post-#287 main (fix target_compilation OOM first)

[MaxGhenis]

Goal

Run Build E — the certification-candidate US staging build — from current main, which now carries the complete recipe: #266 immigration, #279 input-carry, #280+#287 full-surface chained target aging (default ON, 1,480 targets, ×1.1203 chained TY2022→TY2024; national AGI $15.29T→$16.62T, income tax $2.11T→$2.36T at the target level), #286 stuck-at-default input gate, #283 L2/concentration diagnostics, pinned Ledger bundle enforcement.

Two prerequisites (both builds B and D died before calibration)

1. Fix the target_compilation kill. Builds B (us-2024-immigration-266-b-20260702T114815Z) and D (populace-us-2024-32fcf6b-de00cae57dc7-20260702T145207Z, nonpreemptible, 220 GiB) both died at the same locus: mid-target_compilation on the 224,026-household pool, with 2 of 5 JCT reform materializations complete — D's death was a hard kill, most likely OOM (the driver's own warning puts dense worst-case near ~120 GB; observed exceeds 220 GiB), with the 10h timeout secondary. The JCT reform materialization step (full engine runs over the pool, despite Stream reform validation simulations #236's streaming) is the prime suspect. Options: chunk/stream the reform materialization further, checkpoint per-reform (the Add calibration warm-start and target-frame checkpoints #227 checkpoint machinery may already support resume — verify why the restart lost the materialization cache in B), or provision >300 GiB.
2. Inputs per Consume pinned Ledger artifacts and enforce the period contract #287's provenance contract: the pinned, hash-verified Ledger bundle (f6fa6152…) — do not substitute unpinned facts. Confirm the base pool H5 (populace-us-2024-cd-concept-budget-dbbdcec-512e-b2500-r2-20260627T022640Z, 224,026 hh) carries the Sparse-57k certified release zeroes untargeted input bases (IRA/HSA/SE-pension/childcare); CDCC and ALD reforms silently score ~$0 on bundle 4.18.8 #278 input columns so the Gate releases on input columns stuck at the engine default #286 gate passes with Gate persisted input mass and carry the #278 bases through the sparse pipeline #279's carry (if the base itself lacks them, the base needs a rebuild first).

Guardrails (unchanged)

Staging only (policyengine/populace-us-staging); never touch policyengine/populace-us, its latest.json, or root files; certification is a separate explicit human decision after the results comparison vs the published baseline (2025: net income $14.50T, tax $3.43T, AGI $16.01T, income_tax $2.15T, poverty 13.7%, citizens 100%, IRA/HSA/childcare inputs absent, OBBBA CTC-SSN Δ −$37.1B vs JCT).

Context: failure analysis on #204 (2026-07-03 comment); Modal runner state in the B/D orchestrator's STATE.md (session scratchpad).

[MaxGhenis]

Prerequisite 1 complete (#317 merged): per-reform materialization checkpoints are now durable across restarts. Root cause of build B's lost cache was checkpoint paths defaulting under the ephemeral --out dir — a Modal restart into a fresh out-dir orphaned the completed entries; --checkpoint-root decouples them. The #217 finer cache key reuses completed reforms across commit/registry bumps while invalidating on base-H5 sha, reform vector, PE-US version, period, and CD-crosswalk sha. Resume is TDD-proven (kill after 2 of 3 → restart computes only the third).

Box-sizing correction for the launch: the compile stage's realistic peak is ~5–9 GiB (hard ceiling ~25 GiB) — the ~120 GB figure in the driver's warning is the calibration loss matrix, a later stage D never reached. Size the box for calibration (~220 GiB held for the dense worst case), raise the 10h timeout, and rely on checkpoint resume for anything that still dies mid-compile.

[MaxGhenis]

Build E executed end-to-end overnight (2026-07-05, ~02:00–06:15 ET) — pipeline PROVEN, artifact NOT certifiable, diagnosis attached

What ran

Local (128GB box), staging-only, nothing touched prod HF. Base pool REBUILT from the recorded raw recipe (the pinned dbdcec pool is already-expanded AND missing all four #278 leaves — verified on the artifact): raw ASEC 2024+2023 pooled 50/50 → 112,013 hh → PUF support clone → 224,026 hh, dimensions matching the pinned pool exactly. First artifact ever built under the #308 weights audit (resolved_weight_kinds: {us_puf_tax_detail_support: "design"}, passed), geography ladder applied (119th-Congress CD vintage, sha-pinned), all four #278 leaves carrying mass. Base: out/base-e-20260705/ in the build-e-base worktree + summary JSON.

The facts feed (the f6fa6152 reconstruction)

The pinned bundle sha existed only in the dead orchestrator's session state. Reconstructed by refusal-directed search — six correct gate refusals, each fixed in turn: dirty-tree provenance → run-exactly-once clone guard → period contract (CBO leg missing) → period contract (SOI leg missing) → L0 mass-infeasibility at the library-default ratio 5 (certified recipe = 50) → reference ambiguity across the arch→ledger key rename (same source_record_id, two key namespaces). Final feed consumer_facts_builde_aging_v5.jsonl = the curated 37,053-fact surface + 24 CBO income_by_source projections (ty2023–26) + 21 national SOI table-1.1/1.4 actuals, deduped on lineage.source_record_id. A full multi-year splice (160k facts) was tried and REJECTED: it over-binds (25 pattern-bound CHIP families with zero support; income tax torn to +16%) — the curated feed is a selected target surface, not a lazy subset.

En route: ledger#83 (the #78 loud-drift guard caught ici/fact_book_table_30 + irs_soi/soca_table_1 unregistered on its first production run), and the #286 degenerate-gate anti-rot check demanded RETIRING the TANF/EITC exclusions because the #312 seeding now carries real signal (register updated on the build-e-base branch, plus reason'd exclusions for the un-imputed second_home_mortgage_* trio → #38).

Aging verified numerically

Raw ty2022 income-tax fact $2.105T × chained factor = $2.359T target — exactly the ×1.1203 this issue documents. #287's chain (SOI observed leg + CBO projected leg) resolves correctly from the v5 feed.

#317 checkpointing

Deployed in production (--checkpoint-root outside the repo tree — note: an in-repo checkpoint dir would re-dirty the tree and block resume). Multiple restarts crossed compile without loss.

Terminal result (run 20260705T085035Z)

Calibration completed; final release gates FAILED — honestly:

metric	Build E (224k pool)	certified sparse-57k
selected records	31,061	57,240
final loss	0.125	0.044
within 10%	48.4%	94.7%
ESS	5,728	—
realized max ratio	50.0 (pegged)	50

Critical misses: federal income tax +20.8% ($2.849T vs aged $2.359T), Social Security benefits +6.3% ($1.564T vs $1.471T). Plus 24 CMS CHIP state-enrollment targets with zero materialized support (the pool cannot express CHIP enrollment — #170's class; needs either an enrollment-support fix or a reviewed zero-support exclusion mechanism as flagged on #256).

Diagnostics: out/release-e-20260705/releases/populace-us-2024-builde-a86c947-20260705T{082123,085035}Z/calibration_diagnostics.json.

Recommendation

The 224k cd-concept pool was the B/D step-isolation base, not the certified lineage — the published sparse default descends from the 337k dense support (f0af251). This run proves every stage and gate of the modern pipeline on real scale; it also shows this pool cannot reach certification quality (ESS 5.7k at a pegged ratio cap = severe concentration; the L0 selection is starving). Proposed Build F: rebuild the 337k dense support on current main (same asec-mode reconstruction, larger clone budget per the f0af251 recipe) and run this exact release recipe from it. Separately worth a look before Build F: the ESS/anchor question is María's #309/#285 territory — refit_l2_anchor="design" may materially help the refit concentration here.

Feed shas + all intermediates preserved under ~/PolicyEngine/_builde-runtime/; every code change pushed on build-e-base (includes the asec_pool reconstruction, which must relocate out of us/ per #159 before merge).

[MaxGhenis]

Build F campaign report — certified-lineage rerun on current main (five attempts)

Build F set out to rerun the exact Build E release recipe but on the certified 337k dense-support lineage instead of the 224k experiment pool, on current main (HEAD ed78097) with this week's merged hardening. The base was reconstructed correctly and the pipeline was driven to its last gate. No artifact is certifiable this campaign, but the failures are now fully diagnosed: the dense parent is one mis-referenced gate from clean, and the sparse default is blocked on uncommitted informed-L0 selection machinery. Two issues filed (#326, #327), one code fix landed (#321), all deviations documented.

Phase 0 — the certified pool IS reconstructable (no guessing)

The certified pool = a 3-year raw CPS ASEC pool (2024+2023+2022), equal-thirds shares, mass anchored to target-year 2024, then a 2× PUF-tax-detail support clone. Evidence:

• The live default (sparse-l0-refit-57k, == current main HF ref) carries three source_years {2022, 2023, 2024} and two support channels {asec, puf_tax_detail} (frame-loader read).
• Record-count arithmetic closes exactly: raw ASEC households 55,762 (2024) + 56,251 (2023) + 56,839 (2022) = 168,852 → ×2 clone = 337,704, matching the "32,633 × 337,704" production problem in Sweep λ_L2 in the production US build: ESS-vs-accuracy frontier for the sparse L0+refit dataset #285.
• Shares = equal thirds is not a guess: it generalizes the committed 2-year 0.5/0.5 spine, is asec_pool._resolved_shares's own default for N sources, and (because the release builder unconditionally rescales the base to the Census 334.2M person benchmark) is the least-opinionated target-year-anchored choice. No 3-year spine exists in the repo on any branch; the certified 3asec run config was ad-hoc.
• The f0af251 tag in the Build F rec is loose shorthand for the lineage; the published f0af251 artifact is a 75,112-record June-20 intermediate, not the pool.

Base rebuilt and verified (base-f-20260705, sha 18833fb6…): 337,704 hh, channels balanced 168,852/168,852, source_year 2024/2023/2022, all four #278 leaves present with mass (IRA, SE-pension, childcare on their entities; HSA on tax_unit). Base gates pass: weights_audit (#308, us_puf_tax_detail_support=design), geography_ladder, immigration_composition.

The five attempts — gates peeled in sequence

The release evaluates gates in stages; each fix exposed the next layer. Because the export-stage mass-parity gate only runs after an export frame exists, attempts dying earlier never reached it.

1. Attempt 1 — died on reviewed-exclusion register drift: the build-e-base register edits (retire seeded takes_up_{tanf,eitc} exclusions now that Seed TANF and EITC take-up from administrative rates; engine-asserted take-up contract (#312) #315 seeds real signal; add the un-imputed second_home_mortgage_* trio → populace#38) were never merged. Replayed on build-f (b5980d2).
2. Attempt 2 (v5 + register fix) — sparse died on the pre-registered L0 mass-infeasibility (mass="conserve" infeasible at max_weight_ratio=5.0, 54,093 survivors); dense died on 20 CHIP zero-support targets.
3. Attempt 3 (v6 = v5 − 20 direct CHIP rows, sparse --max-weight-ratio 50) — sparse cleared L0 but still failed the same 20 CHIP + income tax +17.4%; dense failed the same 20 CHIP. Root cause (CHIP enrollment targets are concept-mismatched: CMS total_chip_enrollment includes M-CHIP; model materializes separate CHIP only (20 states zero-support) #321): the CHIP targets are tri-sourced — removing the direct 2024-12 rows didn't remove the targets because (a) surviving 2025-12 rows back-cast onto the 2024 period, and (b) _with_derived_chip_enrollment_targets re-derives CHIP (combined − Medicaid) exactly where direct rows are absent, so deletion armed the fallback.
4. Attempt 4 (v7 = M-CHIP CHIP removed at all periods + derivation skips M-CHIP state FIPS, with tests; sparse --l2-lambda 0.08) — CHIP gate cleared completely. But the λ=0.08 addition was a misread of the lam008 run-dir tag: Sweep λ_L2 in the production US build: ESS-vs-accuracy frontier for the sparse L0+refit dataset #285's frontier table shows the certified sparse arm ran at production defaults (λ=0); λ=0.08 produced +258% income tax and was reverted.
5. Attempt 5 (clean checkpoint room, λ default, dense-first) — the diagnostic run. Dense reached export and failed input mass parity; sparse failed critical income-tax/SS targets. Details below.

Dense parent (attempt 5, dense-8e99d08) — one gate from clean

Calibration on the certified pool is excellent and matches/beats the certified reference:

metric	Build F dense	certified f0af251 dense
final loss	0.04159	0.0423
within-10%	86.13%	86.4%
n_records	337,704	75,112
initial loss	0.20665	0.0542

Every structural gate passes: target_profile_coverage, base_population_scale (334.2M), immigration_composition (#266), degenerate_input_signal (#286 — CHIP now clean via #321), validation_input_coverage (#278/#279), eCPS parity (ratchet, #316), health_input_signal; base weights_audit (#308) and geography_ladder pass. calibration_diagnostics.json records release_gates.passed=True.

The only failure is the export-stage _export_input_mass_gate, which fires after the diagnostics are written. It compares the calibrated export frame to the raw pre-calibration base with a symmetric ±50% band, and flags 14 PUF-imputed columns — all gains, not losses (capital gains +230.7%, dividends +142.3%, charitable +88.9%, mortgage interest +54.1%, etc.).

Verdict: the masses are right; the gate is mis-referenced (→ #327). Comparing each flagged column's calibrated national mass to its SOI/CBO national target (the honest reference):

• qualified_dividend_income → CBO qualified-dividend target $354B, est $320B (−9.7%) — hitting the target.
• capital-gains family (LT/ST/non-Sch-D) → CBO net_capital_gain target $1.291T, est $0.798T (−38.2%) — even after +230% vs raw base, capital gains still undershoot CBO.
• partnership_self_employment_net_earnings → CBO net_business_income target $1.916T, est $1.917T (+0.1%).
• AGI control −0.9%.

CPS structurally under-reports capital/dividend/business income; PUF imputation seeds it low; calibration correctly scales it up toward SOI/CBO. The ±50%-vs-raw-base band asks calibration not to do its job. (miscellaneous_income's base is negative −$8.15B, making relative drift ill-defined.) Provenance confirms mis-application: the gate was added in #279, after certified f0af251 (Jun 20) and the live-default lineage; neither certified manifest records input_mass_parity (only base_population_scale mass_repair, which build-f passes); the certified 57k live default was built without this gate and would itself fail export-vs-raw-base on the same columns. There is no pre-calibration income-mass alignment step in the certified pipeline.

Recommended remedy (option a): reference a certified release for the export comparison (the release builder already has an --input-mass-reference-h5 path; the underlying input_mass_parity_gate already accepts reference totals) and/or thread reviewed_exclusions through _export_input_mass_gate for the 14 PUF columns with the SOI/CBO-target justification. Keep the loss/zeroing arm strict (that is the real #278 signature). Not --allow-input-mass-drift globally (it disables the whole gate, re-opening #278). Rejected alternatives: a base-side income-mass repair (no such step in the certified pipeline — inventing one deviates from certified lineage); "genuinely wrong masses" (refuted by the national-target evidence).

Sparse default (attempt 5, sparse-8e99d08) — blocked on uncommitted informed-L0

Cold L0 at λ default on the certified pool: selected 53,527 of 337,704, loss 0.1192, within-10% 50.2% — cold-L0 quality (Build E's 224k sparse was 0.125 / 48.4%). Two critical-target failures: federal income tax +19.9% (est $2.828T vs $2.359T) and Social Security +5.3% ($1.550T vs $1.471T).

The certified sparse headline (loss ~0.044, within-10% ~94.7%, λ=0) is not a from-scratch cold-L0 selection. Per its manifest it is a 1,500-epoch dense polish of a frozen, pre-selected 57,240 support (warm_start_calibration from refit-selected-current-surface-a7a77c4, ESS = the paper's 4,726). That informed-L0 selection artifact is not in the repo — it descends from an ad-hoc run. Cold from-scratch L0 on the raw pool starves and overshoots income tax. Build F sparse is blocked on the uncommitted informed-L0 selection / warm-start machinery — a Build G item. The certified 57k live default remains the laptop artifact meanwhile.

Documented deviations from the pure certified recipe

1. Feed v7 (v5 minus the M-CHIP total_chip_enrollment rows at all periods) + the M-CHIP derivation skip — a reason'd degenerate-target exclusion for a target-concept mismatch (CHIP enrollment targets are concept-mismatched: CMS total_chip_enrollment includes M-CHIP; model materializes separate CHIP only (20 states zero-support) #321), not a quality shortcut.
2. Sparse --max-weight-ratio 50 (certified default 5.0) — the Phase-0 pre-registered fallback for L0 mass="conserve" infeasibility at 5.0 on this pool. Documented, not iterated. (Moot for the verdict — sparse is blocked upstream on informed-L0.)

Defects found / fixed / filed

• CHIP enrollment targets are concept-mismatched: CMS total_chip_enrollment includes M-CHIP; model materializes separate CHIP only (20 states zero-support) #321 (open, code fix + tests landed) — CHIP tri-source concept mismatch; the derivation re-created the 20 unmaterializable targets whenever direct rows were removed.
• Export input-mass-parity gate references the raw base, flagging correct calibration gains on PUF-imputed columns #327 (filed) — export input-mass-parity gate references the raw base, flagging correct calibration gains on 14 PUF-imputed columns. Includes the export-vs-CBO evidence, provenance, and the proposed fix.
• Calibration-stage cache/checkpoint identity omits solver config (no solver_config_sha256) #326 (filed) — calibration-stage cache/checkpoint identity omits a solver_config_sha256. Framed accurately: not the cause here (the clean-room attempt 5 reproduced byte-identical, proving deterministic behavior, not stale cache), but the omission nearly misled the diagnosis. (TargetRegistry.version already content-hashes the feed, and the target-frame checkpoint is legitimately solver-independent, so the gap is narrow.)
• Register drift (attempt 1) — the reviewed-exclusion register edits were never merged from build-e-base; replayed on build-f. The Gate releases on input columns stuck at the engine default #286 anti-rot check correctly caught the staleness.
• λ misread (attempt 4) — lam008 run-dir tag misread as l2_lambda=0.08; reverted to the certified λ=0.

Cache-key components (for reproducibility)

base_dataset_sha256=18833fb6…; policyengine-us 1.752.2 / core 3.26.11; target_period 2024; seed 0; CD-crosswalk sha 383a6666…; feed v7 sha 735f326a…; reform vector = default US_JCT_TAX_EXPENDITURE_REFORMS. Base artifact: base-f-20260705/base_populace_us_2024_puf_support.h5 (18833fb6…).

Verdict

Not certifiable this campaign. The certified 3-year pool was faithfully reconstructed and calibrates to certified quality (dense loss 0.0416 / within-10% 86.1%, beating f0af251). The dense parent is blocked only by a gate that is mis-referenced for the dense-parent-from-raw-base case (#327); with that gate correctly referencing a certified release or carrying reviewed exclusions for the 14 PUF-imputed columns, the dense build passes on the certified pool. The sparse default is blocked on the uncommitted informed-L0 / warm-start selection machinery (cold L0 gives income tax +19.9%), which is a Build G reconstruction. Everything staging-only; prod HF and latest.json untouched.

[MaxGhenis]

---

Attempt 6 (pre-flight only, no run) + campaign closed

Attempt 6 was authorized as dense-only with one change — point the mass-parity gate at a certified reference instead of the raw base (implementing the #327 verdict). I ran the pre-flight as a dry analysis before launching, and it showed the run should not be launched as scoped:

1. The flag alone doesn't reach the failing gate. --input-mass-reference-h5 feeds only _input_mass_reference_gate (base-vs-reference); the gate that failed attempt 5 is _export_input_mass_gate(export_frame, base_frame, …), called unconditionally with base_frame. So the flag-only change would fail identically on export-vs-base and additionally arm a new base-vs-reference gate. Threading a certified reference into the export gate is a 1-line code change (tracked on Export input-mass-parity gate references the raw base, flagging correct calibration gains on PUF-imputed columns #327).

2. Corrected reference: 11/14 clean, 3 residual. Simulating export-vs-certified drift for the 14 columns (export masses = attempt-5's exact reported values; certified masses read live; tol ±50%, floor $1e9):

   • vs f0af251 (75k) → 5 fail (a bad reference: it is a June-20 different-vintage build with negative/tiny masses on some columns — short_term_capital_gains ref is negative, giving a +1397% artifact).
   • vs live default (57k) → 3 fail: first_home_mortgage_interest +62.5%, home_mortgage_interest +70.7%, miscellaneous_income −62.9%.
     The other 11 of 14 columns come cleanly in-band against the live default (capital gains, dividends, charitable, rental, tax-exempt interest, unemployment), confirming Export input-mass-parity gate references the raw base, flagging correct calibration gains on PUF-imputed columns #327: those were a gate mis-reference, not wrong masses.

3. The 3 residuals are real, not gate artifacts. base-f raw mortgage interest is $336B (~+8% vs certified $311B); cold dense calibration inflates it to $531B chasing the JCT deductible_mortgage_interest.revenue_loss target, which it still overshoots +59.7% (tgt $24.8B, est $39.6B). miscellaneous_income is a small sign-unstable residual (base −$7.9B). These are the same cold-vs-informed-selection gap that blocks sparse — now filed as Build G: reconstruct the informed-L0 / warm-start selection as committed machinery (blocks both sparse and dense defaults) #328 (informed-L0/warm-start reconstruction, which blocks both artifacts).

Verdict ratified: no artifact shipped this campaign, by choice. A passing dense build via reference-swap-plus-exclusions would require excluding the 3 genuinely-off mortgage/misc columns — papering over the cold-calibration overshoot rather than certifying clean. The honest fix for both artifacts is #328. The gates worked: they caught a concept-mismatched target (#321, fixed), a gate mis-reference (#327, 11/14 vindicated), and the missing informed-selection layer (#328) — and refused to certify until they're addressed.

Reusable asset: the reconstructed certified base 18833fb6 (3-year ASEC → 337,704 hh) calibrates to certified dense quality (loss 0.0416 / within-10% 86.1%); it is the input for Build G #328. Worktree build-f + _buildf-runtime/ left intact. Staging-only throughout; prod HF and latest.json untouched.