ACS local-area rebuild on the certified O lineage: multispine port, release chain, #398 role contract (#512)#528
Merged
Merged
Conversation
…donor (#512) Ports the buildl-acs-spine ACS modules (acs_pums/acs_sources/acs_inputs/ acs_transfer/acs_multispine/base_pool), the staging driver, and their test suites onto current main, with the 23-stage-base adaptations: - QRF plan: drop the stale partnership/s_corp exclusion (the base now disaggregates both), add the Schedule D capital-gain-distributions memo leg and the CDCC adult-care pair as declared families (87 -> 91 targets). - Geography: a donor carrying the certified block-ladder triple keeps its congressional district and county through pooling, with PUMA derived exactly from its 2020 tract via the ladder tract overlap; partial assignment fails loudly. Legacy donors keep the state-conditional draw. - Donor provenance: --donor-release-manifest verifies the donor H5 sha against its release manifest and pins the release identity into the staging summary; the sub-PUMA limitation is donor-mode aware. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…#512) The committed successor of the Build L runtime scripts, so a lineage refresh is one command: state-admin surface from the production compile path (RI Medicaid substitution included), household-chunked engine pass under the nullable-artifact contract, PUMA-ladder population marginals, jetsam-safe epoch-batched calibrate with resume, a per-spine SSI QA probe (the populace#403 signature, measured), a gate report with spine-composition evidence, the lineage reviewed-limitations register (inherited #507 SSI aged-band collapse, #393 miscellaneous-income defect, CD marginal vintage, ESS concentration, sparse-selection donor, mixed sub-PUMA coverage), and the non-default local-area release bundle with donor identity chain and refresh recipe. Publication stays a separate reviewed step; the tool never touches latest.json. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… guard (#398) validate_release_dir now dispatches on release_manifest.json's dataset_role: national_default (or absent) keeps the full historical contract unchanged; non_default_local_area gets its own contract - gates object with every gate passed, complete per-target diagnostics rows, provenance-chain source coverage (acs_sources / geography_ladder / transfer_coverage / donor_release), a reviewed-limitations register, an empty default_datasets map, and every artifact pinned to the immutable release id. Unknown roles are refused. publish_release additionally refuses update_latest for any non-default role - defense in depth beyond --no-latest, so a non-default release can never move the global pointer. First consumer: the buildo-acs-local rebuild (#512); retires the direct-Hub-commit workaround the Build L publish needed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…fied donor (#512) Verified against the O-1 release H5: partnership_self_employment_net_earnings carries 1,457 person rows, but s_corp_income is present with zero carriers (combined income still lives in partnership_income), so the transfer plan restores only the SE-earnings leg. The donor-coverage gate would otherwise hard-fail the staging run on an all-engine-default target. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…er plan (#512) The 23-stage base carries second_home_mortgage_{balance,interest, origination_year} as all-zero schema columns (verified on base-O: 0 of 462,014 tax units, vs 62,043 first-home carriers), so the donor-coverage gate correctly hard-failed the staging smoke on them. The mortgage surface consolidated onto the first-home leaves; declaring the second-home legs would transfer nothing but engine defaults. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The donor-coverage gate is enforced upstream (the staging driver hard-fails before transfer), so reaching finalize means it held; the #398 local-area contract requires every gate entry to carry a passed flag, which the end-to-end chain smoke caught. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… binding (#512) Adversarial cross-family review (sol) of the port found three blockers and ten majors; the confirmed set, fixed: - Schedule D CGD memo leg: never QRF-fitted. Derived post-transfer from the transferred parents at the packaged SOCA share with route exclusivity (an independent fit broke the base's accounting identity: both routes positive, memo without parent, memo above share x parent). - Adult-care: post-fit statutory reconciliation (qualifying carriers only, at most one per tax unit, deterministic keep-largest) + the statute-structure signal gate hard-fails on the transferred ACS frame. - Consumer-ready export: the published H5 gets the engine-pass contract applied to its bytes (input-schema projection + reviewed-null default fill, all recorded), so plain USSingleYearDataset/Microsimulation consumers load it directly — empirically the old shape crashed on NaN. The QA probe now loads the packaged bytes plain, doubling as the loadability proof; finalize gates on it. - Calibrated weights attach only after household-id vector equality between the lean checkpoint and the freshly loaded staging H5; resume checkpoints carry the staging sha and refuse foreign warm-starts; an already-complete resume no longer crashes. - Run identity (staging/ladder shas, household count, target digest) written at materialize, verified at calibrate/finalize/package; packaging refuses bytes the QA probe did not certify. - Geography preflight before the expensive transfer; integral-coercion checks and tract/county coherence on preserved donor geography. - Silent-surface-shrink guards: dropped ladder population cells hard-fail without --allow-partial-geography; compiled-away calibration targets hard-fail; declared-vs-compiled counts recorded. - Calibration gate additionally requires improvement over design weights (the cap criterion alone is near-tautological). - #398 contract: sha256sums.txt is a validated ledger (coverage, digests, safe names); typed/finite diagnostics checks; donor identity required; explicit invalid dataset_role values rejected; non-default publishes upload the whole bundle so the remote release satisfies its own contract. Declined for now (recorded): the memmap densification at checkpoint assembly (measured buildl behavior, 103GB peak proven on this box). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
CI's 'uv sync --all-packages' installs no extras, so the eight ported
tests that round-trip H5 fixtures guard with the repo's established
pytest.importorskip('tables') pattern.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The restore path reads pe-us variable metadata through the adapter; in CI's extras-free env the documented dtype fallback applies and the donor float dtype survives, so the test requires the engine like the other engine-metadata tests do. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…#512) Sol's merge-gate round-diff named three real gaps, all closed: - A release can never package dropped ladder population cells (the --allow-partial-geography smoke flag no longer reaches a shippable bundle), and spine_qa.json without artifact_sha256 refuses packaging instead of silently skipping the byte binding. - The per-column consumer fill ledger (consumer_reviewed_null_fills.json) ships in the bundle next to the materialize-time ledger. - Declared and compiled admin surfaces must agree exactly at materialize. - The five ACS unit-frame tests actually depend on microunit (the sanctioned tax-unit constructor extra), not pytables — CI's ImportError proved it; the earlier tables guard masked the real dependency. The published populace-us-2024-buildo-acs-local-77e2061-20260724T110908Z run satisfies every new hard gate retroactively (0 dropped cells, 3,974==3,974 admin specs, QA sha bound, consumer ledger present in its checkpoint evidence). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
#512) Absent spine_qa.json or consumer_export.json now refuses packaging outright instead of reading as vacuously green through the empty-dict load — a stale finalized summary can no longer pair with deleted or never-produced evidence. Both files (and the per-column consumer fill ledger) are unconditional bundle members, and the absent-evidence refusals are tested. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Jul 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Lands the tooling that built and published
populace-us-2024-buildo-acs-local-77e2061-20260724T110908Z(receipts on #512; #449 cross-post).What
buildl-acs-spinebranch (acs_pums / acs_sources / acs_inputs / acs_transfer / acs_multispine / base_pool + staging driver + suites), updated for the 23-stage certified donor:s_corp_incomeand the zeroed second-home mortgage legs excluded as verified all-default donor columns; adult-care pair added).split_component_by_share(an independent fit breaks the accounting identity).tools/build_us_acs_local_release.py(materialize → calibrate → qa → finalize → package): production compile path for the state admin surface, jetsam-safe chunked engine pass under the nullable-artifact contract, resumable epoch-batched calibrate, per-spine SSI QA that loads the packaged bytes plain (the consumer-loadability proof), run-identity binding across stages, and the non-default bundle with donor chain + one-command O-2 refresh recipe.non_default_local_areavalidation class (gates all passed, per-target diagnostics shape, provenance-chain coverage, validated sha256 ledger, artifacts pinned to the release id), strict rejection of invalid roles, a hard guard so non-default releases can never movelatest.json, and whole-bundle uploads so the remote release satisfies its own contract. National-default validation is unchanged (sol's differential run over 28 cached release dirs: identical outcomes).Review
Sol cross-family review (adversarial, committed-diff scoped) found 3 blockers + 10 majors; 13 fixed here, 1 declined with rationale (memmap densification — the measured buildl memory profile, 103 GB peak proven on the build box). Full adjudication on #512.
Evidence
Published artifact: 1,588,854 households; 4,461-target surface; within-10% 92.83% (buildl: 87.87% on 742 targets); ESS 14,382 (3× buildl); mass conserved 1.0; all 6 gates green; contract-validated;
latest.jsonverified untouched. The #403 weight-composition skew is essentially resolved (persons/hh 2.72 donor vs 2.66 ACS; buildl was 1.50 vs 2.93).Fixes #398. Progresses #512 (artifact published; pe.py repin: PolicyEngine/policyengine.py#488).
🤖 Generated with Claude Code