Restore ASEC ED_VAL via a pinned sidecar and close the source-carry class - #421
Merged
Conversation
…lass The frozen census_cps h5s never carried raw ASEC ED_VAL (the archived pipeline read it directly: cps.py:1493 educational_assistance = person.ED_VAL, a direct carry), so the education-inputs stage failed the first full-scale base 17 stages in. Restore it with the weeks_unemployed sidecar pattern extended to every pooled income year: per-year pinned official Census archives (zip+member sha256/size/CRC and audit pins), fetch-or-local via --asec-education-source INCOME_YEAR=PATH, and an exact per-income-year PERIDNUM join with redundant identity checks that never predicts a value. Income years map to survey-year+1 archives — verified by 100.0% PERIDNUM coverage per year (adjacent vintages only overlap the ~33% rotation group, so a wrong pin fails the join loudly). Replaying the exact failed call (base-r3 checkpoint 016) with the sidecar: the stage completes and educational assistance lands at 2.09% weighted on both channels. That replay exposed the stage's other band as mis-set: qualified tuition is a PUF-only tax-detail chain target, so the ASEC channel is exactly zero BY DESIGN (partnership income, charitable donations, and mortgage interest all decompose the same way) and the pool-wide share is 0.288%, not the retired file's blended 2.64%. The gate now asserts the channel invariant directly — ASEC exactly zero (contamination check), PUF share in [0.002, 0.05], total in [0.001, 0.05] — and passes on the real frame. A new closure test makes this failure class static: every base-stage REQUIRED_SOURCE_COLUMNS entry must be covered by the pinned census_cps person columns, pool-constructed identity columns, a pinned sidecar restoration, or another stage's declared outputs. The sweep that motivated it found exactly one gap (ED_VAL); the three org_wages columns are release-time and excluded with documentation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
MaxGhenis
added a commit
that referenced
this pull request
Jul 13, 2026
#421 added the education sidecar to the locked run config but not to _stage_cli_args, so --stage all locked the parent context with the sidecar and spawned every child without it: the first child recomputed a config lacking asec_education_source, hit the run-context guard, and base-r4 died in six seconds. Forward the flag beside the other appended inputs, and pin the whole class with a round-trip regression test: _stage_run_config(parent) must equal _stage_run_config over the _stage_cli_args-reconstructed child argv, so any future input that enters the run config without entering the reconstruction fails in CI. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The incident chain
base-r3 (the first full-scale base past the QRF wall and the child-support gate) failed at stage 17:
education_inputsrequires raw ASECED_VAL, and no census_cps h5 ever carried it — the archived pipeline read it straight off the survey (cps.py:1493:educational_assistance = person.ED_VAL, a direct carry with no recode) while the frozen h5 generation dropped the column. A 23-constant sweep found exactly one such gap (the three org_wages columns are release-time and satisfied upstream there).The fix — the weeks_unemployed pattern, extended
LKWEEKShad the identical problem for one cohort and solved it with a pinned official-archive sidecar; this PR extends that pattern to every pooled income year:d2e00025…), 2024 (cdb39cda…), 2025 (318845a2…).weeks_unemployedalready reviewed, cross-validating the download path.--asec-education-source INCOME_YEAR=PATHconsumes local verified copies; omitted years fetch from census.gov and verify against the same pins. The fill is an exact per-yearPERIDNUMjoin with redundantPH_SEQ/P_SEQ/A_LINENOidentity checks, refuses uncovered keys/years, and never overwrites or predicts a value.The second find — a mis-set band, adjudicated with evidence
Replaying the exact failed call (base-r3 checkpoint 016 + the real sidecar): the stage completes,
educational_assistancelands at 2.09% weighted on both channels (survey-measured, channel-symmetric — as it should be). That replay exposed the stage's other band: qualified tuition showed 0.288% against a [0.003, 0.05] floor. Decomposition proves this is the designed value, not a deficiency: tuition is a PUF-only tax-detail chain target, exactly zero on the ASEC channel like every other PUF-only detail column (partnership income, charitable donations, mortgage interest all decompose identically on the full-scale pool), with the PUF half at the PUF-faithful 0.576%. The original floor encoded the retired file's blended 2.64% — a post-imputation file rate that isn't comparable to the pre-selection two-channel pool. The gate now asserts the channel invariant directly: ASEC exactly zero (a contamination tripwire), PUF share in [0.002, 0.05], total in [0.001, 0.05]. Verified passing on the real frame.The closure test
TestBaseStageSourceClosuremakes this failure class static: every base-stageREQUIRED_SOURCE_COLUMNSentry must be covered by the pinned census_cps person columns (162, frozen in the test), pool-constructed identity columns, a pinned sidecar restoration, or another stage's declared outputs. Adding a stage requirement with no provider now fails CI in seconds instead of a full-scale build in 35 minutes.Verification
🤖 Generated with Claude Code