Skip to content

Fill missing current CD targets from state totals - #209

Merged
MaxGhenis merged 1 commit into
mainfrom
codex/cd-state-total-proxy-20260627
Jun 27, 2026
Merged

Fill missing current CD targets from state totals#209
MaxGhenis merged 1 commit into
mainfrom
codex/cd-state-total-proxy-20260627

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Jun 27, 2026

Copy link
Copy Markdown
Contributor

Summary

  • synthesize source-vintage CD proxy facts from matching SOI state facts when a crosswalk source state has a single statewide/at-large source geography and no SOI source-CD facts
  • limit proxying to shapes already present in true SOI source-CD facts, including SOI return universe, to avoid state-only target expansion
  • update the CD hierarchy completeness profile to all 436 current 119th Congress districts, including one-district states and DC

Validation

  • uv run --python 3.13 --package populace-build --group dev python -m pytest packages/populace-build/tests/test_us_congressional_district_vintage.py packages/populace-build/tests/test_us_congressional_district_geography.py packages/populace-build/tests/test_us_fiscal_targets.py packages/populace-build/tests/test_us_fiscal_refresh_builder.py packages/populace-build/tests/test_us_puf_support_base_builder.py -q
  • uv run --python 3.13 --package populace-build --group dev ruff check packages/populace-build/src/populace/build/us_runtime/congressional_district_vintage.py packages/populace-build/tests/test_us_congressional_district_vintage.py packages/populace-build/tests/test_us_fiscal_targets.py packages/populace-build/src/populace/build/us_runtime/fiscal_targets.py tools/build_us_fiscal_refresh_release.py tools/build_us_puf_support_base.py
  • python3 -m json.tool packages/populace-build/src/populace/build/us/fiscal_target_references.json >/dev/null && git diff --check
  • Real Ledger/crosswalk compile: 33,127 total specs; 24,776 CD specs; 436/436 current 5001900US geographies; no old-prefix CD specs; 56 SOI CD measure IDs
  • Translated support distribution: 436 rows across 51 jurisdictions; no missing/extra current CD geoids

Review cycle

  • First subagent review found mixed-source suppression, confounded double-counting test, and SOI universe matching gaps.
  • Fixed all three, reran focused checks, and second subagent review reported no actionable findings.

@MaxGhenis
MaxGhenis merged commit dbd50ca into main Jun 27, 2026
4 checks passed
@MaxGhenis
MaxGhenis deleted the codex/cd-state-total-proxy-20260627 branch June 27, 2026 20:46
This was referenced Jun 27, 2026
Closed
Merged
MaxGhenis added a commit that referenced this pull request Jul 19, 2026
@MaxGhenis @claude
Build the 117th->119th CD vintage crosswalk from Census sources
4ae7a00 
The CD geography-vintage translation (#207/#208/#209) consumed a crosswalk
passed as an external CLI path with no versioned, reproducible, or cited
artifact in the repo. This adds that artifact and its generator, and makes it
the packaged default.

The crosswalk is built by a single-vintage block overlay, so no 2010<->2020
block bridge is needed:

- old (117th) district of each 2020 block: the 2020 Block Assignment File CD
  layer (BlockAssign_ST{fips}_{usps}_CD.txt), which carries the 116th-Congress
  plan (identical district geography to the 117th) on 2020 tabulation blocks;
- current (119th) district of each 2020 block: the 119th BEF (NationalCD119.txt),
  the same source the block ladder already uses;
- weight: 2020 P.L. 94-171 POP100 per block (the block ladder's
  parse_pl_geo_blocks convention).

Both district assignments are read on the same 2020 blocks weighted by the same
2020 block populations, so each old district's population is redistributed
across the current districts it overlaps and never invented. Population is the
correct default basis (apportionment and equal-population redistricting are
population operations); ACS income/tax proxy weights for fiscal targets are a
documented future refinement.

The committed national crosswalk covers all 436 current 119th-Congress districts
from 436 source districts, with exact per-state population conservation over all
331,449,281 people in the 50 states + DC (zero unmatched or cross-state). The
apportionment-shrunk districts (CA-53, IL-18, MI-14, NY-27, OH-16, PA-18, WV-03)
appear only as sources; the new districts (CO-08, FL-28, MT-02, NC-14, OR-06,
TX-37, TX-38) appear as populated targets. Montana's at-large district splits
~50/50 into MT-01/MT-02, as equal-population districts require.

The derived crosswalk is a regenerable build artifact, not a Ledger fact -- the
fact-vs-computed boundary of PolicyEngine/chronicle#71; the same declared-consumer-
side-transform pattern applies to the Belgian NIS-code vintage work in
PolicyEngine/chronicle#69.

Files:
- congressional_district_vintage_crosswalk.py: pure parsers (BAF CD layer, CD
  BEF) and the population-weighted join with conservation diagnostics.
- tools/build_us_congressional_district_vintage_crosswalk.py: download +
  cache + provenance orchestration, mirroring build_us_block_ladder_artifact.
- us/congressional_district_vintage_crosswalk.csv (+ .provenance.json + .md):
  the committed artifact, per-source SHA-256s, and the data-source doc.
- congressional_district_vintage.py: packaged-default loader helpers.
- build_us_fiscal_refresh_release.py: default to the packaged crosswalk when CD
  targets are requested and no path is passed.

Closes #205.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
MaxGhenis added a commit that referenced this pull request Jul 19, 2026
@MaxGhenis @claude
Build the 117th->119th CD vintage crosswalk from Census sources (#288)
ee8f7fc 
* Build the 117th->119th CD vintage crosswalk from Census sources

The CD geography-vintage translation (#207/#208/#209) consumed a crosswalk
passed as an external CLI path with no versioned, reproducible, or cited
artifact in the repo. This adds that artifact and its generator, and makes it
the packaged default.

The crosswalk is built by a single-vintage block overlay, so no 2010<->2020
block bridge is needed:

- old (117th) district of each 2020 block: the 2020 Block Assignment File CD
  layer (BlockAssign_ST{fips}_{usps}_CD.txt), which carries the 116th-Congress
  plan (identical district geography to the 117th) on 2020 tabulation blocks;
- current (119th) district of each 2020 block: the 119th BEF (NationalCD119.txt),
  the same source the block ladder already uses;
- weight: 2020 P.L. 94-171 POP100 per block (the block ladder's
  parse_pl_geo_blocks convention).

Both district assignments are read on the same 2020 blocks weighted by the same
2020 block populations, so each old district's population is redistributed
across the current districts it overlaps and never invented. Population is the
correct default basis (apportionment and equal-population redistricting are
population operations); ACS income/tax proxy weights for fiscal targets are a
documented future refinement.

The committed national crosswalk covers all 436 current 119th-Congress districts
from 436 source districts, with exact per-state population conservation over all
331,449,281 people in the 50 states + DC (zero unmatched or cross-state). The
apportionment-shrunk districts (CA-53, IL-18, MI-14, NY-27, OH-16, PA-18, WV-03)
appear only as sources; the new districts (CO-08, FL-28, MT-02, NC-14, OR-06,
TX-37, TX-38) appear as populated targets. Montana's at-large district splits
~50/50 into MT-01/MT-02, as equal-population districts require.

The derived crosswalk is a regenerable build artifact, not a Ledger fact -- the
fact-vs-computed boundary of PolicyEngine/chronicle#71; the same declared-consumer-
side-transform pattern applies to the Belgian NIS-code vintage work in
PolicyEngine/chronicle#69.

Files:
- congressional_district_vintage_crosswalk.py: pure parsers (BAF CD layer, CD
  BEF) and the population-weighted join with conservation diagnostics.
- tools/build_us_congressional_district_vintage_crosswalk.py: download +
  cache + provenance orchestration, mirroring build_us_block_ladder_artifact.
- us/congressional_district_vintage_crosswalk.csv (+ .provenance.json + .md):
  the committed artifact, per-source SHA-256s, and the data-source doc.
- congressional_district_vintage.py: packaged-default loader helpers.
- build_us_fiscal_refresh_release.py: default to the packaged crosswalk when CD
  targets are requested and no path is passed.

Closes #205.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Move crosswalk artifact to us_runtime/data; default builder crosswalk

CI (wheels job) surfaced two issues:

1. The `us/` country package is a spec-only directory: the governance tests
   (test_spec_only_country_packages) and country_spec.py require it to contain
   only .json resources, all declared in country_package.json. Move the
   crosswalk CSV, its provenance JSON, and the data-source doc to a new
   us_runtime/data/ package (us_runtime is exempt from the spec-only rule), and
   re-anchor the packaged-default loader at populace.build.us_runtime.data.

2. test_cd_targets_require_vintage_crosswalk asserted the release builder errors
   when CD targets are requested without a crosswalk; the builder now defaults
   to the packaged crosswalk, so rename/rewrite the test to assert the default
   is applied and an explicit path still overrides it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Correct crosswalk help text to us_runtime.data location

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Normalize crosswalk weights; keep population masses; strict packaged contract

Cross-model review findings (gpt-5.6-sol adversarial pass on #288):

1. The committed CSV's weight column held raw pair populations (per-source
   sums 543k-1.08M, not 1.0) while the PR text and column name promised
   weights — correct only because the loader silently re-normalized any
   positive mass. The artifact is now self-describing: pair_population
   carries the 2020 P.L. 94-171 mass (conserving state totals exactly)
   and weight is that mass over the source district's total, summing to
   1.0 per source in the raw file (worst deviation 2.2e-16).

2. The generic loader accepted duplicate pairs (silently aggregated) and
   arbitrary positive magnitudes. Duplicates now raise; when
   pair_population is present the weight column is cross-checked against
   it (1e-9), so a corrupted weight column cannot load.

3. Nothing enforced the published redistricting geometry on the packaged
   artifact. Default-path loads now run a strict contract: exactly 436
   districts per side across 51 jurisdictions, contiguous per-state
   rosters, weights in (0,1] summing to 1 per source, and identity rows
   (weight 1.0) for the seven both-vintage at-large jurisdictions
   (AK/DE/DC/ND/SD/VT/WY). External override files keep the permissive
   any-positive-mass semantics.

Tests pin the raw-CSV contract (not just loader-derived shares), the MT
50/50 split as shares, duplicate and inconsistency rejection, and the
packaged validator's failure modes. Docs and provenance sha updated;
generator emits both columns.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@MaxGhenis