Replace all remaining policyengine-us-data artifacts with Populace

[MaxGhenis]

Goal

Make Populace the complete replacement for policyengine-us-data, including the national dataset, local/state/congressional-district outputs, release manifests, diagnostics, and the .py default stack contract. Once those are live, archive the old data repo rather than continuing old-stack fixes.

Scope

• Publish a Populace US release that passes release gates on the small national support frame while carrying the full current fiscal target surface, with congressional-district rows diagnostic-only until the support frame is designed to satisfy them.
• Add standard common-surface scoring so every candidate is compared against the incumbent on the same target registry before publication.
• Replace inherited policyengine-us-data artifacts in .py manifests with Populace-owned artifacts or explicit temporary inheritance notes.
• Provide Populace-owned state, local, and congressional-district dataset outputs or a clear loader strategy that supersedes policyengine-us-data/states/*, districts/*, and related local files.
• Preserve useful release-manifest and compatibility metadata, but keep incumbent/eCPS benchmark comparisons outside the live Populace repo.
• Close or migrate remaining old-repo issues/PRs only as Populace specs, source loaders, target profiles, diagnostics, or release-contract work. Do not port old implementation structure.

Current blockers / active work

• A guarded 32k+ target-surface build is queued from current main after Keep CD targets diagnostic for small releases #203; it will start when memory pressure from the active L0 benchmark clears.
• The full CD target surface should not block the small national release solely on CD zero-support rows, but those rows must remain visible in diagnostics.
• We still need an explicit artifact-level replacement table for every policyengine-us-data artifact currently referenced by .py or historical release manifests.

Acceptance criteria

• The current .py default US dataset is Populace, with verified compatible policyengine-us/policyengine-core versions and release manifest metadata.
• Populace has a documented replacement path for national, state, district, city/local, long-term, and auxiliary artifacts previously supplied by policyengine-us-data.
• Release gates include source coverage, formula-owned export assertions, target materialization diagnostics, common-surface incumbent comparison, and critical-target tolerances.
• All open policyengine-us-data issues/PRs are either linked to Populace issues, explicitly marked benchmark-only/out-of-scope, or closed as obsolete.
• policyengine-us-data is archived or made read-only with a README pointing to Populace and any benchmark repo for historical comparisons.

[MaxGhenis]

Current replacement inventory from 2026-06-27 audit:

policyengine.py still has Populace as the default national dataset, but the US release manifest still references 567 artifacts from policyengine/policyengine-us-data:

category	count	examples
congressional districts	436	districts/AK-01, districts/AL-01, ..., districts/WY-01
states	51	states/AK, states/AL, ..., states/WY
long-term CPS	75	long_term_cps_2026 through long_term_cps_2100
other national/local artifacts	5	cities/NYC, cps_2024, national/US, policy_data, small_enhanced_cps_2024

The current manifest has 571 datasets total: 4 from policyengine/populace-us, 567 from policyengine/policyengine-us-data. So replacing only the default national H5 is not enough for this issue to close.

Populace source code also still has a build-time dependency on policyengine-us-data through packages/populace-build/src/populace/build/us/sources.py and the us extra in packages/populace-build/pyproject.toml. The remaining imports cover CPS/ASEC helpers, SCF, SIPP/TIP/vehicle models, ACS storage, mortgage-interest helpers, and asset imputation helpers. That means the replacement plan needs both:

1. Populace-owned published artifacts for the state/local/CD/long-term/default auxiliary datasets that policyengine.py currently gets from the old data repo.
2. A Populace/Ledger-owned source-loading path so future Populace builds do not import old data repo loaders.

Operational status: a current-main full-surface Populace release build is queued behind the active L0 run via launchd label com.policyengine.populace.full-cd-release-rerun-20260627. It will use the 32,319-target CD-expanded surface, with CD gates diagnostic-only by default after #203.

[MaxGhenis]

Correction to my prior inventory comment: the Populace source-loader dependency part was from a dirty local checkout branch, not current origin/main.

Fresh audit on current Populace main (2deaa2c, after #203) finds no policyengine-us-data, policyengine_us_data, enhanced_cps, or abandoned-repo references in Populace source, package metadata, or uv.lock. The us extra currently depends on policyengine-us>=1.745.0,<2, h5py, microunit, and tables, not the old data package.

So the remaining blocker in this issue is now narrower and cleaner: the published/consumed policyengine.py US release manifest still inherits 567 artifacts from policyengine/policyengine-us-data, even though Populace main itself is no longer importing the old data package.

[MaxGhenis]

Builder audit update: current tools/build_us_fiscal_refresh_release.py writes only the national microdata artifact plus calibration/diagnostic files. _build_manifests() registers at most populace_us_2024, populace_us_2024_calibration, calibration_diagnostics, us_source_coverage, reform_validation, and demographics. There is no current manifest/publish path for replacement state H5s, congressional-district H5s, NYC/local H5s, long-term CPS files, cps_2024, policy_data, or small_enhanced_cps_2024. So the next code gap after the full-surface national build is an artifact expansion/publisher path, not source dependency cleanup.

[MaxGhenis]

Follow-up from the area-artifact implementation: the splitter can produce policyengine.py-compatible state/CD root artifacts, but a real-H5 spec check caught a current support blocker before we should publish CD artifacts. The current base H5 at out/cd-concept-budget-dbbdcec-512e-b2500-r2-20260627 has 436 raw CD geoids, but they are the old 117th district set: extras CA-53, IL-18, MI-14, NY-27, OH-16, PA-18, WV-03 and missing current keys CO-08, FL-28, MT-02, NC-14, OR-06, TX-37, TX-38. The incumbent policyengine.py manifest uses the current 436-key set, so Populace should not publish area artifacts from this support base. Root cause appears to be support assignment from SOI CD return-count Ledger facts (, ). For current state/CD artifacts we need support assignment from a current-boundary source, likely ACS/Census CD population, while SOI CD rows remain diagnostic/calibration rows until IRS releases current-boundary fiscal CD data.

[MaxGhenis]

Clean follow-up to the prior area-artifact note: the stale CD support appears to come from the SOI congressional-district facts currently represented as irs_soi.ty2023.congressional_district_2022.all_returns, sourced from the 22incd-style SOI CD file and tagged with the older 117th_congress geography vintage. The area-artifact splitter correctly rejects a real candidate H5 with CA-53; see #205 for the proposed Ledger/target-aging crosswalk from old CD vintages to the current policyengine.py 436-district surface.

[MaxGhenis]

PolicyEngine-only downstream split-H5 scan from 2026-07-01.

Confirmed PolicyEngine/populace#237 has merged the single-national-H5 release path. I scanned canonical local PolicyEngine/<repo> checkouts plus GitHub code search for policyengine-us-data/states, policyengine-us-data/districts, and the corresponding hf:///gs:// forms. Follow-up issues now filed or confirmed:

Existing / previously filed:

• Migrate impact computation off split state H5 files state-legislative-tracker#189
• Migrate Rhode Island microsim off split state H5 file ri-ctc-calculator#48
• Migrate district modal pipeline off split district H5 files working-parents-tax-relief-act#31
• Migrate WV analyses off split state/district H5 datasets wv-sb392-tax-cut#8
• Migrate state SPM extraction off split state H5 datasets policyengine-spm-decomposition#2
• Migrate LIHEAP analysis off split state H5 datasets liheap-repeal-dashboard#5
• Update district analysis examples away from split district H5s salt-cap-ny17#3
• Migrate district analysis off split state H5 datasets obbba-household-by-household#243
• Remove place fallback to split state H5s once Populace supports place scoping policyengine-app-v2#1079
• Remove regional H5 path templates from US Populace release contract policyengine.py#445
• Update microsimulation skills away from split state/district H5s policyengine-skills#28
• State-specific .h5 datasets undercount population by 30-65% analysis-notebooks#135 was already open and covers broken state-H5 usage.
• Make policyengine-api-v2-alpha runtime execution honor model and dataset version selection policyengine-api-v2-alpha#219 was already open and covers dataset-version/runtime selection; direct split importer refs remain in import_state_datasets.py.

Additional issues opened from org code search:

• Migrate child poverty microsim off split state H5 datasets child-poverty-impact-dashboard#49
• Migrate Georgia analyses off split state/district H5 datasets georgia-2026-tax-changes#1
• Migrate NC analyses off split state/district H5 datasets nc-stein-2027-tax-proposals#4
• Migrate NJ analyses off split state/district H5 datasets nj-ctc-increase#4
• Update tests away from split policyengine-us-data state/district URIs policyengine-sim-api#587
• Update agent docs away from split state/district H5 datasets policyengine-claude#162
• Migrate poverty dashboard fixtures off split state H5 datasets poverty-dashboard#5
• Update docs away from split policyengine-us-data state H5 example python-package-docs#2
• Migrate SC EITC calculator off split state H5 dataset sc-h3492-eitc-calc#9
• Migrate South Carolina analyses off split state/district H5 datasets south-carolina-2026-tax-changes#8
• Migrate state EITC/CTC analyses off split state H5 datasets us-state-eitcs-ctcs#25
• Migrate Utah analyses off split state/district H5 datasets utah-2026-tax-changes#8
• Migrate Washington analysis off split state H5 dataset wa-millionaires-tax#4

Notes:

• PolicyEngine/policyengine-api remote origin/master had no active split-H5 refs; the local checkout was on a stale branch with old refs.
• PolicyEngine/policyengine-bundles only has historical bundle manifests, so I did not file a migration issue.
• PolicyEngine/policyengine-us-data still has historical/release-promotion tests with states/ paths; the live deprecation tracker should decide whether to archive those as part of winding down the repo rather than migrating them to Populace.

[MaxGhenis]

Also added the old-repo cleanup tracker: PolicyEngine/policyengine-us-data#1175 for deprecating/archive-cleaning the split state/district H5 pipeline after Populace certification.

[MaxGhenis]

Data point from wiring axiom-oracles onto populace-us (TheAxiomFoundation/axiom-oracles#73, which now defaults its US oracle population to the certified artifact): on the sparse 57k release, county_fips and place_fips calculate empty for every household, and in_nyc recomputes false for all households (no county inputs), so nyc_income_tax totals $0B nationally — exactly the failure mode #34 predicted when the stored in_nyc was dropped without exporting county inputs. (An earlier version of this comment said all-true — that was a bug in my probe, corrected with a controlled repro; details on #34.) Consequence here: NYC-scoped oracle comparisons must keep the legacy per-city eCPS file until the spine carries place grain — the last eCPS dependency in that repo (TheAxiomFoundation/axiom-oracles#74). State-level scopes filter fine off state_fips.