Publish a small calibrated national sample (populace_us_20k) for Colab/CI/tutorials

[MaxGhenis]

Request

Publish a small, calibrated national Populace US sample (e.g. populace_us_20k, ~20k households) as a first-class, registered dataset, so tutorials, Colab, and CI can run a national microsimulation without the full file.

Motivation

The full populace_us_2024 (224,026 households / 572,780 people, 1.2 GB) is too heavy for free Google Colab: a single national household_net_income run peaked at ~21 GB RSS in testing. That blocks hands-on population analysis in the IMA 2026 tutorial (and is generally awkward for CI / quick iteration).

A uniform 20k-household subsample (household integrity preserved, weights rescaled) already reproduces full-file headline aggregates to ~1–3% (net income to ~0.02%) — so a calibrated 20k artifact would be accurate enough for headline cost/poverty/decile work while being light and fast.

Requirements

1. Size: ~20k households (~50k people). Mirror the full file's HDF5 schema/keys exactly so it loads through the same path.
2. Household integrity: when a household is sampled, keep all members and their tax_unit / spm_unit / family / marital_unit rows.
3. Calibration: reweight with the existing L0 / microcalibrate machinery (the bundle already exposes populace_us_2024_calibration) to hit national targets — population by age/state, aggregate employment income / AGI / net income / federal tax, major program totals (SNAP, SSI, EITC, CTC, Social Security), SPM poverty rate.
4. Register it in the bundle manifest so pe.us.managed_microsimulation(dataset="populace_us_20k") works without allow_unmanaged.
5. Validate vs full Populace: net income, tax, benefits, SNAP/SSI/EITC/CTC, SPM poverty, deciles — document residual error per metric.

Notes

• The bundle already ships per-state datasets (states/XX) as a lighter-than-national option; this is the national small-sample complement.
• A quick uniform sample exists locally from tutorial prep for testing only (not calibrated, not for publishing).
• Surfaced during IMA World Congress 2026 PolicyEngine tutorial prep.

[MaxGhenis]

Data point for this issue, from IMA tutorial prep: verified a workaround that runs a national reform analysis on free Colab (12.7 GB) by subsampling the pinned national file in-notebook.

Recipe (verified end-to-end on a real free-tier VM, 2026-07-03):

• The pinned populace_us_2024.h5 is 354 MB / 57,240 households — downloads to Colab in ~1 s (HF dataset repo, public).
• Uniform K=5,000-household sample, household_weight × (N/K), entities filtered through person membership ids, sim rebuilt via SimulationBuilder.
• Key constraint: baseline + reform sequentially in one kernel OOMs 12.7 GB even with del + gc.collect() (allocator fragmentation — the floor is the instantiated tax-benefit system, not the data; memory barely drops from K=10k to K=2k). Running each pass in a subprocess caps the peak at a single sim: measured 10.2 GB (baseline, 108 s) / 11.8 GB (reform, 122 s).
• Accuracy at K=5,000 vs the full file: net income +3.7%; a $3,000 CTC's national cost $33.5B vs $32.0B (+4.7%). K=10,000 (needs ~more headroom than free Colab has) was −2.5%/−1.9% locally.

Working notebook with baked outputs from the real Colab run: https://colab.research.google.com/github/PolicyEngine/tutorial/blob/main/notebooks/populace_subsample_colab.ipynb

Implication for this issue: a shipped small calibrated file remains the right long-term answer (a calibrated K≈10–20k file would beat uniform-sample noise and remove the subprocess gymnastics), but the subsample route unblocks Colab teaching use today. The 11.8 GB reform peak also suggests any shipped small file should target a comfortably-sub-10 GB envelope.