Skip to content

Switch to canonical SPM variable names; populate 2026 baseline#1

Closed
PavelMakarchuk wants to merge 1 commit into
mainfrom
switch-canonical-vars-and-populate
Closed

Switch to canonical SPM variable names; populate 2026 baseline#1
PavelMakarchuk wants to merge 1 commit into
mainfrom
switch-canonical-vars-and-populate

Conversation

@PavelMakarchuk
Copy link
Copy Markdown
Contributor

Summary

  • Switched poverty methodology to canonical policyengine.py variable names: spm_unit_is_in_spm_poverty and spm_unit_is_in_deep_spm_poverty, both mapped to person entity. Formulas are mathematically identical to v1 API's person_in_poverty / in_deep_poverty, but the names match policyengine/outputs/poverty.py (USPovertyType.SPM / SPM_DEEP). Worth noting: the v1 API references person_in_deep_poverty, which no longer exists in policyengine-us 1.690.7.
  • Added scripts/compute_local.py so we can populate data/baseline.json without Modal. Falls back to the documented HuggingFace dataset URL convention when policyengine.countries fails to import (editable installs with mismatched manifests).
  • Pinned Next.js to 15.5.4 — Next 16 + Turbopack hits an RSC bundler bug ("Could not find the module global-error.js#default in the React Client Manifest").
  • Populated baseline.json for 2026 (current-law, no reforms): all 52 regions, zero errors.

Baseline highlights (2026, SPM, weighted by person_weight)

Region Poverty Child poverty Deep poverty Deep child
US 28.98% 19.02% 13.45% 4.33%
AK 9.15% (lowest)
NE 9.17%
MI 9.21%
NY 19.28%
MA 21.10%
CA 25.72% (highest)

Computed with policyengine 4.3.0 / policyengine-us 1.690.7 / policyengine-us-data 1.17.0.

Observations worth digging into separately

  1. Federal (29%) is higher than the state distribution (range 9–26%). The national enhanced_cps_2024.h5 and the per-state states/{XX}.h5 files appear to be calibrated differently — possibly different SPM thresholds, or the state files use state-specific thresholds while the national file uses a single national one. We're using the same datasets the API would route to for each region, so the gap reflects the data, not the methodology.
  2. State person_weight totals look low (e.g. AL ~2.2M vs real ~5.1M). The state-file weight columns may not be calibrated to full state population — also a property of the data, not this code.

Test plan

  • make dev — page renders 52-row table sorted by child poverty desc
  • Versions strip shows the three pinned versions
  • "Check latest" (after modal deploy) flags any package that has moved on PyPI
  • "Upgrade & recompute" round-trip produces a downloadable JSON that, once committed, looks identical in shape

🤖 Generated with Claude Code

Methodology:
- Use spm_unit_is_in_spm_poverty and spm_unit_is_in_deep_spm_poverty
  mapped to person, matching policyengine.py outputs/poverty.py conventions
  (USPovertyType.SPM / SPM_DEEP).
- Formulas are mathematically identical to v1 API's person_in_poverty /
  in_deep_poverty; this only changes the variable name we reference.

Local compute:
- New scripts/compute_local.py runs the same methodology locally without
  Modal, writes data/baseline.json and frontend/public/baseline.json.
- _resolve_dataset falls back to the documented HuggingFace URL convention
  when policyengine.countries import fails (e.g. editable installs with a
  mismatched manifest).

Frontend:
- Pin Next.js to 15.5.4 (16 + Turbopack hits an RSC bundler bug).
- Replace turbopack.root with outputFileTracingRoot.

Baseline (2026, current-law, no reforms):
- All 52 regions populated, zero errors.
- Computed with policyengine 4.3.0 / policyengine-us 1.690.7 /
  policyengine-us-data 1.17.0.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@PavelMakarchuk
Copy link
Copy Markdown
Contributor Author

Superseded by #2, which contains these commits plus the deploy and senior-poverty work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant