Sparse-57k export drops 94 no-formula input columns from the dense parent (tips, overtime, disability flags, ESI premiums, rent, QBI basis)

[MaxGhenis]

Summary

The sparse-57k export (populace-us-2024-sparse-l0-refit-57k-71a0887, certified default for bundles 4.18.8 and 4.20.0) drops 94 no-formula PolicyEngine-US input columns that the dense parent (f0af251, 4.18.7 default) populates. #278 documented mass-zeroing for a handful of bases and #356 the three SSI asset columns; this is the full inventory of the column-presence regression. The Build-H dense parent published today (populace-us-2024-buildh-dense-warmstart-b449eb7-20260708T100442Z) drops essentially the same set (129 of 133 columns), so the Build-I successor (#359) inherits the gap unless the export carries them.

Method: person-table column diff (dense 249 cols → sparse 222), filtered to pe-us 1.752.2 variables with no formula and no adds/subtracts (i.e., true inputs; formula-owned exports like employment_income/social_security are correctly dropped per #24 and excluded here). Mass = dense column × household weight, TY2024. Full list + per-column mass attached below.

Dollar inputs lost (default 0 → mass vanishes; selected, dense-weighted)

column	dense mass	consequence example
stock_assets + bank_account_assets + bond_assets	$44.0T	SSI/TANF/Medicaid resource tests dead (#356)
unadjusted_basis_qualified_property (+SSTB variants)	$2.1T	QBI deduction mis-scores
employer_sponsored_insurance_premiums	$1,129B	ESI/ACA interactions
pre_subsidy_rent	$590B	SNAP excess-shelter deduction, housing analyses
w2_wages_from_qualified_business (+SSTB)	$332B	QBI wage limit
unreimbursed_business_employee_expenses	$218B	itemized/state conformity
veterans_benefits	$177B	benefit-conditioned programs
taxable_401k/403b/SEP/keogh distributions, tax_exempt_ira_distributions	$174B	retirement-income policy
survivor_benefits	$112B	income surface
fsla_overtime_premium	$103B	OBBBA overtime deduction scores ~$0
tip_income	$59B	OBBBA tips deduction scores ~$0
salt_refund_income	$44B	AMT/itemizer interactions
child_support_received/expense, alimony_income/expense	$89B	means-tested income definitions
financial_assistance, disability_benefits, workers_compensation, casualty_loss, qualified_tuition_expenses, educator_expense, investment_interest_expense, …	$10–40B each	various

Flag inputs lost with real signal (default False → population zeroed)

• is_disabled (12.6% weighted) — disability pathways in SSI/Medicaid, SNAP ABAWD exemptions (being actively seeded in Carry SNAP eligibility and exemption inputs through Populace US outputs #350–Seed SNAP ABAWD discretionary exemptions at the statutory cap #353; those seeds vanish at export unless presence-gated)
• is_blind (3.1%) — SSI/Medicaid blind category, extra standard deduction
• is_pregnant (1.1%), receives_wic (0.8%) — WIC/Medicaid pregnancy pathways (Seed pregnancy at the national rate; document the never-carried SNAP exemption inputs #352)
• is_full_time_college_student (3.8%), is_separated, is_surviving_spouse, AOTC enrollment flags (~1.7%) — filing status and education credits
• immigration_status_str, ssn_card_type (categorical) — SSN/citizenship-conditioned policy (Dataset imputes 100% of the population as citizens with valid SSNs (breaks SSN/citizenship-conditioned policies, e.g. OBBBA CTC) #225 family)

Currently-degenerate but contract-relevant (default True, stored all-true)

takes_up_ssi_if_eligible, takes_up_medicaid_if_eligible, takes_up_medicare_if_eligible, takes_up_{early_}head_start_if_eligible, would_claim_wic, is_wic_at_nutritional_risk, *_would_be_qualified QBI flags. Dropping these is a no-op today because the dense columns are all-true and the engine defaults match — but the moment real take-up rates are seeded (per the #312/#315 contract), the export must carry them or the seeds silently revert to full take-up.

Suggested fix

• Extend the Build-I export gate (Build I: selection-side misc-income support fix for a certifiable 57k default successor #359 checks export mass on 35 columns) to the full input surface: column presence + mass vs the dense parent, with reviewed exclusions only for formula-owned exports (Certified populace-us persists 18 formula-owned variables eCPS doesn't (formula_owned_export gate never wired into the build) #24) and justified band-math cases.
• The selection step doesn't need new data — every dropped column exists on the pool rows; the subset export just needs to keep them.
• Classify each column in the Verify US wealth, property, and asset imputation backlog #49 inventory (populated / upstream-populated / reviewed-excluded) as acceptance criteria.

Relates #278, #356, #359, #312, #315, #350, #352, #225, #38, #49. Found while re-running the SSI asset-limit analysis (PolicyEngine/ssi-asset-limits) on the certified small default via policyengine.py 4.20.0.

[MaxGhenis]

Two updates:

Overlap with #340 — I missed it when filing (it predates this by two days and covers the tips/overtime/auto-loan/education/retirement-contribution/casualty/misc families with the same "never persisted as H5 keys" mechanism). Treat this issue as the full column-manifest inventory for #340's proposed gate: beyond #340's families, the diff adds the SSI asset columns (#356), disability flags (is_disabled 12.6% weighted / is_blind / is_pregnant), employer_sponsored_insurance_premiums ($1.13T), pre_subsidy_rent ($590B), QBI basis/wage machinery (~$2.4T), alimony/child-support ($89B), veterans/survivor benefits ($289B), and the all-true take-up seeds that go live the moment #312/#315-style real rates ship. Same fix, one gate: fail release when a consumed input column is absent from the persisted H5.

Build I status (populace-us-2024-buildi-sparse-rmloss100-6e8e929, certified today via policyengine.py#463): restores 7 of the 94 — traditional_ira_contributions_desired, self_employed_pension_contributions_desired, weekly_hours_worked_before_lsr, hours_worked_last_week, weeks_worked, immigration_status_str, ssn_card_type — i.e., the #278 IRA/SE-pension items plus hours and citizenship categoricals. The other 87 remain absent (person table 229 cols). Measured on the new default with pe-us 1.764.6: ssi_countable_resources = 0 for every record and the $10k/$20k SSI resource-limit reform scores $0.000B. One calibration note: Build I's baseline SSI is 7.97M recipients / $61.1B — right on SSA admin totals with no resource test binding, so the weight solve is absorbing the missing test; when the asset columns are restored, baseline SSI will undershoot unless the SSI targets are refit against a binding test.

[MaxGhenis]

Build J status (buildj-sparse-rmloss100-75d5add, certified default on bundle 4.20.4): 16 of the 94 restored — the three SSI asset columns (#373, SCF-2022 stage; SIPP blend tracked in #374), is_disabled, is_blind, is_pregnant, is_full_time_college_student, own_children_in_household, veterans_benefits, takes_up_medicaid_if_eligible, plus the Build-I seven (IRA/SE-pension intents, hours family, immigration_status_str, ssn_card_type). Person table 222 → 240 columns across the two builds, and the #369 input-coverage gate + reform-coverage smoke now guard the surface.

Still absent (78): tip_income, fsla_overtime_premium, employer_sponsored_insurance_premiums, pre_subsidy_rent, the QBI basis/wage machinery, alimony/child-support, survivor_benefits, disability_benefits, workers_compensation, education-credit inputs, takes_up_ssi_if_eligible and the other all-true seeds, and the rest of the #340 families — so the OBBBA tips/overtime rows still score ~$0 on the certified default.

[MaxGhenis]

Same disposition as #340: the Build M coverage campaign restored several of the 94 (tip_income, fsla_overtime_premium, ESI premium columns are on the certified export now); the residual absent set gets per-column verification and disposition in #451 under the #449 Build N epic.