Skip to content

Add SOI-actuals baseline backtest to reform validation - #247

Merged
PavelMakarchuk merged 2 commits into
mainfrom
soi-baseline-backtest
Jul 1, 2026
Merged

Add SOI-actuals baseline backtest to reform validation#247
PavelMakarchuk merged 2 commits into
mainfrom
soi-baseline-backtest

Conversation

@PavelMakarchuk

Copy link
Copy Markdown
Contributor

What

Extends leg-2 (external) validation with a second benchmark family: simulated baseline totals vs published IRS SOI actuals, for tax items that are not calibration targets. Unlike the JCT reform scores, these are realized tax-year data — no behavioral, fiscal-year-timing, or projection caveats.

Nine lines from the TY2023 Individual Complete Report (Pub 1304, rev. March 2026, Tables 3.3 / 1.4), values extracted from the published XLS and cross-checked against the release target set (calibrated national SOI concepts are income lines, EITC, and itemized components — none of these):

line PE variable SOI TY2023 actual
Net income tax (minus refundable credits) income_tax $2,042.0B
Alternative minimum tax alternative_minimum_tax $2.75B
Child & dependent care credit cdcc $3.47B
Education credits (nonrefundable) education_tax_credits $7.55B
Saver's credit savers_credit $2.04B
CTC+ODC (nonrefundable) non_refundable_ctc $84.42B
ACTC (refundable) refundable_ctc $34.53B
Net investment income tax net_investment_income_tax $39.33B
Self-employment tax self_employment_tax $81.09B

Scope alignment is documented per line (e.g. PE income_tax also carries retirement-penalty and unreported-payroll amounts, ~0.5%; education_tax_credits = nonrefundable AOTC + LLC, matching SOI's nonrefundable column). Dataset period is 2024 vs TY2023 actuals — a one-year vintage gap, same convention as the ty2023 calibration targets; populace should run slightly above.

How

  • soi_baseline_levels.json — curated config with values + citations.
  • BaselineLevelSpec + soi_baseline_level_specs(); reform_validation_payload(baseline_levels=…) emits rows (category "IRS SOI actual", window TY2023). All levels share the one baseline simulation — the whole backtest costs one sim.
  • Builder passes the default levels; an unsimulated backtest sets out_of_sample_simulated=false, so the publish guard (Refuse to publish releases with skipped out-of-sample reforms #222) covers it.
  • Dashboard renders these rows as-is (benchmark/populace/error/Year/Source), no consumer change needed.

Tests

Config loads with unique ids + verified income-tax value; levels share exactly one simulate(None) call; null + guard flag when unsimulated. 21 pass.

🤖 Generated with Claude Code

PavelMakarchuk and others added 2 commits July 1, 2026 16:13
Second external-validation family alongside the JCT reform scores:
compare simulated baseline totals to published IRS SOI actuals for tax
items that are NOT calibration targets (the calibrated national SOI
concepts are income lines, EITC, and itemized components). Nine lines
from the TY2023 Individual Complete Report (Pub 1304, rev. March 2026),
Tables 3.3/1.4: net income tax, AMT, CDCC, nonrefundable education
credits, saver's credit, nonrefundable CTC+ODC, refundable ACTC, NIIT,
and self-employment tax — each mapped to the PE-US variable with the
same scope (documented where scope differs, e.g. income_tax also carries
retirement-penalty and unreported-payroll amounts, ~0.5%).

Levels need no counterfactual, so all nine share the one baseline
simulation. Rows publish in reform_validation.json with category
'IRS SOI actual', window TY2023, and the SOI table as source; an
unsimulated backtest marks out_of_sample_simulated=false like skipped
OBBBA rows, so the publish guard covers it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@PavelMakarchuk
PavelMakarchuk merged commit 9c594af into main Jul 1, 2026
4 checks passed
@PavelMakarchuk
PavelMakarchuk deleted the soi-baseline-backtest branch July 1, 2026 20:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant