Add SOI-actuals baseline backtest to reform validation - #247
Merged
Conversation
Second external-validation family alongside the JCT reform scores: compare simulated baseline totals to published IRS SOI actuals for tax items that are NOT calibration targets (the calibrated national SOI concepts are income lines, EITC, and itemized components). Nine lines from the TY2023 Individual Complete Report (Pub 1304, rev. March 2026), Tables 3.3/1.4: net income tax, AMT, CDCC, nonrefundable education credits, saver's credit, nonrefundable CTC+ODC, refundable ACTC, NIIT, and self-employment tax — each mapped to the PE-US variable with the same scope (documented where scope differs, e.g. income_tax also carries retirement-penalty and unreported-payroll amounts, ~0.5%). Levels need no counterfactual, so all nine share the one baseline simulation. Rows publish in reform_validation.json with category 'IRS SOI actual', window TY2023, and the SOI table as source; an unsimulated backtest marks out_of_sample_simulated=false like skipped OBBBA rows, so the publish guard covers it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Extends leg-2 (external) validation with a second benchmark family: simulated baseline totals vs published IRS SOI actuals, for tax items that are not calibration targets. Unlike the JCT reform scores, these are realized tax-year data — no behavioral, fiscal-year-timing, or projection caveats.
Nine lines from the TY2023 Individual Complete Report (Pub 1304, rev. March 2026, Tables 3.3 / 1.4), values extracted from the published XLS and cross-checked against the release target set (calibrated national SOI concepts are income lines, EITC, and itemized components — none of these):
income_taxalternative_minimum_taxcdcceducation_tax_creditssavers_creditnon_refundable_ctcrefundable_ctcnet_investment_income_taxself_employment_taxScope alignment is documented per line (e.g. PE
income_taxalso carries retirement-penalty and unreported-payroll amounts, ~0.5%;education_tax_credits= nonrefundable AOTC + LLC, matching SOI's nonrefundable column). Dataset period is 2024 vs TY2023 actuals — a one-year vintage gap, same convention as the ty2023 calibration targets; populace should run slightly above.How
soi_baseline_levels.json— curated config with values + citations.BaselineLevelSpec+soi_baseline_level_specs();reform_validation_payload(baseline_levels=…)emits rows (category "IRS SOI actual", window TY2023). All levels share the one baseline simulation — the whole backtest costs one sim.out_of_sample_simulated=false, so the publish guard (Refuse to publish releases with skipped out-of-sample reforms #222) covers it.Tests
Config loads with unique ids + verified income-tax value; levels share exactly one
simulate(None)call; null + guard flag when unsimulated. 21 pass.🤖 Generated with Claude Code