Skip to content

Split unsplit dividend totals by SOI qualified share#202

Merged
MaxGhenis merged 1 commit into
mainfrom
claude/income-seam-fix
Jun 3, 2026
Merged

Split unsplit dividend totals by SOI qualified share#202
MaxGhenis merged 1 commit into
mainfrom
claude/income-seam-fix

Conversation

@MaxGhenis
Copy link
Copy Markdown
Contributor

Summary

Fix the qualified / non-qualified dividend split inversion in normalize_dividend_columns.

When a record carries only a dividend total (e.g. CPS DIV_VAL, which reports no qualified/non-qualified breakdown) and no observed component split, the function dumped 100% of the total into non_qualified_dividend_income and left qualified_dividend_income at $0. Because this runs on the merged persons (pipelines/us.py:10854,11090) where every CPS-native dividend row has a total but no split, it zeroed qualified dividends on the entire CPS spine and inverted the national split.

Measured on the collapse-scaffold RC (audit dashboard, scored via microdf weighted .sum() against the SOI targets):

qualified non-qualified qualified share
collapse RC (before) $102B $465B 18% (inverted)
eCPS $352B $148B 70%
PUF 2015 source (E00650/E00600) $204B $57B 78% (correct)
SOI target (qualified_dividend_income) $291B

qualified_dividend_income was −65% vs its $291B target; the total dividend was roughly right, so the error was purely in the split.

Fix

Allocate an unsplit dividend total by the SOI qualified share (UNSPLIT_DIVIDEND_QUALIFIED_SHARE = 0.78, basis: SOI 2015 PUF E00650/E00600 = $204.0B/$260.9B = 0.782) instead of defaulting the whole amount to non-qualified. Applies in the two branches that previously produced all-non-qualified output (has_qualified and has_non_qualified total-only rows, and the no-components else branch).

This mirrors how the incumbent eCPS splits CPS DIV_VAL via a qualified fraction. The 0.78 flat share is a documented first-order constant; a future refinement is an income-conditional share.

Blast radius

  • normalize_dividend_columns has 4 call sites. PUF data is unaffected — it carries the E00650/E00600 split, so the total-only branch never fires there. Only merged-persons rows that have a dividend total but no observed split (the CPS-native spine) change.

Tests

  • Updated test_normalize_dividend_columns_coalesces_sparse_total_aliases_by_row (row 0 now splits the unsplit total 78/22 instead of 0/100; rows with observed components unchanged).
  • Added test_normalize_dividend_columns_splits_unsplit_total_by_qualified_share.
  • test_normalize_dividend_columns_prefers_atomic_components_over_totals still passes (observed components are preserved, never reallocated).

Validation note

Unit-tested. Full national validation requires a data rebuild (the qualified/non-qualified split feeds calibration + tax computation); expected to move qualified_dividend_income from ~$102B toward the ~$291B SOI target and deflate non_qualified_dividend_income correspondingly. Part of the calibration-surface fixes tracked in #200 (this is the confirmed, self-contained one; the interest seam, wage/NIPA target profile, and tax-expenditure target handling remain there).

normalize_dividend_columns dumped a dividend total with no observed
qualified/non-qualified breakdown (e.g. CPS DIV_VAL) 100% into
non_qualified_dividend_income, zeroing qualified dividends on every
CPS-native row and inverting the national split (qualified 18% vs the
SOI ~78%; qualified_dividend_income -65% vs its $291B target).

Allocate an unsplit total by the SOI qualified share (0.78, from PUF
E00650/E00600) instead. PUF data is unaffected (it carries the split).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@MaxGhenis MaxGhenis marked this pull request as ready for review June 3, 2026 19:59
@MaxGhenis MaxGhenis merged commit 8973b4b into main Jun 3, 2026
5 checks passed
@MaxGhenis MaxGhenis deleted the claude/income-seam-fix branch June 3, 2026 19:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant