Export input-mass-parity gate references the raw base, flagging correct calibration gains on PUF-imputed columns

[MaxGhenis]

Summary

_export_input_mass_gate (in tools/build_us_fiscal_refresh_release.py) compares the calibrated export frame against the raw pre-calibration base frame with a symmetric ±50% tolerance. For a dense parent built from a raw pooled-ASEC base, this fires on exactly the PUF-imputed income columns that calibration is supposed to scale up toward SOI/CBO fiscal targets — flagging correct, target-aligned masses as failures. The gate as wired is mis-referenced for the dense-parent-from-raw-base case. It should either (a) reference a certified release instead of the raw base, or (b) thread a reviewed-exclusions register for the PUF-imputed columns, or both. It should not be silenced globally with --allow-input-mass-drift (that would also mask the genuine #278 zeroing it was built to catch).

Where it bit

Build F (populace#299), attempt 5, dense-default run on the certified 3-year ASEC pool (337,704 hh). Calibration was excellent — final_loss 0.0416, within-10% 86.1% (beating certified f0af251's 0.0423 / 86.4%) — and every structural gate passed (target_profile_coverage, degenerate_input_signal, base_population_scale, immigration_composition, ecps_parity, validation_input_coverage). The only failure was _export_input_mass_gate, which raises after the export frame exists (which is why earlier attempts, dying at the CHIP gate, never reached it).

14 columns flagged, all export >> base (gains, not losses):

column	export vs base	drift
long_term_capital_gains_before_response	7.02e11 / 2.12e11	+230.7%
short_term_capital_gains	9.58e10 / 2.90e10	+230.7%
miscellaneous_income	1.76e10 / -8.15e9	+315.6% (base is negative → % meaningless)
partnership_self_employment_net_earnings	3.47e10 / 1.25e10	+177.2%
qualified_dividend_income	3.20e11 / 1.32e11	+142.3%
non_sch_d_capital_gains	7.43e10 / 3.38e10	+119.7%
tax_exempt_interest_income	6.13e10 / 2.82e10	+116.9%
charitable_cash_donations	2.61e11 / 1.38e11	+88.9%
charitable_non_cash_donations	5.57e10 / 3.05e10	+82.6%
rental_income	3.84e11 / 2.13e11	+80.4%
non_qualified_dividend_income	1.35e11 / 8.22e10	+64.3%
unemployment_compensation	3.35e10 / 2.05e10	+63.3%
home_mortgage_interest	5.31e11 / 3.45e11	+54.1%
first_home_mortgage_interest	5.05e11 / 3.33e11	+51.9%

(tolerance 0.5, minimum_reference_total $1B, 41 columns checked, reviewed_exclusions empty.)

Why these masses are RIGHT (not a calibration bug)

Compared each flagged column's calibrated national mass to its SOI/CBO national target (the honest reference, vs the un-aligned raw base):

• qualified_dividend_income: export $320B; CBO qualified-dividend target $354B → est is hitting the target (−9.7%). The gate flags +142% vs raw base for a mass that is slightly under its fiscal target.
• capital-gains family (LT/ST/non-Sch-D): CBO net_capital_gain target $1.291T, calibrated est $0.798T → −38.2%, i.e. even after +230% vs raw base, capital gains still undershoot CBO. The increase the gate flags is genuinely needed and still insufficient.
• partnership_self_employment_net_earnings → CBO net_business_income target $1.916T, est $1.917T → +0.1%, essentially exact.
• AGI control: −0.9%.

CPS structurally under-reports capital, dividend, and business income; PUF imputation seeds it low; calibration reweights toward SOI/CBO. The ±50%-vs-raw-base band asks calibration not to do its job. miscellaneous_income additionally has a negative base total, making a relative-drift check ill-defined.

Why the raw-base reference is the wrong yardstick here

The gate's docstring states its purpose: "L0 selection and reweighting optimize the target surface; an untargeted input column can lose its mass without moving any residual. The export must keep every material input base." It is the #278 mass-LOSS guard (sparse selection zeroing IRA/HSA/pension/childcare — candidate << reference). It applies abs(drift) > tol symmetrically, so it also fires on calibration gains — the opposite of the defect it targets.

Provenance corroborates: input_mass_parity_gate was added in #279 (44868be), after the certified f0af251 (Jun 20) and the live-default lineage. Neither certified release manifest records input_mass_parity (only the base_population_scale person-population mass_repair, which build-f also passes). The certified 57k live default was built without this gate and would itself fail export-vs-raw-base on the same PUF columns. There is no pre-calibration income-mass alignment step in the certified pipeline.

Proposed fix

1. Reference a certified release, not the raw base, for the export-vs-reference comparison (the underlying input_mass_parity_gate already supports reference_totals from a reference H5; the release builder already has an --input-mass-reference-h5 path used by _input_mass_reference_gate). Against a certified reference, calibration-driven upward alignment of PUF income is in-band, while a genuine sparse zeroing still fails.
2. And/or thread reviewed_exclusions through _export_input_mass_gate (the underlying fn takes it; the wrapper currently doesn't) for the PUF-imputed columns, each with the SOI/CBO-target justification above, so the register documents why these columns legitimately move.
3. Keep the loss/zeroing arm strict (candidate == 0 or candidate << reference must still fail — that is the Sparse-57k certified release zeroes untargeted input bases (IRA/HSA/SE-pension/childcare); CDCC and ALD reforms silently score ~$0 on bundle 4.18.8 #278 signature). Consider making the gain-side tolerance reference the fiscal target rather than the base, or asymmetric.
4. Do not rely on --allow-input-mass-drift as the standing answer — it disables the whole gate, re-opening Sparse-57k certified release zeroes untargeted input bases (IRA/HSA/SE-pension/childcare); CDCC and ALD reforms silently score ~$0 on bundle 4.18.8 #278.

Context

Found during the Build F certified-lineage rerun (populace#299). This is the last gate the dense parent trips; with it correctly referenced, the dense build passes on the certified pool.

[MaxGhenis]

Reference decision (from the Build F attempt-6 pre-flight) — use the live default, not f0af251

Whoever implements the 1-line threading of a reference frame into _export_input_mass_gate: the reference choice is made. I simulated the export-vs-certified comparison for all 14 flagged columns using attempt-5's exact export masses (from the failure log) against both candidate references, at the gate's tol ±50% and floor $1e9.

Live default sparse-l0-refit-57k (HF snapshot 14a2fb9d) — the RIGHT reference (11/14 clean):

column	export	live-default ref	drift	verdict
charitable_cash_donations	2.61e11	2.31e11	+13.1%	ok
charitable_non_cash_donations	5.57e10	5.28e10	+5.4%	ok
long_term_capital_gains_before_response	7.02e11	8.64e11	−18.8%	ok
non_qualified_dividend_income	1.35e11	1.01e11	+33.4%	ok
non_sch_d_capital_gains	7.43e10	7.58e10	−1.9%	ok
partnership_self_employment_net_earnings	3.47e10	6.17e10	−43.8%	ok
qualified_dividend_income	3.20e11	2.96e11	+8.0%	ok
rental_income	3.84e11	4.33e11	−11.4%	ok
short_term_capital_gains	9.58e10	1.18e11	−18.9%	ok
tax_exempt_interest_income	6.13e10	5.41e10	+13.2%	ok
unemployment_compensation	3.35e10	2.97e10	+13.0%	ok
first_home_mortgage_interest	5.05e11	3.11e11	+62.5%	FAIL (real)
home_mortgage_interest	5.31e11	3.11e11	+70.7%	FAIL (real)
miscellaneous_income	1.76e10	4.74e10	−62.9%	FAIL (degenerate)

f0af251 75k (HF snapshot d8f5cff) — a BAD reference (5 fail): it is a June-20 different-vintage build with negative/tiny masses on several columns — e.g. short_term_capital_gains ref is −$7.4e9 (giving a spurious +1397% drift), non_sch_d_capital_gains ref $1.14e10 (+550%), plus mortgage ×2 (+107%/+114%) and rental (+96%). Do not use it.

So: thread the export gate to reference the live default (57k). That cleanly vindicates 11/14 (the gate-mis-reference cases this issue is about). The 3 residuals are not gate artifacts — the two mortgage columns are a genuine cold-calibration overshoot on the JCT mortgage tax-expenditure (+59.7% vs target), and miscellaneous_income is a degenerate sign-unstable residual (base −$7.9B). Those belong to the informed-L0 reconstruction (#328), not to this gate's reference fix. Recommendation: threading the live-default reference resolves this issue for the 11 mis-referenced columns; leave the mortgage/misc residuals to #328 rather than granting them blanket exclusions here.