L2 concentration penalty cannot reach production weights: refit stage hardcodes l2_lambda=0

[juaristi22]

Context

The L0 calibration paper deliberately holds the soft concentration penalty at λ_L2 = 0 to isolate the L0 effect, and lists "trace effective-sample-size vs. accuracy trade-offs by varying λ_L2 and max_weight_ratio" as future work. The concentration problem it defers is real: the reported sparse candidate has ESS ≈ 4,726 out of 57,240 retained households with a max weight ≈ 914k — the effective information content is ~8% of the record count, and a single household carrying ~914k weight is a robustness liability for downstream estimates (one idiosyncratic record can move a congressional-district aggregate).

Problem

The library already implements the right knob — calibrate(..., l2_lambda=...) adds l2_lambda * mean((w/w₀)²), exactly a soft ESS/design-effect control — but it cannot affect production weights:

1. The refit stage hardcodes it away. The production default is the two-stage calibrate_l0_refit path (L0 selection → ordinary refit), and the refit is what ships (it takes loss from 9.86% → 4.74% in the paper). calibrate_l0_refit forwards l2_lambda to the selection stage only; refit_l0_selection's internal calibrate() call passes a literal l2_lambda=0.0 and its signature has no such parameter. Any λ_L2 experiment today regularizes only the throwaway selection weights.
2. The release builder never passes it. tools/build_us_fiscal_refresh_release.py has no --l2-lambda argument, so both calibration call sites run at the default 0.0.
3. No concentration measurement exists in production. ESS lives only as a test helper; calibration_diagnostics.json records fit metrics (final_loss, fraction_within_10pct) but nothing about weight spread, so a sweep would have no frontier coordinates to read.

Also found while investigating

diagnostics_payload() raises AttributeError when handed an L0RefitResult, because fraction_within_10pct was never delegated to the refit stage — and the default sparse release path passes exactly that type to the diagnostics writer. The default build would crash when writing calibration_diagnostics.json. Reproducible on main:

result = calibrate_l0_refit(frame, targets, epochs=120, seed=0, l0_lambda=0.003)
diagnostics_payload(result)  # AttributeError: 'L0RefitResult' object has no attribute 'fraction_within_10pct'

Fix

PR #283 threads l2_lambda into the refit (refit_l0_selection(..., l2_lambda=...), calibrate_l0_refit(..., refit_l2_lambda=...) with the refit_epochs-style inherit/override semantics), adds --l2-lambda / --refit-l2-lambda to the release builder, adds ESS / realized-max-ratio / top-1%-share diagnostics to results and the diagnostics artifact (schema v4), and fixes the L0RefitResult payload crash. All defaults preserve current behavior.

The actual λ_L2 sweep is tracked separately (follow-up issue).

🤖 Generated with Claude Code