Thread L2 into the L0 refit stage; report weight-concentration diagnostics - #283
Merged
Conversation
…stics The l2_lambda concentration penalty existed in calibrate() but could not reach production weights: the release builder never passed it, and refit_l0_selection hardcoded l2_lambda=0.0 in the refit stage — the stage whose weights ship. This enables the ESS-vs-accuracy sweep the L0 paper lists as future work, with no behavior change at the defaults. - refit_l0_selection gains l2_lambda; calibrate_l0_refit gains refit_l2_lambda (the refit_epochs/refit_learning_rate override pattern: None inherits l2_lambda), so both-stage, selection-only, and refit-only penalties are all expressible and recorded in options provenance. - CalibrationResult and L0RefitResult report effective_sample_size (Kish), realized_max_weight_ratio, and top_1pct_weight_share; the standalone effective_sample_size() scores any weight vector. The scalars ship in calibration_diagnostics.json (schema v4; populace-data contract bumped). - build_us_fiscal_refresh_release.py gains --l2-lambda / --refit-l2-lambda, threads them to both calibration paths, and records them in telemetry and the default_dataset manifest block. - Fix: diagnostics_payload(L0RefitResult) raised AttributeError because fraction_within_10pct was never delegated to the refit — the default sparse release path would crash when writing calibration diagnostics. Delegated, with a regression test. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
MaxGhenis
pushed a commit
that referenced
this pull request
Jul 5, 2026
…309) The l2_lambda penalty divides by a reference weight vector whose choice decides which way the penalty pulls: under mass conservation its constrained optimum is w ∝ anchor², so anchoring at heterogeneous starting weights concentrates rather than spreads. The post-L0 refit starts from the selection stage's concentrated weights, which made a refit-stage penalty (the refit_l2_lambda knob shipped in #283) collapse ESS at production scale (8,408 -> 1,057 at lambda=1e-3 on the 337k-record US surface) instead of raising it. - calibrate() gains l2_anchor ("initial" | "uniform"): "uniform" divides by the mean starting weight, making the penalty a direct 1/ESS control regardless of the starting distribution. An explicit l2_anchor_weights vector is the harness seam for anchors the frame cannot express. - refit_l0_selection/calibrate_l0_refit gain "design": anchor the refit at the pre-selection initial weights of surviving records ("stay near the survey design"), derived from the selection result. At a fresh selection "design" collapses to "initial". refit_l2_anchor follows the refit_epochs inherit-or-override pattern; invalid anchors fail before the expensive selection stage. - Direction contract pinned where the effect clears the noise floor by orders of magnitude: from heterogeneous starts (ESS 29.3 of 160), the initial anchor drives ESS below 0.6x start while uniform drives it above 1.4x, at healthy loss. - Fixes #307: the refit threading test asserted an ESS direction on a near-converged fixture where the mass projection cancels the penalty's near-proportional push, leaving the comparison at the run-to-run nondeterminism noise floor (and the direction itself is anchor- and regime-dependent). The threading contract is now pinned via recorded solver options — exactly what the original hardcoded l2_lambda=0.0 bug would have violated — with the behavioral direction living in the anchor test above. Closes #307. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #284. The sweep this enables is specified in #285.
Why
The L0 paper holds the soft concentration penalty at λ_L2 = 0 and lists "trace effective-sample-size vs. accuracy trade-offs by varying λ_L2 and max_weight_ratio" as future work. The current sparse candidate concentrates weight heavily (paper probe: ESS ≈ 4,726 of 57,240 retained records, max weight ≈ 914k), so that sweep is the natural next calibration-quality step after #228.
Running it today is impossible in production:
l2_lambdaexists incalibrate(), but the release builder never passes it, andrefit_l0_selectionhardcodesl2_lambda=0.0in the refit stage — the stage that produces the shipped weights. A sweep would have regularized only the throwaway selection weights.This PR is the enabling change. No behavior change at the defaults (λ = 0 everywhere).
What
Refit-stage L2 threading (
populace-calibrate/solve.py)refit_l0_selection(..., l2_lambda=0.0)passes the penalty into the refit'scalibrate()call.calibrate_l0_refit(..., refit_l2_lambda=None)follows therefit_epochs/refit_learning_rateoverride pattern:Noneinheritsl2_lambda, an explicit value overrides — so both-stage, selection-only (refit_l2_lambda=0), and refit-only (l2_lambda=0+ override) sweep arms are all expressible. Both stages' penalties land inoptionsprovenance. Invalid overrides fail before the expensive selection stage.Weight-concentration diagnostics
CalibrationResult/L0RefitResultgaineffective_sample_size(Kish,(Σw)²/Σw²),realized_max_weight_ratio, andtop_1pct_weight_share;L0RefitResultdelegates to the refit (shipped) weights.effective_sample_size(weights)is exported for scoring published artifacts without re-running calibration.calibration_diagnostics.json; schema version 3 → 4, with thepopulace-datarelease contract and fixtures bumped to match. Every future run records its accuracy-vs-spread frontier coordinates.Production CLI (
tools/build_us_fiscal_refresh_release.py)--l2-lambda(default 0.0) reaches both the dense and L0+refit paths;--refit-l2-lambdaoverrides the refit stage and is rejected with--dense-default-dataset(no refit stage). Both are recorded in telemetry and thedefault_datasetmanifest block.Bug fix
diagnostics_payload()raisedAttributeErroron anL0RefitResultbecausefraction_within_10pctwas never delegated — the default sparse release path would crash when writingcalibration_diagnostics.json. Reproduced onmain, fixed by delegation, pinned with a regression test.Tests
refit_l2_lambdainheritance/override provenance across all three sweep regimes.effective_sample_sizecontract (uniform → n, single carrier → 1, all-zero → 0, rejects negative/non-finite).diagnostics_payload(L0RefitResult)regression test.Full suite: 1140 passed, 10 skipped;
ruff checkclean.Next
With this merged, the λ_L2 sweep runs against production via
--l2-lambda(primary both-stage grid 0 → 1e-1, plus selection-only / refit-only arms at the interesting values), reading ESS/loss frontiers straight from each run'scalibration_diagnostics.json.🤖 Generated with Claude Code