Add the L2 sweep harness over exported production calibration inputs - #310
Closed
juaristi22 wants to merge 1 commit into
Closed
Add the L2 sweep harness over exported production calibration inputs#310juaristi22 wants to merge 1 commit into
juaristi22 wants to merge 1 commit into
Conversation
tools/sweep_us_l2_lambda.py runs the ESS-vs-accuracy sweep from #285: the production two-stage calibration (L0 selection -> refit) across a lambda grid, three penalty arms (both stages, selection-only, refit-only), and the anchor choices from #309, reusing a production run's inputs so a sweep point costs one calibrate stage (~35-55 min) instead of one full build. Two input modes: - --problem-bundle: an exported compiled problem (scipy CSR constraint matrix, target values/names, initial weights, production target loss weights). Each matrix row replays as a Target with a callable measure over a synthetic frame, so calibrate_l0_refit runs unchanged; the reconstruction is pinned bit-for-bit against the export in tests, and was verified exact on the real 32,633 x 337,704 production bundle (28,865,166 nonzeros, max |diff| = 0). --initial-weights swaps in an externally supplied vector (e.g. survey design weights) for production-regime sweeps. - --target-frame-checkpoint + --ledger-facts: a build's materialized frame checkpoint with targets recompiled through the builder's own loaders. The builder's checkpoint reader is split into an identity-agnostic _read_target_frame_checkpoint_file plus the strict cache wrapper (attrs-only precheck keeps cache misses cheap and schema mismatches loud). lambda=0 runs once as the shared baseline; every point writes its full calibration_diagnostics.json (schema v4 concentration scalars) plus a point.json row with input provenance (bundle/checkpoint hashes, anchor, grid config); sweep_summary.json/.csv are rewritten after each point and completed points are skipped on rerun, so an interrupted multi-hour sweep resumes where it stopped. Part of #285. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #285. Stacked on #309 (the harness drives the anchor options; retarget to main when #309 merges).
Why
λ values for the L0/L2 penalties are conditional on the target surface and the starting-weight regime, so they need re-validation whenever either changes — a repeatable, production-faithful sweep is infrastructure, not a one-off. The expensive part of a release build is PolicyEngine materialization, not calibration: by reusing a production run's exported calibration inputs, a sweep point costs one calibrate stage (~35–55 min) instead of one build. This harness produced the #285 frontier (22 points across four arms).
What
tools/sweep_us_l2_lambda.pyruns the production two-stage calibration (L0 selection → refit) across a λ grid, three penalty arms (both,selection-only,refit-only), and the #309 anchor choices. Two input modes:--problem-bundle: an exported compiled problem (scipy CSR constraint matrix, target values/names, initial weights, production target loss weights). Each matrix row replays as aTargetwith a callable measure over a synthetic frame, socalibrate_l0_refitruns unchanged — selection, refit, subsetting, mass conservation all production semantics. The reconstruction is pinned bit-for-bit in tests and was verified exact against the real 32,633×337,704 production bundle (28,865,166 nonzeros, max |diff| = 0.0).--initial-weightsswaps in an external vector (e.g. survey design weights) for production-regime sweeps.--target-frame-checkpoint+--ledger-facts: a build's materialized frame checkpoint, targets recompiled through the builder's own loaders (registry, fiscal loss weights) so points cannot drift from production. The builder's checkpoint reader is split into an identity-agnostic_read_target_frame_checkpoint_fileplus the existing strict cache wrapper (attrs-only precheck keeps cache misses cheap; schema mismatches stay loud).Operational contract: λ=0 runs once as the shared
baseline; every point writes fullcalibration_diagnostics.json(schema v4 concentration scalars) plus apoint.jsonsummary row carrying input provenance (bundle/checkpoint hashes, anchor, grid config);sweep_summary.json/.csvare rewritten after each point and completed points are skipped on rerun, so an interrupted multi-hour sweep resumes where it stopped..gitignoregains entries so sweep outputs and input bundles never get committed.Why it lives in
tools/Its faithfulness guarantee comes from importing the release builder's own loaders — moving it out of the repo would turn it into a reimplementation that drifts.
tools/is exactly this genre (score_us_fiscal_targets.py,compare_us_fiscal_diagnostics.py,inspect_us_fiscal_refresh_checkpoints.py).Tests
10 behavioral contracts: point expansion (baseline dedupe, arm→stage mapping), per-arm penalty landing in the right stage (via stage-level options provenance), bundle replay bit-equivalence, initial-weights override, resume semantics, registry-less diagnostics, checkpoint file-reader round-trip. Full suite 1,343 passed / 47 skipped; ruff clean.
🤖 Generated with Claude Code