Flaky: test_refit_l0_selection_threads_l2_lambda_to_refit ESS comparison at 1e-5 margin

[MaxGhenis]

packages/populace-calibrate/tests/test_solve.py::test_refit_l0_selection_threads_l2_lambda_to_refit fails intermittently on a strict effective-sample-size comparison with a margin of ~1e-5:

E       assert 44.92098676684213 > 44.92099428536288

Observed 2026-07-04 on PR #306's wheels (3.13) and wheels (3.14) jobs (run 28716998027) — a branch that does not touch populace-calibrate — and both passed unchanged on rerun. Main's own run for "Thread L2 into the L0 refit stage" (28640352415, 2026-07-03) also failed, so the flake shipped with the test.

The assertion checks that threading L2 into the refit raises ESS, but the realized ESS difference at this fixture size sits at the noise floor of run-to-run nondeterminism (torch/BLAS reduction order), so the strict > flips sign intermittently.

Possible fixes, for whoever owns the L0 refit stage: seed/deterministic-mode the refit in this test, enlarge the fixture or the L2 lambda so the expected ESS gap clears the noise floor by orders of magnitude, or assert with an explicit margin (ess_l2 > ess_no_l2 + eps with eps chosen from measured run-to-run spread). A tolerance chosen to be just-big-enough would only hide the flake.

[juaristi22]

Root-caused — and your noise-floor diagnosis is exactly right, with a deeper mechanism behind it that we characterized while running the #285 L2 sweeps.

Why the difference sits at the noise floor: the test compares penalized vs free refits on a near-converged fixture. Once the fit loss is flat, the L2 penalty's gradient is nearly proportional across records, and the mass="conserve" projection cancels a proportional push in log-space almost exactly — so the penalty barely reshapes anything, and the realized ESS difference is pure torch/BLAS reduction-order noise (we measured the two runs agreeing to the 6th decimal on this fixture even at 50× the test's λ).

Why a margin would be wrong even if it were stable: the asserted direction is regime-dependent. Under mass conservation the penalty's constrained optimum is w ∝ anchor², and the refit's default anchor is its own starting vector — the selection stage's concentrated calibrated weights. At production scale (337k-record US surface) a refit penalty at λ=1e-3 collapsed ESS from 8,408 to 1,057 rather than raising it. So the test was asserting a direction that the penalty doesn't actually guarantee.

The fix (#309): per your "don't hide it" guidance — no tolerance. The threading test now pins the provenance contract via recorded solver options (exactly what the original hardcoded l2_lambda=0.0 bug would have violated, deterministic by construction), and the behavioral direction moves to a new anchor-direction test in a regime with active fit gradients, where the effect clears the noise floor by orders of magnitude (from a heterogeneous start with ESS 29.3: initial anchor drives ESS below 0.6× the start, uniform anchor above 1.4×). #309 also adds the anchor semantics (uniform/design) that make the refit-stage penalty usable on purpose rather than a trap at its default.

Verified stable across repeated local runs; full experimental evidence lands in #285 shortly.

🤖 Generated with Claude Code