Seed the Optuna sampler: reproducible sweeps, and fix the flaky falsifier test

[bardhh]

Summary

run_optuna_sweep called optuna.create_study(direction=direction) with no sampler, so Optuna drew a fresh random seed on every run. Which trials got explored, and therefore the sweep's results, varied between identical invocations. sample_param_combos() in the same module already takes seed=0 for exactly this reason; the Optuna path was the only sampling left unseeded.

The same gap made test_optuna_falsifier flaky, which has been reddening CI on unrelated pull requests since at least July 9. It reappeared today on main at e3bc7b8.

Why it failed

The test runs 3 trials over alpha in {1.0, 3.0, 5.0}, and the mock runner only fails when alpha > 2.0. Optuna's startup trials are drawn uniformly with replacement, so roughly one run in 27 draws alpha=1.0 three times. Nothing is falsified, every trial runs all 4 seeds, 12 records accumulate, and assert len(records) < 3 * len(seeds) fails on 12 < 12.

Evidence

The failure rate is too low to see by running the test a few times, so I ran the sweep 300 times in-process and recorded which alpha values each run drew.

	Failures	Rate	Draws that failed
Before	11 / 300	3.7%	all 11 on (1.0, 1.0, 1.0)
After	0 / 300	0%	one fixed draw, (3.0, 5.0, 5.0)

3.7% measured against 3.7% predicted from (1/3)^3, and the correlation is exact: (1.0, 1.0, 1.0) came up 11 times and failed all 11, while no other draw failed once.

The fix

run_optuna_sweep takes seed: int | None = 0 and passes it to TPESampler(seed=seed). Deterministic by default, matching sample_param_combos. Passing None opts back into Optuna choosing its own seed.

Note this changes behaviour beyond the test: sweeps are now reproducible by default. For a benchmarking harness that seems like the right default, but it is a change worth a look.

Tests

Two new tests cover both directions: the same seed explores the same trials, and seed=None does not. I checked that test_optuna_sweep_is_reproducible fails against the pre-fix code, so it actually guards the bug rather than passing either way.

The previously-flaky test now passes 40 consecutive runs. Full fast suite: 492 passed, 4 skipped, up exactly 2 from the 490 baseline. ruff check src clean.

The comment in test_optuna_falsifier described a draw the seeded sampler never produces, so it is updated to match what actually runs.

Black wants to reformat two with Live(...) blocks in sweep.py. That is pre-existing drift unrelated to this change, so I left it alone rather than bundling it in.