Validate targeted adaptive stopping with stored comparisons

[davanstrien]

Summary

Read-only, pinned counterfactual validation of the targeted adaptive strategy introduced in #53. No judge calls are made by the replay and no production behavior changes.

The evidence package now includes:

• the original 14-model Britannica board
• a newly collected independent 14-model MOH table-fidelity board
• five smaller published boards spanning BPL, UFO, Rubenstein judges, and an earlier Britannica grid
• current targeted, size-aware, targeted-v2, fixed pair-balanced, and mixed-random designs
• five allocation seeds
• a 200-replicate page-clustered, design-aware bootstrap that reruns allocation inside each replicate
• current sentinel-policy robustness

Two independent 14-model boards

Board/strategy	Used	Saved	Kendall τ	Spearman ρ	Top-3 members/order	Median/max abs(ΔELO)
Britannica balanced	4,293	0.0%	1.000	1.000	3/3, yes	0.0 / 0.0
Britannica targeted	870	79.7%	0.934	0.982	3/3, no	36.3 / 85.5
Britannica targeted + 3×	736	82.9%	0.912	0.978	3/3, no	59.0 / 105.9
MOH balanced	4,505	0.0%	1.000	1.000	3/3, yes	0.0 / 0.0
MOH targeted	996	77.9%	0.846	0.960	3/3, no	46.2 / 115.7
MOH targeted + 3×	803	82.2%	0.780	0.925	3/3, no	45.9 / 114.8

On both boards, adaptive strategies exhaust all ten sample batches. Savings come from allocating fewer pairs, not meeting the stopping criteria early. No targeted-v2 variant passes the original fidelity/savings gate.

Page-clustered finding

The full MOH board itself has:

• median page-bootstrap Kendall τ: 0.780
• top-3 membership stability: 19.0%
• exact top-3 order stability: 10.5%
• 51 robust pair directions out of 91

At a fixed 40% budget, pair-balanced/mixed-random allocation preserves 98.4%/98.3% of robust pair directions, but top-3 membership agreement is only 47.4%/37.2%. The leaders are not stably ordered across page samples even when every comparison is used. This is not a stopping-threshold problem.

Across the smaller boards, fixed 40% subsampling preserves robust decisions when the full reference board is already stable and fails when the full board is itself under-resolved. Pair-balanced and mixed-random allocation do not consistently dominate one another.

Recommendation

Keep targeted opt-in and balanced as default. Do not proceed to a live targeted trial or implement the tested variants. Keep the 3× rule post-hoc only.

The evidence is sufficient to stop allocation tuning. A separate future design issue can consider page-clustered uncertainty, rank tiers, or robust pairwise relations instead of presenting exact ordering among unstable leaders. This experiment PR should not change production behavior directly.

No production bug was found.

Artifacts

• experiments/adaptive-stopping/README.md
• experiments/adaptive-stopping/replay.py
• experiments/adaptive-stopping/multi_board.py
• experiments/adaptive-stopping/moh/README.md
• pinned JSON/CSV primary, MOH, fixed-design, and cross-board results
• tests/test_adaptive_replay_experiment.py

Published MOH reference: https://huggingface.co/datasets/davanstrien/ocr-bench-moh-results

Validation

• uv run pytest tests/ -q — 605 passed
• uv run ruff check src/ tests/ experiments/adaptive-stopping/*.py
• uv run ruff format --check experiments/adaptive-stopping/*.py tests/test_adaptive_replay_experiment.py
• uv run ty check src/ experiments/adaptive-stopping/*.py