Phase EJ Step 3: One-sided posterior rollback bar

[aberson]

Build step 3 of 8 — Phase EJ (umbrella #281)

Build doc: documentation/plans/evolve-judging-plan.md §7 Step EJ.3

What to build

run_regression_eval rolls back on wins_new < games // 2 + 1 (src/orchestrator/evolve.py L746, L782) — a truly-neutral promotion is destroyed 50% of the time at n=5, and draws/crashes count against the new parent because _count_wins (L450-463) shrinks the decided count while the majority bar stays at 3.

Build src/orchestrator/gate_stats.py:

• posterior_prob_worse(wins: int, losses: int) -> float returns P(p < 0.5 | Beta(1+wins, 1+losses)) via the exact identity I_0.5(a, b) = Σ_{k=a}^{a+b-1} C(a+b-1, k)·0.5^(a+b-1) (integer params, math.comb, no scipy / no new deps);
• one_sided_rollback(wins_new, wins_prior, *, min_decided=4, threshold=0.85) -> tuple[bool, str] returns (rollback?, reason). At n=5 decided: rollback only on 0-5 (P≈0.984) and 1-4 (P≈0.891); 2-3 (P≈0.656) keeps. Fewer than min_decided decided → keep (fail-open) with an explanatory reason.

Add rule: str = "majority" to run_regression_eval with the early-stop threshold DERIVED from the active rule (the hardcoded stop-at-majority logic at L743-772 is wrong for the one-sided rule and at n≠5). Thread scripts/evolve.py --regression-rule {majority,one-sided} (default majority, byte-identical).

One-sided semantics: rollback iff decided ≥ 4 AND P(p<0.5) ≥ 0.85.

Existing context

• gate_stats.py is a new leaf module — single source of truth for both the gate and its tests (.claude/rules/code-quality.md one-source-of-truth). Phase R can extend it later.
• run_regression_eval read in full L687-808; needed = games // 2 + 1 (L746), rolled_back = wins_new < needed (L782), early-stop locks once either side hits majority (L743-772).
• _count_wins (L450-463) excludes draws/crashes from both tallies → undecided games count against the new parent (the draw/crash-forces-rollback bias EJ.3 fixes by conditioning on decided games).
• Only caller of run_regression_eval: scripts/evolve.py (1 call site) + tests (grep-confirmed, no other consumers). Default rule="majority" keeps signatures compatible.

Files to modify/create

• src/orchestrator/gate_stats.py (new)
• src/orchestrator/evolve.py (rule param in run_regression_eval; rule-derived early-stop)
• scripts/evolve.py (--regression-rule flag, threaded to run_regression_eval)
• tests/test_gate_stats.py (new); tests/test_evolve.py (extend regression cases)

Done when

Exact-value tests for the posterior table (0-5→rollback, 1-4→rollback, 2-3→keep at n=5; correct behavior at n=3/7/9; draw-heavy records keep under the min-decided floor); early-stop fires at rule-correct thresholds in both rules; rule="majority" decisions byte-identical to today across a golden table of all (wins_new, wins_prior) pairs at n=5 and n=9. uv run pytest / mypy src bots --strict / ruff check . clean; test count ≥ baseline.

Flags (recommended)

--reviewers code

Depends on

none

Parallel-safe with

none — strictly sequential (edits scripts/evolve.py argparse+loop shared with sibling EJ steps; plan §4 mandates serial execution). Note: EJ.4 depends on this step.

Produces

src/orchestrator/gate_stats.py; rule param in run_regression_eval; --regression-rule flag; tests/test_gate_stats.py; extended tests/test_evolve.py regression cases

Operator workflow notes (autonomous + UI-bundle + parallel)

• Runs end-to-end via /build-step; the Done when is the gate.
• No --ui — pure statistics + backend.
• Serial: shared scripts/evolve.py; EJ.4 builds on this.

---

Synced from documentation/plans/evolve-judging-plan.md by /repo-sync at c6099d9

[aberson]

build-phase: Step EJ.3 started

Step: One-sided posterior rollback bar
Phase: EJ
Flags: --reviewers code (--isolation none)
Baseline tests: 1858 passing
Baseline HEAD: 6722a1e (EJ.2)
Started: 2026-07-06

New leaf module src/orchestrator/gate_stats.py (posterior_prob_worse via exact binomial-tail identity, math.comb, no scipy + one_sided_rollback); rule: str = "majority" param on run_regression_eval with rule-derived early-stop; --regression-rule {majority,one-sided} flag (default majority, byte-identical). Verified oracle: at n=5 decided, rollback P is 0-5→0.984, 1-4→0.891, 2-3→0.656; threshold 0.85 → rollback on 0-5 and 1-4 only.

[aberson]

build-phase: Step EJ.3 complete

Verdict: PASS (iteration 2/3)
Commit: becf306
Files changed: src/orchestrator/gate_stats.py (new), src/orchestrator/evolve.py, scripts/evolve.py + tests (test_gate_stats.py new, test_evolve.py, test_evolve_cli.py)
Tests: 1909 passing (+51 over 1858 EJ.2 baseline)
Gates: mypy strict OK (808 files), ruff check OK, ruff format --check (no new debt)
Reviewers: 4 code — correctness CLEAN (posterior recomputed in exact rationals; soundness brute-forced 0 mismatches n=5..11), bugs CLEAN (28,727 state/completion checks, 0 violations), style CLEAN (modeled on staleness.py sibling), test-quality (1 med + 3 low). Iteration 2 closed the medium + 2 lows.

Changes made

• src/orchestrator/gate_stats.py (new leaf module, single source of truth, no scipy): posterior_prob_worse(wins, losses) = P(p<0.5 | Beta(1+wins,1+losses)) via the exact binomial-tail identity sum_{k=wins+1}^{n1} C(n1,k)/2^n1 (math.comb). Verified oracle: 0-5→63/64=0.984, 1-4→57/64=0.891, 2-3→42/64=0.656. one_sided_rollback(wins_new, wins_prior, *, min_decided=MIN_DECIDED=4, threshold=ROLLBACK_THRESHOLD=0.85) → fail-open below min_decided, else rollback iff P≥threshold.
• run_regression_eval gains rule: str = "majority" (byte-identical default) + a rule-derived early-stop (_regression_stop_locked): stops only when the verdict is invariant under every completion of remaining games (draws counted via a played counter for the min_decided floor). Majority path is max(live) >= needed, unchanged. One-sided rolls back only on positive evidence of harm (0-5, 1-4 at n=5; 2-3 keeps) — fixes the coin-flip 50% false-rollback + the draw/crash-forces-rollback bias.
• scripts/evolve.py --regression-rule {majority,one-sided} (default majority) threaded to run_regression_fn; end-to-end pass-through tested.

Review findings addressed (iteration 1 → 2)

• MED (test): --regression-rule had zero production-caller coverage → test_default_flags pins the default + two run_loop pass-through tests assert the spy is called with rule="one-sided" / rule="majority". Closes the silent-wiring gap.
• LOW: consolidated test_symmetry 100 parametrized instances → 1 grid-looping test (de-inflation, coverage identical).
• LOW: test_signature_defaults_are_the_named_constants now uses inspect.signature(...).default is MIN_DECIDED — catches a hardcoded-literal-default drift.

Byte-identical default confirmed: rule="majority" golden table pins both rolled_back AND the exact reason string across all (wins_new, wins_prior) at n=5 and n=9. EJ.4 (panel floor) is the mandatory pairing for enabling one-sided in production.

[aberson]

Completed via /build-phase (EJ.3, commit becf306). Gates green: pytest 1909, mypy strict, ruff. Posterior math + early-stop soundness independently brute-forced (0 mismatches). EJ.4 panel floor is the mandatory pairing before enabling one-sided in production.