Phase EJ Step 8: Overnight flags-on A/B validation soak

[aberson]

Build step 8 of 8 — Phase EJ (umbrella #281) — Type: wait

Build doc: documentation/plans/evolve-judging-plan.md §7 Step EJ.8

What to build (overnight validation soak — long-running observation)

The investigation's numbers (null-rollback 50%→18.75%, ~5.9 games saved per avoided re-proposal, reclaimed final-generation games) are analytic; validate them against a real overnight run before any default flips.

Run one standard 4h flags-on soak (/improve-bot-evolve with the EJ.7 flags minus the reduced sizes) and compare against the most recent flags-off soaks on:

• regression-rollback rate,
• screen-null-diff / pool_dedup catch counts,
• games per generation,
• generations completed,
• promotions.

Record the comparison table in the soak log; recommend (do not apply) default flips.

Existing context

• Type: wait — long-running observation. /build-phase halts here by design (halt-contract class Step 4: Scouting and threat assessment #4: wait step); resume in a fresh session after the soak completes.
• Compare against the most recent flags-off soaks under documentation/soak-test-runs/.
• Data source: data/evolve_results.jsonl + run state (data/evolve_run_state.json).

Done when

Soak completes; comparison table filled from evolve_results.jsonl + run state; recommendation recorded. NOT gated on the soak showing improvement — a negative result is a valid outcome that blocks default flips.

Flags (recommended)

n/a — Type: wait (long-running soak; /build-phase halts here intentionally so wall-clock waiting doesn't consume context window)

Depends on

#288 (smoke must pass before committing an overnight run)

Parallel-safe with

none — strictly sequential (hard Depends on: EJ.7)

Produces

Soak report under documentation/soak-test-runs/ with the A/B comparison table + default-flip recommendation

Operator workflow notes

• Wait step: /build-phase halts intentionally. Launch the soak per .claude/rules/evolve.md + .claude/rules/wsl-evolve.md (Windows-SC2 canonical, or Linux-SC2 in Ubuntu-22.04); resume triage in a fresh session.

---

Synced from documentation/plans/evolve-judging-plan.md by /repo-sync at c6099d9