Phase EJ Step 2: Mechanical null-diff screen

[aberson]

Build step 2 of 8 — Phase EJ (umbrella #281)

Build doc: documentation/plans/evolve-judging-plan.md §7 Step EJ.2

What to build

A dev-apply sub-agent run that makes zero .py edits is treated as SUCCESS (if not validate or not changed_py: return, src/orchestrator/evolve_dev_apply.py L309-313) — the byte-identical candidate then burns ~4 games and passes fitness ~50% of the time.

Behind --screen-null-diff:

• detect null diffs (empty changed_py OR every changed file AST-equivalent to its before-snapshot — comment/formatting-only edits must count as null);
• route them through the existing retry-with-feedback machinery (L325-339) with a "no semantic change" feedback message;
• on final-attempt null, raise DevApplyNullDiffError.

In scripts/evolve.py, catch it (both the serial path and the --concurrency>1 worker path via scripts/evolve_worker.py) and emit a phase: "fitness", outcome: "screen-null-diff" results row WITHOUT playing games and WITHOUT incrementing retry_count (do NOT reuse the crash handler at L2932-2958 unchanged — it evicts and increments); the imp returns to active; evict after 2 consecutive null-diffs via a new additive PerItemState.consecutive_null_diffs field. Verify whether frontend/src/hooks/useEvolveRun.ts enumerates outcome strings; bump cacheKey if so.

Existing context

• Zero-edit-as-success at evolve_dev_apply.py L309-313; changed_py = sorted(_diff_py_snapshots(py_before, py_after)) is a content-snapshot diff (L300), so comment-only edits count as "changed" today — AST-equivalence (not just zero-edit) is load-bearing or the screen is trivially evaded.
• Existing retry-with-feedback machinery at L325-339 is reusable.
• Crash handler (scripts/evolve.py L2932-2958) flips the imp to _EVICTED and increments retry_count — the null-diff outcome must NOT reuse this path.
• AST-equivalence edge case: docstrings are AST-visible (ast keeps docstrings) → docstring-only edits classify as real changes. Accepted — docstring-only edits are near-null but rare; zero-edit + comment/format-only are the observed failure modes.
• Risk (flagged): scripts/evolve_worker.py error-propagation shape is unknown — read it first at build time; DevApplyNullDiffError could surface as a generic crash at --concurrency>1 → wrong eviction path. Explicit worker-path test required in Done-when.

Files to modify/create

• src/orchestrator/evolve_dev_apply.py (DevApplyNullDiffError + AST-equivalence check)
• scripts/evolve.py (--screen-null-diff flag + outcome handling; catch error, emit row, no retry increment)
• scripts/evolve_worker.py (propagate the error type across the concurrency>1 worker boundary)
• PerItemState.consecutive_null_diffs (additive optional field, default 0)
• frontend/src/hooks/useEvolveRun.ts (possibly — cacheKey bump only if it enumerates outcome strings strictly)
• tests (unit + integration)

Done when

Unit tests cover: zero-edit → retry feedback → error; comment-only edit classified null; real edit passes through unchanged; screen outcome does not increment retry_count; 2nd consecutive null evicts; flag OFF → today's silent-success path byte-identical; worker path propagates the error type. Integration test drives the scripts/evolve.py loop (fake run_batch/claude fns, real dispatch code) with the flag on and asserts a null-diff imp produces a screen-null-diff row and zero games. uv run pytest / mypy src bots --strict / ruff check . clean; test count ≥ baseline.

Flags (recommended)

--reviewers code (backend + tests; any useEvolveRun.ts change is a one-line cacheKey bump with no rendered-UI surface to review)

Depends on

none

Parallel-safe with

none — strictly sequential (edits scripts/evolve.py argparse+loop shared with sibling EJ steps; plan §4 mandates serial execution)

Produces

DevApplyNullDiffError + AST-equivalence check in evolve_dev_apply.py; --screen-null-diff flag + outcome handling in scripts/evolve.py + worker propagation in scripts/evolve_worker.py; PerItemState.consecutive_null_diffs; tests

Operator workflow notes (autonomous + UI-bundle + parallel)

• Runs end-to-end via /build-step; the Done when is the gate.
• Read scripts/evolve_worker.py serialization shape FIRST (flagged risk).
• Serial: shared scripts/evolve.py.

---

Synced from documentation/plans/evolve-judging-plan.md by /repo-sync at c6099d9

[aberson]

build-phase: Step EJ.2 started

Step: Mechanical null-diff screen
Phase: EJ
Flags: --reviewers code (--isolation none; serial shared-file)
Baseline tests: 1818 passing
Baseline HEAD: a87930b (EJ.1)
Started: 2026-07-06

Adds --screen-null-diff: dev-apply runs that produce zero semantic change (empty diff OR all-files AST-equivalent) route through retry-with-feedback, then raise DevApplyNullDiffError → loop emits a screen-null-diff fitness row (no games, no retry_count bump), imp stays active, evicts after 2 consecutive. Worker path propagates via the existing error_type crash-payload field. Frontend useEvolveRun.ts types outcomes as | string (permissive) → no cacheKey bump needed.

[aberson]

build-phase: Step EJ.2 complete

Verdict: PASS (iteration 2/3)
Commit: 6722a1e
Files changed: src/orchestrator/evolve_dev_apply.py, scripts/evolve.py, scripts/evolve_worker.py + tests (test_evolve_dev_apply.py, test_evolve_parallel.py, test_evolve_cli.py, test_evolve_worker.py)
Tests: 1858 passing (+40 over 1818 EJ.1 baseline)
Gates: mypy strict OK (807 files), ruff check OK, ruff format --check (no new debt)
Reviewers: 4 code — correctness CLEAN (all 7 invariants line-traced), style CLEAN, bugs (1 low), test-quality (1 med + 2 low). Iteration 2 closed the medium + low.

Changes made

• --screen-null-diff (default OFF → byte-identical): a dev-apply run producing zero semantic change is no longer silent success. Null = empty diff OR every changed .py AST-equivalent to its before-snapshot (ast.dump(ast.parse(...)); comment/whitespace/format-only = null, docstring-only = REAL, unparseable-after = REAL). Null diffs route through the existing retry-with-feedback loop; a final-attempt null raises DevApplyNullDiffError.
• Loop handling (scripts/evolve.py): catches the error in BOTH the serial path and the --concurrency>1 worker path; emits a phase:"fitness", outcome:"screen-null-diff" row with zero games and no retry_count bump (does NOT reuse the crash-evict path); imp returns to active; evicts after 2 consecutive null-diffs via additive PerItemState.consecutive_null_diffs (resets on a real games-played eval).
• Worker propagation: --screen-null-diff reaches the subprocess via its argv; the null-diff error propagates back through the crash payload's error_type, compared refactor-safely against DevApplyNullDiffError.__name__ and routed to the screen handler (bypassing crash eviction).
• Frontend: confirmed no change — useEvolveRun.ts types outcomes as | string, OutcomeBadge/classifyOutcome fall back to neutral on unknown values → no cacheKey bump.

Review findings addressed (iteration 1 → 2)

• MED (test): producer→consumer round-trip gap — the worker's own null-diff plumbing was untested and the routing used a bare string literal. Fixed: parent + tests compare DevApplyNullDiffError.__name__; new test_screen_flag_wraps_spawn_and_serializes_null_diff_error drives the real worker main(["--screen-null-diff",…]), asserts the flag arrives bound True AND the real _write_crash serializes error_type == DevApplyNullDiffError.__name__.
• LOW (bug): non-UTF-8 .py → UnicodeDecodeError escaped except OSError → broadened to (OSError, ValueError), biases to REAL per docstring.
• LOW (test): exercised the attempts=None → "retries" fallback path.

Byte-identical defaults verified: flag OFF → the functools.partial wrap is skipped, worker passes spawn_dev_subagent straight, and if not validate or not changed_py: return still short-circuits.

[aberson]

Completed via /build-phase (EJ.2, commit 6722a1e). Gates green: pytest 1858, mypy strict, ruff. Producer-side worker round-trip test closes the --concurrency>1 propagation risk. See result comment above.