Phase EJ Step 5: Refresh-time proposal dedup

[aberson]

Build step 5 of 8 — Phase EJ (umbrella #281)

Build doc: documentation/plans/evolve-judging-plan.md §7 Step EJ.5

What to build

The pool-refresh site appends generate_pool output with zero dedup (scripts/evolve.py L4128-4132) — near-duplicate titles re-enter and burn ~4-6 games each.

Behind --refresh-dedup: normalize titles (casefold, strip punctuation/whitespace) and drop any fresh imp that:

• (a) exact-matches an in-run promoted/stacked title,
• (b) has difflib.SequenceMatcher ratio ≥ 0.85 vs any existing pool title, or
• (c) duplicates another imp within the same fresh batch.

One audit row per drop (phase: "pool_dedup", the matched title + ratio in reason). Accept-short after dedup — never re-call generate_pool to top up (short-pool retry raises on persistent shortfall, evolve.py L1740-1744; the pool-exhausted stop only triggers at 0 active, so a short pool is safe).

Existing context

• Refresh append: scripts/evolve.py L4128-4132 — for offset, imp in enumerate(fresh_imps): pool.append(imp); per_item_state[start_idx+offset] = PerItemState(). No dedup.
• Dedup is mechanical, not LLM: stdlib-only (difflib.SequenceMatcher), deterministic, auditable. The local_judge semantic pre-filter scored 6/10 in the investigation and is deferred.
• generate_pool short-pool retry raises (L1740-1744) — accept-short is mandatory.

Files to modify/create

• scripts/evolve.py (_normalize_title + dedup filter + --refresh-dedup flag; pool_dedup audit rows)
• tests

Done when

Unit tests cover: exact promoted-title drop; 0.85-similar drop; sub-threshold survives; intra-batch duplicate drop; audit row shape; flag OFF → byte-identical append. uv run pytest / mypy src bots --strict / ruff check . clean; test count ≥ baseline.

Flags (recommended)

--reviewers code

Depends on

none

Parallel-safe with

none — strictly sequential (edits scripts/evolve.py argparse+loop shared with sibling EJ steps; plan §4 mandates serial execution)

Produces

_normalize_title + dedup filter + --refresh-dedup flag in scripts/evolve.py; pool_dedup audit rows; tests

Operator workflow notes (autonomous + UI-bundle + parallel)

• Runs end-to-end via /build-step; the Done when is the gate.
• No --ui — backend + tests.
• Serial: shared scripts/evolve.py.

---

Synced from documentation/plans/evolve-judging-plan.md by /repo-sync at c6099d9

[aberson]

build-phase: Step EJ.5 started

Step: Refresh-time proposal dedup
Phase: EJ · Depends on: none
Flags: --reviewers code (--isolation none)
Baseline tests: 1923 passing
Baseline HEAD: d69ae6a (EJ.4)
Started: 2026-07-06

Behind --refresh-dedup: at the pool-refresh append site, normalize fresh-imp titles (reusing EJ.1's normalize_prior_title — one source of truth) and drop any that exact-match an in-run promoted title, score difflib.SequenceMatcher ratio ≥ 0.85 vs any existing pool title, or duplicate another imp in the same fresh batch. One pool_dedup audit row per drop. Accept-short (never re-call generate_pool). Default OFF → byte-identical append.

[aberson]

build-phase: Step EJ.5 complete

Verdict: PASS (iteration 2/3)
Commit: 5ca62ca
Files changed: scripts/evolve.py, tests/test_evolve_cli.py
Tests: 1938 passing (+15 over 1923 EJ.4 baseline)
Gates: mypy strict OK (808 files), ruff check OK, ruff format --check (no new debt)
Reviewers: 4 code — correctness CLEAN (7 invariants line-verified), bugs CLEAN (1 low: cross-lineage note, documented), test-quality (1 med + 1 low), style (3 med + 1 low). Iteration 2 closed all.

Changes made

• --refresh-dedup (default OFF → byte-identical): at the pool-refresh append, drop a fresh imp when its normalized title (a) exact-matches an in-run promoted title, (b) scores difflib.SequenceMatcher ratio ≥ 0.85 vs any existing pool title, or (c) duplicates an earlier survivor in the same batch. One pool-dedup audit row per drop (phase: pool_dedup, outcome: pool-dedup).
• One source of truth: reuses EJ.1's normalize_prior_title (imported, identity-tested via a monkeypatch spy — no duplicate normalizer); _REFRESH_DEDUP_THRESHOLD is a single private named constant.
• Accept-short: never re-calls generate_pool to top up after drops; empty/None-normalized titles are always KEPT (safe direction); pool-wide dedup spans lineages (docstring notes this for the --lineages+--refresh-dedup combination).

Review findings addressed (iteration 1 → 2)

• 3 style MED: privatized _REFRESH_DEDUP_THRESHOLD; hyphenated the outcome value (pool-dedup, phase stays pool_dedup); fixed the colliding test section header to # Phase EJ.5: sibling style.
• 1 test MED: added test_dedup_keeps_empty_normalized_title (pins the safe-direction guard).
• lows: boundary-straddle test (~0.85±ε) + inclusive->= pin; audit-row key ordering (outcome penultimate); cross-lineage docstring note.

Byte-identical off verified (duplicate-heavy batch, flag off → all appended, zero drop rows). Through-the-caller CLI tests drive the real run_loop.

[aberson]

Completed via /build-phase (EJ.5, commit 5ca62ca). Gates green: pytest 1938, mypy strict, ruff. Reuses EJ.1 normalize_prior_title (one source of truth). See result comment above.