Lexical divergence improvements

[guillaumemeyer]

@poorvith-mp in my benchmarks, I noticed we have great results, but at the cost of high lexical divergence (~0.92).
Not surprising, as we are currently "forcing" to rewrite everything.

Here are suggestions to instead find the best minimal rewrite that satisfies the detector.

Tell me what you think about it.

---

Plan: "just enough" rewriting in three levers

Lever 1 — a minimal rewrite strength (prompt-level budget)
Add a new strength to PROMPTS and the --strength choices in rewrite_text.py:

Instruction: change as little as possible — keep sentence structure, word order, and every token that can stay; change only function words and a few non-essential content words; an explicit budget like "change well under half the words; if a handful of swaps clears the mark, do that"; preserve facts/numbers/names.
This is the single biggest lever: the same loop now produces ~0.3–0.5 divergence attempts instead of ~0.92, and the first-pass stop rule becomes a feature instead of a bug (a passing minimal edit is returned untouched).

Lever 2 — a strength ladder across loops (incremental escalation)
New --ladder minimal,paraphrase option that maps loop index → strength, e.g. minimal → humanize → paraphrase:

Loop 1 rewrites minimally; if detection still fails, loop 2 rewrites more forcefully; only when earlier rungs fail do you pay for the full paraphrase.
Combined with the existing stop-on-pass, this is the "rewrite just enough" loop: the returned document is the first rung that clears, so divergence is the smallest the detector will tolerate. Lex div in the report naturally collapses toward the minimal rung's value.
Caveat to document: --max-loops must be ≥ ladder length to be useful; the bench would expose --rewrite-ladder mirroring --rewrite-loops.

Lever 3 — minimality-aware selection (use the candidate budget for quality)
Free change: when a pass happens, return the least-diverged passing candidate rather than the last one. With today's early-stop there's only ever one pass, so this is behavior-neutral until combined with the next point.
Opt-in --minimal-select: generate all --candidates each round (no early break), then select the least-diverged passing candidate (optionally with a safety margin: minimize divergence subject to score ≤ threshold − ε, so the win isn't a knife-edge). This finally makes paraphrase:3 spend its budget on selection instead of wasting it.

Optional Lever 4 — segment-wise incremental splicing (extent, not force)
Rewrite one sentence at a time, splice it back into the original, re-detect, move to the next sentence only while detection still fails. This is the literal "more incremental" interpretation.

[poorvith-mp]

Went and read the actual current rewrite_text.py on main before replying, not just working off the description here, since the loop mechanics matter for which of these are cheap and which aren't.

Confirmed diagnosis first: the 0.92 isn't surprising once you look at what's actually running. DEFAULT_CANDIDATES = 1 and DEFAULT_MAX_LOOPS = 1, so out of the box it's one generation, no comparison, no retry. And every strength in PROMPTS right now (paraphrase, humanize, structural, backtranslate, code) is written as a "rewrite everything" instruction. There's no dial between "full rewrite" and "nothing." So even when someone does raise --candidates/--max-loops, the loop (rewrite_text.py:418-456) generates one candidate, checks it, and breaks the instant passed_i is True. There is never more than one passing attempt to choose between right now, the loop stops before a second one ever gets generated.

Lever 1 is the one I'd ship first and alone. It's the smallest change with the biggest effect, add a minimal entry to PROMPTS and to --strength's choices, same shape as the five that already exist. Because of the stop-on-first-pass behavior above, this doesn't even need the other levers to help, a minimal prompt that passes on attempt one already collapses divergence toward whatever "just enough" looks like for that model, for free.

Lever 3, the opt-in --minimal-select, is correctly described as free for the default case, and I'd double confirm why: today, with early-stop, there is exactly one passing attempt ever recorded, so "return the least-diverged passing one" and "return the only passing one" are the same statement. The opt-in version (finish the full round, then pick min-divergence among passes with the epsilon margin) is a real, contained change though, not free, it means gating that break on the flag and swapping the final selection at rewrite_text.py:479-484 from "last attempt" to "min divergence where passed". Clean scope, doesn't touch the loop structure otherwise.

Lever 2, the ladder, is the one I'd flag as costing more than it looks like. prompt = build_prompt(strength, text, ...) is built once, before the loop starts (line 360), and reused unchanged across every loop iteration right now, raising --max-loops today just retries the same prompt hoping temperature gives you a different roll. A ladder means moving that call inside the for loop in range(n_loops) and keying strength off loop. Straightforward given the loop already exists from #153, but it's a real structural edit, not a flag. Worth doing, just sequencing it after 1 and 3.

Lever 4 I'd split off entirely. Splicing per-sentence changes what a "detection" even means mid-document, and probably deserves its own issue once 1 through 3 are in and you can see whether the ladder alone gets divergence low enough without going that far.

One unrelated thing I hit while tracing this: _select_candidate() is dead in production now, rewrite() stopped calling it when the inline loop selection landed in #153, the only thing still touching it is test_select_candidate_prefers_more_divergent. Small cleanup, not blocking, just flagging since I was already in the file.

If it's useful I can take Lever 1 plus the --minimal-select half of Lever 3 as a PR, that's a self-contained slice that doesn't wait on the ladder.

[guillaumemeyer]

cc @bentalay