feat: Phase C — format-correction retry + per-model output paths

[coredipper]

First half of Phase C per the approved plan: format-correction retry infrastructure and the CLI plumbing for preserving per-model artifacts. Reruns land in a follow-up commit on this branch once benchmark data is in.

Problem

v0.34.5's grounded Phase 2 run had 27 of 30 submissions sanitizer-rejected with diff-format issues the sanitizer can identify (placeholder hunks, truncated bodies, invented paths) but the model never got a chance to correct. The v0.34.5 blog post named format-correction retry as the #2 wedge for moving the result; this PR implements it.

Changes

Sanitizer — reason-code surfaces

eval/_patch_sanitizer.py gets a parallel tuple-return API:

patch, reason = sanitize_with_reason(raw_patch, "django/django", tree_paths=tree)
# reason ∈ {"", "placeholder_hunk", "truncated_hunk", "overlong_hunk",
#          "malformed_metadata", "path_not_found", "ambiguous_path",
#          "empty_extraction"}

The old sanitize() is kept as a backward-compat wrapper so every existing test and caller continues to work unchanged. Internal helpers (_fuzzy_correct_paths, _validate_hunks) refactored to propagate reason codes.

Runner — retry callback plumbing

patch = _sanitize_for_submission(
    raw, repo_slug, tree_paths,
    retry_callback=lambda reason, failed: _llm_call(provider, retry_prompt(reason, failed)),
)

• _build_retry_prompt(original_prompt, reason, failed_output) builds a targeted retry prompt with reason-specific guidance (placeholder_hunk → "use real line numbers"; path_not_found → "use exact paths from the repo context"; etc.). The failed output is embedded verbatim so the model corrects rather than regenerates.
• Up to _FORMAT_RETRY_MAX = 1 retries per instance. The constant exists so a future experiment can tune it without diff churn.
• All three runners (run_baseline, run_organism, run_langgraph) accept retry_on_reject: bool = False. When True, each constructs a runner-appropriate callback — direct _llm_call for baseline; whole-organism re-run for the multi-stage conditions.

CLI

Two new flags, both opt-in so v0.34.5 behavior is preserved:

Flag	Purpose
--retry-on-reject	Enable the retry callback. Off by default.
--output PATH	Write the artifact somewhere other than eval/results/swebench_phase2.json so Phase C reruns can be preserved per-model without overwriting the v0.34.5 baseline.

Tests (20 new, all TDD RED→GREEN)

Sanitizer reason codes — tests/unit/test_patch_sanitizer.py (+12):

• Happy path returns "" reason.
• Empty input / whitespace / headers-only / prose-only → empty_extraction.
• @@ -XXX,N +XXX,N @@ → placeholder_hunk (XXX and single-letter variants).
• Declared counts > body → truncated_hunk.
• Body extra beyond counts → overlong_hunk.
• Invented path with no basename match in tree oracle → path_not_found.
• Basename matches multiple files → ambiguous_path.
• sanitize() wrapper returns sanitize_with_reason()[0] (backward-compat contract).

Retry plumbing — tests/unit/test_swebench_phase2_retry.py (+8):

• _FORMAT_RETRY_MAX == 1 (diagnostic default contract).
• No callback = old behavior.
• Callback fires on reject with reason + failed output.
• Callback does NOT fire on success.
• Retry also rejects → "" (no infinite loop, no fallback to original).
• Retry cap enforced (chronically failing model can't trigger unbounded retries).
• _build_retry_prompt embeds reason + failed output.
• Reason-specific guidance differs between codes.

Test plan

• pytest tests/unit/test_patch_sanitizer.py tests/unit/test_swebench_phase2_retry.py tests/unit/test_swebench_phase2_timeout.py tests/unit/test_swebench_phase2_identity.py tests/unit/test_patch_extraction.py tests/unit/test_repo_cache.py tests/unit/test_repo_grounding.py -q — 139/139 pass (+20 from v0.34.5's 119).
• python -m eval.swebench_phase2 --help — both new flags render cleanly.
• Live smoke on --model gemma4:latest --grounding --retry-on-reject --n 10 --output eval/results/swebench_phase2_gemma4_retry.json — follow-up commit on this branch once the 2-3h benchmark completes.

Follow-up on this branch (not in this commit)

Once this is merged (or before, if you prefer squash): rerun artifacts with the new flag against gemma4 and deepseek-r1:8b (now unblocked by PR #57's 900s timeout), Paper 5 §6.3 Phase C subsection, v0.35.0 version bump, blog follow-up, release.

🤖 Generated with Claude Code