Review-quality downstream experiment: fault-localization MRR + LLM-judge at matched budgets

[nikolay-e]

Context

#137 measures the agent/SWE-resolution arm; the product's stated use — LLM code review — has no downstream quality measurement at all. Everything so far is retrieval proxies. This is the second money metric and the one CI users (#145) buy.

Design (pre-registered before first run)

• Corpus: bug-fix commits with known fix locations (Defects4J + BugsInPy + a multilingual slice from Co-change mining pipeline: training/eval corpus (>=5k instances) #128), n>=150
• Task A (objective): fault localization from (diff, context) — rank suspicious regions; MRR / P@k against known fix lines
• Task B (secondary): review-comment quality via LLM-judge rubric + human spot-check on 30 instances
• Arms at matched B=8000: diffctx (deployed fusion) vs internal BM25 vs whole-changed-files vs raw-diff-only
• Stats: paired bootstrap + permutation, seeds 42/43/44; per-instance CSVs committed

Tasks

• Pre-registration doc; harness; run; artifacts
• Report tokens-to-equal-MRR alongside quality deltas (the marketing number for GitHub Action: diffctx context step for CI LLM review #145/GEO: canonical docs, comparison page, listings, name migration, bake-off benchmark #151)

Acceptance

• committed table: does better context measurably improve review outcomes at fixed budget; result stated either way

Distinct from: #137 (agent resolution). Depends on: #125 or #129.