In sweep #64, file_recall on swebench_verified is 1.000 for every method at every budget >= 8000 — by construction the gold files are inside the diff, so at non-trivial budgets selection never drops them. File-level recall carries zero discriminating power on this test set (B=0 -> 0.000 sanity bound still works).
The discriminating signal there is fragment_recall / fragment_precision / line_f1, which the evaluator already computes and the checkpoints already store (n_with_fragment_gold etc.), but SWEEP_TABLE's headline matrix and the per-test-set tables lead with file recall only.
To do: in aggregate_sweep.render_sweep_table (or a sibling section), render fragment-level recall/precision for test sets where file recall is saturated (mean file_recall >= 0.995 across methods), so the paper's swebench column says something. Data exists; no new runs needed.
In sweep #64, file_recall on swebench_verified is 1.000 for every method at every budget >= 8000 — by construction the gold files are inside the diff, so at non-trivial budgets selection never drops them. File-level recall carries zero discriminating power on this test set (B=0 -> 0.000 sanity bound still works).
The discriminating signal there is fragment_recall / fragment_precision / line_f1, which the evaluator already computes and the checkpoints already store (
n_with_fragment_goldetc.), but SWEEP_TABLE's headline matrix and the per-test-set tables lead with file recall only.To do: in
aggregate_sweep.render_sweep_table(or a sibling section), render fragment-level recall/precision for test sets where file recall is saturated (mean file_recall >= 0.995 across methods), so the paper's swebench column says something. Data exists; no new runs needed.