Skip to content

v1.56.4

Choose a tag to compare

@github-actions github-actions released this 04 Oct 03:41
f327113

Added

  • SWE-bench outcome, medium effort (benchmarks/results/swebench-outcome-300-medium/): with the shell-search fix, claude-sonnet-5-5 at effort medium and 60 steps — distil-served 213/298 vs plain 218/298, −1.7 pts (95% CI −5.1 to +1.7), cost at parity. The second run at about −2 pts; non-inferiority still not shown. Not a long-horizon test after all (median 4 steps), so savings on long sessions remain unmeasured. $16.97.

Fixed

  • SWE-bench outcome harness: command output that is not valid UTF-8 no longer aborts a run as internal_error (decoded with errors="replace").