SWE-bench Pro V2 (v2.0.0)

SWE-bench Pro V2 (v2.0.0)

The validated public split of SWE-bench Pro: 642 tasks across 11 repositories, shipped as Harbor task
directories under v2/tasks/ with verifier, reference solution and public container image, plus the HARD-51
subset (v2/hard51_ids.txt) and the locked-protocol tooling (v2/tooling/). See v2/README.md for the run recipe.

Highlights

• 731 -> 642 tasks: 89 dropped after review; 529 instructions rewritten and validated blind by a second engineer;
  214 test patches and 38 gold patches revised; test lists regenerated against repaired parsers.
• Images rebuilt from sanitised git history and published publicly at ghcr.io/scaleapi/swe-bench_pro-v2:<instance_id>.
• Verifier hardened on all 642 tasks (test-surface reset, fixture/snapshot revert, bytecode purge, index-lock fix).
• Locked protocol: agent phase offline except the model endpoint; every diff re-graded on a pristine image.
• HuggingFace ScaleAI/SWE-bench_Pro: config default is now V2 (642), hard = HARD-51, v1 = the original 731 (also tag v1.0).

Release gate (this exact tree)

Check	Result
Reference patch (-a oracle)	642 / 642 tasks resolved
Empty patch (-a nop)	642 / 642 tasks unresolved (0 spurious passes)
Images	ghcr.io/scaleapi/swe-bench_pro-v2, pulled anonymously; 642 manifests verified, 211 rebuilt images digest-identical to the graded originals
Infrastructure retries	oracle 3 trial(s), empty-patch 3 trial(s) hit Modal sandbox errors on the first pass and passed on re-run (no task-level failures)

Assets

• swe-bench-pro-v2.0.0-tasks.zip — v2/tasks, hard51_ids.txt, README.md, GATE.md, SHA256SUMS (no hidden files)
• SHA256SUMS — checksum of the zip

🤖 Generated with Claude Code