feat: add SWE-rebench-V2 TaskSet

[rasdani]

Summary

• Adds SWERebenchV2TaskSet backend (swerebench-v2) for nebius/SWE-rebench-V2 — 32 079 real multilingual bug-fix instances across 20 languages, each shipping a pre-built docker image, gold patch, test_patch, and an install_config dict pinning per-instance test_cmd + log_parser.
• Grading mirrors upstream scripts/eval.py: apply patch then test_patch with git apply -v --3way --recount --ignore-space-change --whitespace=nowarn, run the pinned test_cmd, parse output with the parser named by install_config.log_parser, check FAIL_TO_PASS ∪ PASS_TO_PASS ⊆ PASSED with timing-suffix normalization on both sides. Workdir is /{repo-name} (upstream convention), not /testbed.
• Vendors upstream lib/agent/log_parsers.py as swe_rebench_v2_log_parsers.py (upstream isn't pip-installable). Attribution header + # ruff: noqa on the vendored module; sole edit vs. upstream is inlining TestStatus (replaces the unresolvable from lib.agent.swe_constants import TestStatus). 76 parsers covering all 20 supported languages.

Files

• verifiers/envs/experimental/composable/tasksets/swe/swe_rebench_v2.py — class + rubric (new)
• verifiers/envs/experimental/composable/tasksets/swe/swe_rebench_v2_log_parsers.py — vendored (new)
• verifiers/envs/experimental/composable/tasksets/swe/swe_tasksets.py — make_swerebench_v2_taskset + swerebench-v2 in the make_swe_taskset factory map
• verifiers/envs/experimental/composable/tasksets/swe/__init__.py, verifiers/envs/experimental/composable/tasksets/__init__.py — re-exports

Drop-in via make_swe_taskset(backend="swerebench-v2") — no research-environments changes needed (rlm-swe picks this up automatically like swebench / swelego-real / etc).

Test plan

• uv run ruff check … + uv run pre-commit run --files … — clean (vendored module excluded via # ruff: noqa).

• Import smoke: dataset loads (32 079 rows), NAME_TO_PARSER registers 76 parsers, factory dispatch works.

• Live gold-patch validate — await make_swerebench_v2_taskset().validate(n=1, concurrency=1) → valid=True, 79s (TS instance, parse_log_js_4).

• Multi-language validate sweep — validate(n=1) per language, 4 distinct parsers, all green:

  language	parser	image (first row)	elapsed	valid
  python	parse_log_pytest	docker.io/swerebenchv2/wtforms-wtforms:614-848d28d	41.4s	✅
  java	parse_java_mvn	docker.io/swerebenchv2/crawler-commons-crawler-commons:227-ab9e33a	52.1s	✅
  go	parse_log_gotest	docker.io/swerebenchv2/filecoin-project-specs-actors:592-db0bbaa	66.4s	✅
  rust	parse_log_cargo	docker.io/swerebenchv2/dtolnay-cxx:839-c5f472e	70.8s	✅

• vf-eval end-to-end, reward=1 achieved — uv run vf-eval rlm-swe -a '{"task_type":"swerebench-v2"}' -p prime -m openai/gpt-5.4 -n 1 -r 1 -s completed cleanly with reward=1.000, stop_conditions: agent_completed=1.000, 4m18s, 18 turns / 17 tool calls. Instance: elastic__synthetics-316 (TS, parse_log_js_4).

Notes

• install_config.install is intentionally not re-run: the published images ship with deps pre-installed, matching upstream's eval.py which only runs test_cmd.
• language filter takes the dataset's native labels (python, go, rust, java, ts, js, php, kotlin, julia, elixir, scala, swift, dart, c, cpp, csharp, r, clojure, ocaml, lua); omit for the full 32k cross-language mix.
• rlm-swe host image installs are fixed upstream: PrimeIntellect-ai/rlm#33 + Idea: Expanding N-turn conversations into N different "up to k turns" conversations and compute reward separately #34 pin UV_INSTALL_DIR / UV_TOOL_BIN_DIR so both uv and its tool shims colocate at $HOME/.local/bin regardless of the image's XDG configuration.

🤖 Generated with Claude Code

---

Note

Medium Risk
Adds a sizable new sandbox-executed evaluation path that runs per-instance test_cmd and parses diverse test outputs; issues are mostly contained to the new backend but could affect sandbox stability/timeouts.

Overview
Adds a new SWE backend swerebench-v2 backed by the nebius/SWE-rebench-V2 dataset, including a new SWERebenchV2TaskSet + rubric that grades by applying the instance’s patches, running its pinned test_cmd inside the provided Docker image/workdir, and comparing parsed test statuses against FAIL_TO_PASS/PASS_TO_PASS (with timing-suffix normalization).

Wires the backend into make_swe_taskset and re-exports it from the SWE taskset packages. Vendors the upstream SWE-rebench-V2 log parser suite (large swe_rebench_v2_log_parsers.py) to match upstream evaluation behavior.

^{Reviewed by Cursor Bugbot for commit a17fa55. Bugbot is set up for automated code reviews on this repo. Configure here.}

[hallerite]

can we maybe group everything related to swe_rebench_v2 by putting it in a folder?

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit cd24c97. Configure here.}