Repository navigation
The task packages of benchmark v1.1, the version evaluated in the paper.
Run from Harbor Hub
31 of the 32 tasks are published as the public Harbor dataset accelerate-bench/accelerate-bench (tag v1.1); ICON is left out because its grid files must be downloaded from their official source.
harbor run -d accelerate-bench/accelerate-bench@v1.1 -a claude-code -m anthropic/MODEL_ID --ak reasoning_effort=high -n 1Download
The 32 tasks are Harbor task packages on Hugging Face (access on request):
| File | Size | SHA-256 |
|---|---|---|
accelerate-bench-v1.1-harbor-tasks.tar.gz |
2,559,519,483 bytes | 2bb8944ae801a6a6bf011e8e3059e6b2f33e98b11e327077ff8caf9077f1f77e |
third-party-sources/fluidity-ppa-sources.tar |
228,085,760 bytes | f5adf308c8da02403b0fa456dcbdf6612b67b74b3db2e7a69e1dadccf2d48d26 |
Dataset revision 9b7825dbee991573edbc290f6effaffbffff8d7e. python scripts/download_data.py downloads the archive at this revision and checks every task file against canonical/manifest.json (SHA-256 3027144da88fcd87db9be2ba713c297bb188572ea5e7235b6e9aa4d4f8e6661e); see the quickstart.
Contents
Each task is a Harbor task directory with task.toml, instruction.md, environment/ (the agent's environment, with the application source and the public-test service) and tests/ (the private verifier). The archive holds 3,772 task files; ICON's two grid files are downloaded from their official source, giving 3,774.
Changes from task source 20261010-v5-timing3
Version 1.1 takes the following files from the task packages used in the v1.1 runs:
- MHM2: task version 1.1.1, with a revised statement, revised correctness checks (partition-invariant comparison of FASTA, GFA and SAM-derived depths) and the updated reference source archive.
- Fluidity: the output parser accepts all valid VTK appended encodings (UInt32 or UInt64 headers, either byte order, uncompressed data); the Ubuntu packages the task needs are bundled in
.fluidity-ppa/instead of being installed from the online PPA. - SU2: verifier time limit 14,000 s instead of 3,600 s; command timeouts are reported as evaluation errors.
- IO500: the system-call trace parser accepts restarted
wait4records, and strace runs with--seccomp-bpf.
Several of the bundled Fluidity packages are LGPL or GPL licensed. Their source packages, checked against the PPA's source index, are attached to this release as fluidity-ppa-sources.tar (also in the dataset under third-party-sources/).
Results
Paper Table 1 reports 16 agents on this version; see the leaderboard.