Skip to content

Releases

Accelerate Bench v1.1

Choose a tag to compare

@AdAstraAbyssoque AdAstraAbyssoque released this 16 Nov 09:24

The task packages of benchmark v1.1, the version evaluated in the paper.

Run from Harbor Hub

31 of the 32 tasks are published as the public Harbor dataset accelerate-bench/accelerate-bench (tag v1.1); ICON is left out because its grid files must be downloaded from their official source.

harbor run -d accelerate-bench/accelerate-bench@v1.1 -a claude-code -m anthropic/MODEL_ID --ak reasoning_effort=high -n 1

Download

The 32 tasks are Harbor task packages on Hugging Face (access on request):

File Size SHA-256
accelerate-bench-v1.1-harbor-tasks.tar.gz 2,559,519,483 bytes 2bb8944ae801a6a6bf011e8e3059e6b2f33e98b11e327077ff8caf9077f1f77e
third-party-sources/fluidity-ppa-sources.tar 228,085,760 bytes f5adf308c8da02403b0fa456dcbdf6612b67b74b3db2e7a69e1dadccf2d48d26

Dataset revision 9b7825dbee991573edbc290f6effaffbffff8d7e. python scripts/download_data.py downloads the archive at this revision and checks every task file against canonical/manifest.json (SHA-256 3027144da88fcd87db9be2ba713c297bb188572ea5e7235b6e9aa4d4f8e6661e); see the quickstart.

Contents

Each task is a Harbor task directory with task.toml, instruction.md, environment/ (the agent's environment, with the application source and the public-test service) and tests/ (the private verifier). The archive holds 3,772 task files; ICON's two grid files are downloaded from their official source, giving 3,774.

Changes from task source 20261010-v5-timing3

Version 1.1 takes the following files from the task packages used in the v1.1 runs:

  • MHM2: task version 1.1.1, with a revised statement, revised correctness checks (partition-invariant comparison of FASTA, GFA and SAM-derived depths) and the updated reference source archive.
  • Fluidity: the output parser accepts all valid VTK appended encodings (UInt32 or UInt64 headers, either byte order, uncompressed data); the Ubuntu packages the task needs are bundled in .fluidity-ppa/ instead of being installed from the online PPA.
  • SU2: verifier time limit 14,000 s instead of 3,600 s; command timeouts are reported as evaluation errors.
  • IO500: the system-call trace parser accepts restarted wait4 records, and strace runs with --seccomp-bpf.

Several of the bundled Fluidity packages are LGPL or GPL licensed. Their source packages, checked against the PPA's source index, are attached to this release as fluidity-ppa-sources.tar (also in the dataset under third-party-sources/).

Results

Paper Table 1 reports 16 agents on this version; see the leaderboard.