Skip to content

Accelerate Bench v1.1

Latest

Choose a tag to compare

@AdAstraAbyssoque AdAstraAbyssoque released this 16 Nov 09:24
· 6 commits to main since this release

The task packages of benchmark v1.1, the version evaluated in the paper.

Run from Harbor Hub

31 of the 32 tasks are published as the public Harbor dataset accelerate-bench/accelerate-bench (tag v1.1); ICON is left out because its grid files must be downloaded from their official source.

harbor run -d accelerate-bench/accelerate-bench@v1.1 -a claude-code -m anthropic/MODEL_ID --ak reasoning_effort=high -n 1

Download

The 32 tasks are Harbor task packages on Hugging Face (access on request):

File Size SHA-256
accelerate-bench-v1.1-harbor-tasks.tar.gz 2,559,519,483 bytes 2bb8944ae801a6a6bf011e8e3059e6b2f33e98b11e327077ff8caf9077f1f77e
third-party-sources/fluidity-ppa-sources.tar 228,085,760 bytes f5adf308c8da02403b0fa456dcbdf6612b67b74b3db2e7a69e1dadccf2d48d26

Dataset revision 9b7825dbee991573edbc290f6effaffbffff8d7e. python scripts/download_data.py downloads the archive at this revision and checks every task file against canonical/manifest.json (SHA-256 3027144da88fcd87db9be2ba713c297bb188572ea5e7235b6e9aa4d4f8e6661e); see the quickstart.

Contents

Each task is a Harbor task directory with task.toml, instruction.md, environment/ (the agent's environment, with the application source and the public-test service) and tests/ (the private verifier). The archive holds 3,772 task files; ICON's two grid files are downloaded from their official source, giving 3,774.

Changes from task source 20261010-v5-timing3

Version 1.1 takes the following files from the task packages used in the v1.1 runs:

  • MHM2: task version 1.1.1, with a revised statement, revised correctness checks (partition-invariant comparison of FASTA, GFA and SAM-derived depths) and the updated reference source archive.
  • Fluidity: the output parser accepts all valid VTK appended encodings (UInt32 or UInt64 headers, either byte order, uncompressed data); the Ubuntu packages the task needs are bundled in .fluidity-ppa/ instead of being installed from the online PPA.
  • SU2: verifier time limit 14,000 s instead of 3,600 s; command timeouts are reported as evaluation errors.
  • IO500: the system-call trace parser accepts restarted wait4 records, and strace runs with --seccomp-bpf.

Several of the bundled Fluidity packages are LGPL or GPL licensed. Their source packages, checked against the PPA's source index, are attached to this release as fluidity-ppa-sources.tar (also in the dataset under third-party-sources/).

Results

Paper Table 1 reports 16 agents on this version; see the leaderboard.