OpenBench v1.0 — benchmark agent stacks on your own repo

Model routers pick the cheapest model that can do the job. OpenBench answers the next question: which agent harness + model combination should run your tasks on your codebase — measured on performance × efficiency (tokens and latency), not correctness alone.

What's in v1.0

• Repo-specific task sets — create tasks from your own repo (instruction + hidden checker + isolated workspace), validated for polarity before results count
• Harness × model matrix — run any task set across pi, opencode, claude code, codex, grok build, cursor, devin, and more, on frontier and open models
• BYO harnesses — add your own harness or harness variant with a thin manifest; admission-gated (obench doctor / obench gate) before comparisons
• Packs — versioned, installable-by-name bundles (org/name@version) for both task sets and harness manifests, with SHA-256 provenance and pack verify
• Efficiency-first methodology — solve rate with Wilson CIs, matched-cell cost-per-solve on a uniform token basis, contamination (rate limits/infra) excluded from denominators, docker isolation
• Docs site — leaderboard, release analyses, community results, and pack index on GitHub Pages

Headline findings so far

• Same 3 open models, 5 harnesses, 180 runs: glm-5.2 solves 11/11 under pi and ≤7/12 under every other harness; pooled pi 81% vs the field's 47–58%
• On tasks every harness solved: ~40% latency spread and ~1.5× token spread for the identical outcome
• On frontier models correctness saturates — the harness is the efficiency story

See the docs site for full analyses.

🤖 Generated with Claude Code