OpenBench v1.0 — benchmark agent stacks on your own repo

Model routers pick the cheapest model that can do the job. OpenBench answers the next question: which agent harness + model combination should run your tasks on your codebase — measured on performance × efficiency (tokens and latency), not correctness alone.

What's in v1.0

• Repo-specific task sets — create tasks from your own repo (instruction + hidden checker + isolated workspace), validated for polarity before results count
• Harness × model matrix — run any task set across pi, opencode, claude code, codex, grok build, cursor, devin, and more, on frontier and open models
• BYO harnesses — add your own harness or harness variant with a thin manifest; admission-gated (obench doctor / obench gate) before comparisons
• Packs — versioned, installable-by-name bundles (org/name@version) for both task sets and harness manifests, with SHA-256 provenance and pack verify
• Efficiency-first methodology — solve rate with Wilson CIs, matched-cell cost-per-solve on a uniform token basis, contamination (rate limits/infra) excluded from denominators, docker isolation
• Docs site — leaderboard, release analyses, community results, and pack index on GitHub Pages

Headline findings so far

• Same 3 open models, 5 harnesses, 180 runs: glm-5.2 solves 11/11 under pi and ≤7/12 under every other harness; pooled pi 81% vs the field's 47–58%
• On tasks every harness solved: ~40% latency spread and ~1.5× token spread for the identical outcome
• On frontier models correctness saturates — the harness is the efficiency story

See the docs site for full analyses.

🤖 Generated with Claude Code

v0.1.0 — Same model, different wrapper

First release: the complete M3→M4 benchmark arc.

Findings: harness efficiency separates up to 8× in tokens where correctness saturates; 3 of 4 open models (GLM-5.2, DeepSeek V4 Flash, Kimi K2.7) reach frontier parity on our tasks — the entire 72-run open-model matrix cost ~$1.02.

In the box: 5 harness adapters (codex, pi, opencode, cursor, devin), open-model support via first-party APIs, validated partial-credit tasks + an imported Exercism tier, Docker isolation mode, Wilson-CI statistics, transcript persistence with PII scrubbing, and 4 committed datasets.

📖 Start with WRITEUP.md · 📊 Live results · 🔁 Reproduce for ~$1: see README.