Skip to content

Releases: morganlinton/VulcanBench

Release list

v0.6.0

v0.6.0 Latest
Latest

Choose a tag to compare

View all tags
@github-actions github-actions released this 06 Jul 16:27
v0.6.0
bcce39a
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Highlights

The first feature release since v0.5.1. Two themes: a harder task tier that un-saturates the frontier, and the tooling to benchmark it affordably.

Frontier-hard task tier — net-new-algorithm tasks from real post-cutoff merged PRs (networkx Leiden, sqlglot canonicalize, three PennyLane resource-estimation templates), each individually screened to defeat low-effort frontier models, with grading calibrated to be fair-hard (difficult, not under-specified).

Cost-efficient reporting — a new-model report becomes one column instead of a full-matrix re-run, because grading is deterministic and every run records the task hash it was scored against:

  • --max-run-cost — per-run cost ceiling; stop and grade the partial result
  • compare — assemble the model × effort matrix from cached runs (baselines never re-run)
  • regrade — re-score existing runs against the current tests for $0
  • --only-missing — resume a suite run, filling only the uncached cells

Net effect: a new-model report drops from ~$100 to ~$10. See docs/QUICKSTART.md for the workflow.

Install

git clone https://github.com/morganlinton/VulcanBench.git
cd VulcanBench
make setup
source .venv/bin/activate
vulcanbench --help

Build the sandbox image once:

make sandbox-image

See CHANGELOG.md for full notes.

What's Changed

  • v2: realistic hard tier - 10 real-PR tasks, both frontier models in the 70-90% band, $7.36/run by @morganlinton in #2
  • Anthropic provider: refusal guard and output headroom for Fable 5 by @morganlinton in #3
  • Add published benchmark results to docs/results by @morganlinton in #4
  • Add 5 hard multi-site v2 task candidates (anyio, pytest, urllib3, celery, redis) by @morganlinton in #5
  • Discount OpenAI cached prompt tokens in cost accounting by @morganlinton in #7
  • Add 11 post-cutoff OSS algorithm tasks: veryhard tier reaches 5 confirmed frontier-hard by @morganlinton in #8
  • Calibrate the veryhard tier: fair-hard instead of under-specified by @morganlinton in #9
  • Add per-run cost cap and zero-cost regrade command by @morganlinton in #10
  • Add compare: assemble model×effort matrix from cached runs (freeze baselines) by @morganlinton in #11
  • Add --only-missing: resume a suite run, filling only uncached cells by @morganlinton in #12
  • docs: note the runs-dir consolidation caveat for --only-missing and compare by @morganlinton in #13
  • docs: document cost-efficient commands in the README by @morganlinton in #14
  • Release v0.6.0: frontier-hard tier + cost-efficient reporting by @morganlinton in #15

New Contributors

Full Changelog: v0.1.0...v0.6.0

Contributors

morganlinton
Assets 2
Loading

Sonnet 5 vs Opus 4.8: reasoning-effort cost-quality report

Sonnet 5 vs Opus 4.8: reasoning-effort cost-quality report

Choose a tag to compare

View all tags
@morganlinton morganlinton released this 01 Jul 13:20
report-2026-07
ebb0d81

VulcanBench v1 reasoning-effort sweep (936 runs, deterministic test grading). Sonnet 5 owns the entire cost-quality Pareto frontier and reaches 100% pass@1 at high effort; all Opus 4.8 configurations are dominated. 4-page technical report (PDF).

Loading

VulcanBench v0.1.0

VulcanBench v0.1.0

Choose a tag to compare

View all tags
@github-actions github-actions released this 20 Jun 20:52
v0.1.0
8eb28ce

VulcanBench v0.1.0

First public release of VulcanBench — an open-source LLM benchmark for realistic, multi-file software engineering tasks.

Highlights

  • 52 gold-verified tasks across Python, Go, TypeScript, and Rust
  • Real tool-calling agent loop with mock, OpenAI, and Anthropic providers
  • Docker sandbox (default): isolated, non-root, network-off command execution
  • Five-metric scoring: functional, quality, security, efficiency, human_like (3-judge ensemble)
  • Suite tooling: --repeat, --max-concurrency, --max-cost, --fail-under, effort sweeps
  • Artifacts: JSONL trace, summary.json, final.patch, self-contained replay.html
  • CLI: leaderboard, report, calibration, replay
  • FastAPI backend + Next.js dashboard (filesystem or Postgres)
  • CI: lint, typecheck, ≥80% harness coverage, task validation, sandbox image builds

Quick start

git clone https://github.com/morganlinton/VulcanBench.git
cd VulcanBench
make setup
source .venv/bin/activate
vulcanbench --help

Free smoke test (no API key):

vulcanbench run --task hello-world --model mock:synthetic --sandbox local --no-judges

Real model run (build sandbox image first):

make sandbox-image
export OPENAI_API_KEY=...
vulcanbench run --task py-topo-sort-cycle --model openai:gpt-4o-mini --no-judges

Dashboard + API:

pip install -e ".[backend]"
uvicorn backend.app:app --port 8000
cd dashboard && npm install && npm run dev   # http://localhost:3000

Documentation

Notes

  • Install from source with pip install -e ".[dev,test]" — the task corpus and sandbox Dockerfiles ship in the repo clone.
  • Bring your own API keys; see README provider terms.
  • Apache 2.0 licensed.

Full changelog: CHANGELOG.md

Loading