Skip to content

VulcanBench v0.1.0

Choose a tag to compare

View all tags
@github-actions github-actions released this 20 Jun 20:52
v0.1.0
8eb28ce

VulcanBench v0.1.0

First public release of VulcanBench — an open-source LLM benchmark for realistic, multi-file software engineering tasks.

Highlights

  • 52 gold-verified tasks across Python, Go, TypeScript, and Rust
  • Real tool-calling agent loop with mock, OpenAI, and Anthropic providers
  • Docker sandbox (default): isolated, non-root, network-off command execution
  • Five-metric scoring: functional, quality, security, efficiency, human_like (3-judge ensemble)
  • Suite tooling: --repeat, --max-concurrency, --max-cost, --fail-under, effort sweeps
  • Artifacts: JSONL trace, summary.json, final.patch, self-contained replay.html
  • CLI: leaderboard, report, calibration, replay
  • FastAPI backend + Next.js dashboard (filesystem or Postgres)
  • CI: lint, typecheck, ≥80% harness coverage, task validation, sandbox image builds

Quick start

git clone https://github.com/morganlinton/VulcanBench.git
cd VulcanBench
make setup
source .venv/bin/activate
vulcanbench --help

Free smoke test (no API key):

vulcanbench run --task hello-world --model mock:synthetic --sandbox local --no-judges

Real model run (build sandbox image first):

make sandbox-image
export OPENAI_API_KEY=...
vulcanbench run --task py-topo-sort-cycle --model openai:gpt-4o-mini --no-judges

Dashboard + API:

pip install -e ".[backend]"
uvicorn backend.app:app --port 8000
cd dashboard && npm install && npm run dev   # http://localhost:3000

Documentation

Notes

  • Install from source with pip install -e ".[dev,test]" — the task corpus and sandbox Dockerfiles ship in the repo clone.
  • Bring your own API keys; see README provider terms.
  • Apache 2.0 licensed.

Full changelog: CHANGELOG.md

Assets 2
Loading