Skip to content

commitarena v0.1.0

Choose a tag to compare

View all tags
@diegoperezg7 diegoperezg7 released this 03 Nov 18:40
· 6 commits to main since this release
v0.1.0
60c14db

First release of commitarena: your git history is the benchmark.

What it does

  • commitarena mine <repo>: finds commits that change source and tests (Python + pytest) and keeps those whose new tests fail without the fix and pass with it (FAIL_TO_PASS), without breaking the rest (PASS_TO_PASS), validated in Docker with no network. The task prompt is built from the failing tests only; the commit message is never shown.
  • commitarena run --models ...: the same minimal agent loop with one bash tool for every model (OpenRouter, OpenAI, Mistral, Ollama). Every command runs in a container with --network none, --cap-drop ALL, CPU, memory, PID and time limits. Test files are restored before scoring. --max-cost (default 1 USD) is checked before every model call.
  • commitarena report: static HTML (EN/ES, light/dark, mobile): leaderboard with how each attempt ended, task x model matrix, pass rate before vs after each model's training cutoff, model diff next to the human fix.
  • commitarena demo: a recorded real run, no keys or Docker needed.

Recorded demo (11 fixes from sqlparse, BSD-3-Clause, 2026-11-03): Mistral Large 4 11/11 ($0.54), Ministral 8B 1/11 ($0.24), Codestral 0/11 ($0.65). Example report: https://diegoperezg7.github.io/commitarena/

Install

uvx --from git+https://github.com/diegoperezg7/commitarena@v0.1.0 commitarena demo

PyPI (uvx commitarena) is coming. The wheel and sdist are attached.

Assets 4
Loading