Skip to content

Releases

commitarena v0.1.1

Choose a tag to compare

@diegoperezg7 diegoperezg7 released this 03 Nov 18:52

Restore sqlparse license notice verbatim; fix leaderboard overflow. No engine or demo-result changes.

  • src/commitarena/demo/LICENSE-sqlparse is again a byte-for-byte copy of the upstream BSD-3-Clause license, copyright notice included.
  • Report: the leaderboard fits at 1280 px (the "Ended by" column wraps instead of pushing "Cutoff" out of the card), and the page no longer overflows horizontally at 390 px; wide tables scroll inside their card.

Install: uvx --from git+https://github.com/diegoperezg7/commitarena@v0.1.1 commitarena demo

commitarena v0.1.0

Choose a tag to compare

@diegoperezg7 diegoperezg7 released this 03 Nov 18:40

First release of commitarena: your git history is the benchmark.

What it does

  • commitarena mine <repo>: finds commits that change source and tests (Python + pytest) and keeps those whose new tests fail without the fix and pass with it (FAIL_TO_PASS), without breaking the rest (PASS_TO_PASS), validated in Docker with no network. The task prompt is built from the failing tests only; the commit message is never shown.
  • commitarena run --models ...: the same minimal agent loop with one bash tool for every model (OpenRouter, OpenAI, Mistral, Ollama). Every command runs in a container with --network none, --cap-drop ALL, CPU, memory, PID and time limits. Test files are restored before scoring. --max-cost (default 1 USD) is checked before every model call.
  • commitarena report: static HTML (EN/ES, light/dark, mobile): leaderboard with how each attempt ended, task x model matrix, pass rate before vs after each model's training cutoff, model diff next to the human fix.
  • commitarena demo: a recorded real run, no keys or Docker needed.

Recorded demo (11 fixes from sqlparse, BSD-3-Clause, 2026-11-03): Mistral Large 4 11/11 ($0.54), Ministral 8B 1/11 ($0.24), Codestral 0/11 ($0.65). Example report: https://diegoperezg7.github.io/commitarena/

Install

uvx --from git+https://github.com/diegoperezg7/commitarena@v0.1.0 commitarena demo

PyPI (uvx commitarena) is coming. The wheel and sdist are attached.