Skip to content

v0.1.0

Latest

Choose a tag to compare

@adeoluo adeoluo released this 02 Nov 04:45
· 17 commits to main since this release
bedca75

First release of llm-eval-action, a GitHub Action that gates pull requests on LLM evaluation regressions.

Use it with uses: adeoluo/llm-eval-action@v0 (moving tag for 0.x) or pin @v0.1.0.

  • Aligns per-example results by id (JSONL, JSON, evaluation-harness reports)
  • Exact McNemar test for binary metrics, paired t-test for continuous metrics, seeded paired bootstrap 95% CI, Cohen's d
  • significance (default) and threshold modes, per-metric allowed drops, lower-is-better metrics
  • Latency p50/p95 and cost deltas with optional budgets
  • Markdown report in the job summary; outputs passed, outcome, report-path, deltas, regressed-metrics
  • Baseline from a workflow artifact; optional pull request comment using the workflow's GITHUB_TOKEN
  • 112 unit tests, 100% line coverage; the self-test workflow runs the bundled action on passing and failing fixtures