Repository navigation
First release of llm-eval-action, a GitHub Action that gates pull requests on LLM evaluation regressions.
Use it with uses: adeoluo/llm-eval-action@v0 (moving tag for 0.x) or pin @v0.1.0.
- Aligns per-example results by id (JSONL, JSON, evaluation-harness reports)
- Exact McNemar test for binary metrics, paired t-test for continuous metrics, seeded paired bootstrap 95% CI, Cohen's d
- significance (default) and threshold modes, per-metric allowed drops, lower-is-better metrics
- Latency p50/p95 and cost deltas with optional budgets
- Markdown report in the job summary; outputs passed, outcome, report-path, deltas, regressed-metrics
- Baseline from a workflow artifact; optional pull request comment using the workflow's GITHUB_TOKEN
- 112 unit tests, 100% line coverage; the self-test workflow runs the bundled action on passing and failing fixtures