Summary
Have the GitHub Action post an eval summary back onto the PR — a sticky comment with the pass-rate delta vs. the baseline, so reviewers see regressions without opening CI logs.
Proposed behavior
After evalcore run ... --baseline <ref>, the action emits a machine-readable summary (we already produce JSON/JUnit reporters) and a composite step upserts a single PR comment:
- overall pass/fail and exit code,
- pass-rate and mean-score delta vs. baseline,
- the newly-regressed cases (names only),
- a collapsible link to the HTML report artifact.
Notes
- Reuse existing reporters — no new output format in the engine unless something's genuinely missing.
- Sticky comment (update in place), not a new comment per run.
- Must degrade gracefully on forks / missing token (skip, don't fail the run).
Good candidate for someone comfortable with GitHub Actions + a bit of the JSON reporter shape.
Summary
Have the GitHub Action post an eval summary back onto the PR — a sticky comment with the pass-rate delta vs. the baseline, so reviewers see regressions without opening CI logs.
Proposed behavior
After
evalcore run ... --baseline <ref>, the action emits a machine-readable summary (we already produce JSON/JUnit reporters) and a composite step upserts a single PR comment:Notes
Good candidate for someone comfortable with GitHub Actions + a bit of the JSON reporter shape.