Skip to content

GitHub Action: annotate the PR with an eval diff summary #5

Description

@abhishekmanyam

Summary

Have the GitHub Action post an eval summary back onto the PR — a sticky comment with the pass-rate delta vs. the baseline, so reviewers see regressions without opening CI logs.

Proposed behavior

After evalcore run ... --baseline <ref>, the action emits a machine-readable summary (we already produce JSON/JUnit reporters) and a composite step upserts a single PR comment:

  • overall pass/fail and exit code,
  • pass-rate and mean-score delta vs. baseline,
  • the newly-regressed cases (names only),
  • a collapsible link to the HTML report artifact.

Notes

  • Reuse existing reporters — no new output format in the engine unless something's genuinely missing.
  • Sticky comment (update in place), not a new comment per run.
  • Must degrade gracefully on forks / missing token (skip, don't fail the run).

Good candidate for someone comfortable with GitHub Actions + a bit of the JSON reporter shape.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requesthelp wantedExtra attention is needed

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions