v0.2.0 - Quality, token, and duration PR reporting

Quality, token, and duration reporting prerelease

Surfaces quality scores, duration, token usage, and per-case outcomes in sticky PR reports and workflow job summaries so authors and marketplace maintainers can review skill improvements at a glance.

Highlights

• Adds a compact Quality & token summary table (Passed, Failed, Avg Score, Total Tokens, Total Duration) to sticky PR reports and job summaries when trial metrics are recorded.
• Surfaces per-case quality score percentages, durations, token usage (in/out), and failed check details directly in the ### Evals table and expandable failure details.
• Preserves existing lint/run/report contracts and zero-leakage handling when optional metrics are absent.
• Includes open-source readiness, contribution terms, license alignment, and Vally telemetry preservation updates (#12, #13, #14, #18, #20, #21, #24).

Release provenance

• Merged PR: #24
• Release commit: 6e4c077e92194c7e75663d8d782426841047e887
• Post-merge model-free CI: https://github.com/github/evals-action/actions/runs/42852601761
• Post-merge CodeQL: https://github.com/github/evals-action/actions/runs/42852600544

v0.1.0 - Initial prerelease

Initial prerelease

One Action to validate configuration, lint, run Vally evals, and update one
sticky PR comment with the selected suite, native eval names and verdicts.
The interface may change during the 0.x pilot.

Usage

name: Evals
on:
  pull_request:
  workflow_dispatch:

permissions:
  contents: read

jobs:
  evals:
    if: github.event_name == 'workflow_dispatch' || github.event.pull_request.head.repo.full_name == github.repository
    runs-on: ubuntu-latest
    timeout-minutes: 15
    permissions:
      contents: read
      copilot-requests: write
      pull-requests: write
    concurrency:
      group: evals-${{ github.workflow_ref }}
      cancel-in-progress: false
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
        with:
          persist-credentials: false
      - uses: github/evals-action@v0.1.0
        with:
          working-directory: evals
          suite: smoke

Replace the project path and suite with your own. No command or manual token
input is required. Organization Copilot usage must be enabled by policy.
Repository access and organization Actions policy still apply while this
Action repository is private.

Included

• Default command: all: validate, lint, run after successful lint, and report.
• Explicit lint, run, and report commands for custom/separate-job flows.
• Automatic GitHub job-token authentication; no stored PAT is needed.
• One sticky PR comment with lint/run status, eval names and evidence links.
• Vally defaults to latest stable; vally-version accepts an exact stable
  version without requiring a new Action release.
• Minimal structured artifacts by default; raw transcripts are opt-in.
• Linux x64 GitHub-hosted runner support using the packaged Node 24 Action.

Important limitations

Default all mode intentionally shares a combined Copilot/PR-write job token.
Use trusted reviewed eval sources; plugins, setup scripts and model tools may
access that token. Explicit commands in separate jobs provide optional
permission isolation. Fork PRs and pull_request_target are excluded.

Latest Vally and its dependency graph may change between jobs. An exact CLI
pin does not freeze upstream transitive ranges. Installed package integrity,
job-local lock identity and native evidence contracts are checked; this is not
a sandbox or a guarantee against malicious dependencies.

The Action's reserved plan command is unsupported and fails explicitly.
Manual dispatch produces summaries/artifacts without posting PR comments.

Release provenance

• Merged PR: #6
• Release commit: 83fa12332c3e44ad899f347eaf153a0d9b6432dc
• Post-merge model-free CI: https://github.com/github/evals-action/actions/runs/39343655011
• Live default-path qualification on the same source tree:
  https://github.com/github/evals-action/actions/runs/39342109327

See README and
SECURITY
for inputs, advanced layouts and trust boundaries.