v0.1.0 - Initial prerelease

Initial prerelease

One Action to validate configuration, lint, run Vally evals, and update one
sticky PR comment with the selected suite, native eval names and verdicts.
The interface may change during the 0.x pilot.

Usage

name: Evals
on:
  pull_request:
  workflow_dispatch:

permissions:
  contents: read

jobs:
  evals:
    if: github.event_name == 'workflow_dispatch' || github.event.pull_request.head.repo.full_name == github.repository
    runs-on: ubuntu-latest
    timeout-minutes: 15
    permissions:
      contents: read
      copilot-requests: write
      pull-requests: write
    concurrency:
      group: evals-${{ github.workflow_ref }}
      cancel-in-progress: false
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
        with:
          persist-credentials: false
      - uses: github/evals-action@v0.1.0
        with:
          working-directory: evals
          suite: smoke

Replace the project path and suite with your own. No command or manual token
input is required. Organization Copilot usage must be enabled by policy.
Repository access and organization Actions policy still apply while this
Action repository is private.

Included

• Default command: all: validate, lint, run after successful lint, and report.
• Explicit lint, run, and report commands for custom/separate-job flows.
• Automatic GitHub job-token authentication; no stored PAT is needed.
• One sticky PR comment with lint/run status, eval names and evidence links.
• Vally defaults to latest stable; vally-version accepts an exact stable
  version without requiring a new Action release.
• Minimal structured artifacts by default; raw transcripts are opt-in.
• Linux x64 GitHub-hosted runner support using the packaged Node 24 Action.

Important limitations

Default all mode intentionally shares a combined Copilot/PR-write job token.
Use trusted reviewed eval sources; plugins, setup scripts and model tools may
access that token. Explicit commands in separate jobs provide optional
permission isolation. Fork PRs and pull_request_target are excluded.

Latest Vally and its dependency graph may change between jobs. An exact CLI
pin does not freeze upstream transitive ranges. Installed package integrity,
job-local lock identity and native evidence contracts are checked; this is not
a sandbox or a guarantee against malicious dependencies.

The Action's reserved plan command is unsupported and fails explicitly.
Manual dispatch produces summaries/artifacts without posting PR comments.

Release provenance

• Merged PR: #6
• Release commit: 83fa12332c3e44ad899f347eaf153a0d9b6432dc
• Post-merge model-free CI: https://github.com/github/evals-action/actions/runs/39343655011
• Live default-path qualification on the same source tree:
  https://github.com/github/evals-action/actions/runs/39342109327

See README and
SECURITY
for inputs, advanced layouts and trust boundaries.