evalgate

Unit tests exist for code; prompts get vibes. evalgate runs YAML-defined eval cases against your LLM prompts on every pull request, posts a pass/fail report as a sticky PR comment, and fails the build when outputs regress. A one-word prompt change that breaks tone, format, or a JSON contract becomes a red X on the PR.

[GitHub Marketplace badge — pending listing]

Real PR (#1): a prompt change invents a "lifetime guarantee" and drops the 30-day window — evalgate posts the report and fails the merge.

60-second quickstart

1. Copy the workflow:

# .github/workflows/evals.yml
name: Prompt evals
on: pull_request
permissions:
  contents: read
  pull-requests: write
jobs:
  evalgate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: arthurcarlsonn/evalgate@v1
        with:
          config: evals/**/*.eval.yaml
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

2. Add one eval file:

# evals/support.eval.yaml
defaults:
  provider: anthropic
  model: claude-haiku-4-5
  temperature: 0

cases:
  - id: refund-policy-tone
    prompt: |
      You are the support agent for {{brand}}.
      Customer asks: "{{question}}"
      Reply in under 120 words.
    vars:
      brand: GlowLab
      question: What is your refund policy?
    assert:
      - type: contains
        value: "30 days"
      - type: not_contains
        value: lifetime guarantee
      - type: llm_judge
        rubric: Reply is polite, mentions the refund window, and does not invent policies.

3. Add ANTHROPIC_API_KEY to your repo secrets. Open a PR — evalgate comments with the table and gates the merge.

Try it locally, free

npx @arthurcarlson/evalgate run --dry-run --config "evals/**/*.eval.yaml"

--dry-run uses the mock provider: zero API calls, zero cost. Canned outputs are read from mocks/<case-id>.txt next to your eval file, so you can develop assertions offline. The examples/ folder in this repo runs green out of the box:

npx @arthurcarlson/evalgate run --dry-run --config "examples/evals/*.eval.yaml"

Config reference

defaults:            # optional, applies to every case
  provider: anthropic   # anthropic | openai | openrouter | mock
  model: claude-haiku-4-5
  temperature: 0        # 0-2, default 0
  max_tokens: 1024

cases:
  - id: my-case            # required, unique, [a-zA-Z0-9._-]
    prompt: "inline {{var}}"   # OR prompt_file: path/relative/to/this/file
    vars: { var: value }       # {{var}} is literal string replacement;
                               # unknown variables fail before any API call
    provider: openai           # per-case overrides of any default
    model: gpt-4o-mini
    temperature: 0.2
    max_tokens: 512
    assert:                    # at least one; case passes only if all pass
      - type: contains
        value: "..."

Assertions

Type	Passes when
contains	Output includes the value (case-sensitive)
not_contains	Output does not include the value
icontains	Case-insensitive contains
regex	JS regex (as string) matches
json_valid	Output parses as JSON (one surrounding code fence is stripped)
json_schema	Parsed JSON validates against schema_file (ajv, draft 2020-12)
max_chars	Output length ≤ value
llm_judge	Judge model (temperature 0, fixed template) answers PASS; anything else fails with the returned reason

Action inputs and outputs

Input	Default
config	evals/**/*.eval.yaml	Eval file glob
min_pass_rate	100	Gate: fail the run below this pass rate (%)
comment	true	Post/update the sticky PR comment
max_cases	50	Cost guard: hard cap on cases per run
fail_on_missing_key	true	Fail fast when a needed provider key is missing

Outputs: pass_rate, passed, failed, total. The report also lands in the job summary.

Providers

Keys are read from env only — never from inputs or flags — and masked with core.setSecret:

Provider	Env var
anthropic	ANTHROPIC_API_KEY
openai	OPENAI_API_KEY
openrouter	OPENROUTER_API_KEY
mock	none

Cost notes

Defaults are chosen to keep runs cheap: temperature 0, haiku-class model, max_cases cap at 50, concurrency 4, retries only on 429/5xx. A 20-case suite on a haiku-class model typically completes in well under 90 seconds for pennies.

Troubleshooting

• "Missing provider key(s)" — add the secret to the workflow env: block. Set fail_on_missing_key: false to run anyway (those cases will fail at runtime).
• No PR comment — the job needs permissions: pull-requests: write and GITHUB_TOKEN in env:; on non-PR events the comment is skipped with a notice.
• Gate failed but you disagree — lower min_pass_rate, or fix the prompt. That's the point.

CLI

evalgate run [--config <glob>] [--dry-run] [--min-pass-rate <n>] [--max-cases <n>] [--json]

Exit codes: 0 gate passed · 1 runtime/config error · 2 gate failed.

License

MIT © Arthur Carlson