evalgate

The build fails when your prompt gets dumber.

evalgate is prompt/agent regression CI. You write a declarative eval suite, evalgate runs it, scores it, stores a baseline, and on every pull request it re-runs the suite, computes the quality delta against the base branch, and fails the build when the score drops - then posts the delta table as a PR comment.

It runs end to end with zero API keys thanks to a deterministic mock provider, so you can try the whole thing (and the test suite) offline.

---

Why

Prompts and agents rot silently. A model swap, a "small" prompt tweak, a new tool - any of them can quietly make your system dumber, and you find out in production. Unit tests don't catch it because there's nothing throwing; the output is just worse.

evalgate treats quality like a build artifact:

• Declarative suites - cases live in version control next to your code.
• Real scorers - exact match, regex, JSON-schema, semantic similarity, LLM-as-judge, latency/cost budgets, and weighted rubrics.
• Baseline + delta - the core value. Not "is this good?" but "is this worse than it was?" - the only question CI can answer objectively.
• Provider-agnostic - a clean adapter layer for OpenAI, Anthropic, Groq, and OpenRouter, plus a deterministic mock so tests need no network.
• A real GitHub Action - drops the delta into a PR comment and gates the merge.

---

npm: https://www.npmjs.com/package/@royalpinto007/evalgate

Quickstart

npm install -D @royalpinto007/evalgate   # or: npm install -g @royalpinto007/evalgate

Write a suite (suite.eval.yaml):

name: my-agent
provider: mock          # works with no API key
threshold: 0.9          # mean score required to pass
cases:
  - id: greeting
    input:
      prompt: |
        Reply with the standard greeting.
        exactly: Hi there! How can I help you today?
    expected: "Hi there! How can I help you today?"
    scorers:
      - type: exact-match
      - type: latency
        budgetMs: 500

Run it:

npx @royalpinto007/evalgate run suite.eval.yaml

Save a baseline, then compare later runs against it:

npx @royalpinto007/evalgate baseline suite.eval.yaml --out baseline.json
npx @royalpinto007/evalgate compare suite.eval.yaml --base baseline.json --tolerance 0.01

compare exits non-zero when any case regresses beyond the tolerance - that is what fails your CI job.

Try it right now on the bundled example: npx @royalpinto007/evalgate run examples/support-agent.eval.yaml

---

Suite format

A suite is YAML or JSON with this shape:

Field	Where	Meaning
name	suite	Suite name shown in reports. Required.
provider	suite / case	Provider to call (mock, openai, anthropic, groq, openrouter).
model	suite / case	Model id. Case overrides suite.
threshold	suite	Mean score in [0,1] required for the run to pass.
cases[].id	case	Unique id. Required.
cases[].input.prompt	case	A single-string prompt.
cases[].input.messages	case	Chat messages [{role, content}] (alternative to prompt).
cases[].expected	case	Reference value shared by several scorers.
cases[].scorers	case	One or more scorer specs. Required.
cases[].tags	case	Tags for --tags filtering.

Each scorer spec has a type, an optional weight (default 1), an optional name, and scorer-specific options. A case passes when every scorer passes; its numeric score is the weighted mean of the scorer scores.

---

Scorer catalog

Scorer	Passes when	Key options
exact-match	output equals expected	expected, caseSensitive, trim
regex	output matches a pattern	pattern, flags, expectMatch
contains	all substrings present (partial credit)	value / values, caseSensitive
not-contains	no banned substring present	value / values, caseSensitive
json-schema	output is valid JSON matching a schema	schema
embedding-similarity	cosine similarity >= threshold	expected, threshold
llm-judge	a judge model scores >= threshold	criteria, expected, threshold, model
latency	call latency within budget	budgetMs
cost	estimated call cost within budget	budgetUsd
rubric	weighted criteria score >= threshold	criteria[], threshold

Two scorers are pluggable and ship with deterministic offline fallbacks so tests and the mock provider need no network:

• embedding-similarity uses the provider's embed() if present, otherwise a stable local bag-of-hashed-words embedding.
• llm-judge calls a real provider and parses a JSON {score, reason} reply; on the mock provider it computes a reproducible word-overlap score instead.

---

The PR comment

On each PR the Action re-runs the suite and upserts a single comment (it edits its own comment in place rather than stacking new ones). A regression looks like this:

---

evalgate: support-agent

FAIL - Quality regressed. 5 case(s) got worse.

Overall score: 94.2% (base) -> 60.1% (head) = -34.2pp

Case	Base	Head	Delta	Change
refund-intent-json	100.0%	0.0%	-100.0pp	down
order-id-format	100.0%	0.0%	-100.0pp	down
greeting-exact	100.0%	66.7%	-33.3pp	down
judge-helpfulness	73.6%	69.1%	-4.5pp	down
paraphrase-quality	86.0%	84.6%	-1.3pp	down

tolerance 0.0% - worst case -100.0pp

---

GitHub Action usage

name: evalgate
on:
  pull_request:
permissions:
  contents: read
  pull-requests: write        # required for the PR comment
jobs:
  quality-gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: AgentPostmortem/evalgate@v1
        with:
          suite: examples/support-agent.eval.yaml
          baseline: examples/support-agent.baseline.json
          provider: mock          # or openai / anthropic / groq / openrouter
          tolerance: "0.01"
          comment: "true"

Input	Default	Description
suite	-	Path to the eval suite (required).
baseline	evalgate.baseline.json	Baseline result to compare against.
provider	mock	Provider to run with.
model	-	Optional model override.
tolerance	0	Allowed per-case score drop before it's a regression.
comment	true	Upsert a PR comment with the delta table.
github-token	${{ github.token }}	Token used to post the comment.

For real providers, pass the relevant key as an env var / secret: OPENAI_API_KEY, ANTHROPIC_API_KEY, GROQ_API_KEY, or OPENROUTER_API_KEY.

---

CLI

evalgate run <suite>          Run a suite and print a report.
evalgate baseline <suite>     Run a suite and save it as a baseline.
evalgate compare <suite>      Run a suite and compare it to a baseline.
evalgate compare             Compare two existing result files (--base, --head).
evalgate init [file]          Write a starter suite you can edit.
evalgate list                 List the available scorers and providers.

Flags: --provider --model --tags --concurrency --out --md --junit --json
       --tolerance --comment --no-fail

Library

evalgate is also a library:

import { loadSuite, runSuite, compareRuns, renderCompareMarkdown } from "evalgate";

const suite = await loadSuite("suite.eval.yaml");
const head = await runSuite(suite);
const baseline = JSON.parse(await fs.readFile("baseline.json", "utf8"));
const cmp = compareRuns(baseline, head, { tolerance: 0.01 });
if (cmp.regressed) process.exit(1);
console.log(renderCompareMarkdown(cmp, suite.name));

Register your own scorer or provider:

import { defaultScorerRegistry, defaultRegistry } from "evalgate";

const scorers = defaultScorerRegistry().register({
  type: "starts-with",
  score: (spec, ctx) => ({
    type: "starts-with",
    name: "starts-with",
    weight: spec.weight ?? 1,
    passed: ctx.output.startsWith(String(spec.prefix)),
    score: ctx.output.startsWith(String(spec.prefix)) ? 1 : 0,
    reason: "prefix check",
  }),
});

---

Development

npm install
npm run build       # compile to dist/
npm test            # vitest
npm run typecheck
npm run lint

Contributing

Issues and PRs are welcome. Please keep the mock-first rule: every feature must run and be tested without network access or API keys. Run npm test, npm run typecheck, and npm run lint before opening a PR, and add a case to the example suites when you add a scorer.

License

MIT (c) royalpinto007