ai-evaluation-framework

Agent this, AI that. But we all know how much they get important things wrong. This is a framework for defining truth, and then evaluating it systematically. Science, baby.

Concretely: it benchmarks a model-based solution on the three numbers that decide whether to ship it, accuracy, latency p95, and cost. You write ground-truth cases and pick a scoring rule for each field; the framework runs your models across the cases and prints a comparison table. Model-agnostic (OpenAI, Anthropic, or any OpenAI-compatible gateway), zero npm dependencies.

Document extraction is the worked example in examples/, because there a wrong answer is unambiguously wrong. The framework itself knows nothing about documents. A case is text in, expected fields out; the input can be a string, an OCR dump, a transcript, or a PDF's text.

The example invoices, the expected values, and the price table are all placeholders you replace. The framework brings the harness and the scoring. You bring the truth.

A benchmark is only as honest as its scoring rule. "Accuracy" means nothing until you say what counts as correct for each field. An invoice number is wrong if one character moves. A vendor name printed three legitimate ways is still one right answer. A total is correct within a cent. So the rule is chosen per field:

Each case may also carry an accept list of alternate correct renderings per field.

export OPENAI_API_KEY=...        # or ANTHROPIC_API_KEY
node bin/eval.mjs examples/document-extraction --models gpt-5-mini,claude-haiku-4-5

task: invoice-field-extraction   cases: 3   price table: placeholder-2026-09-01

model                  acc     p50 ms    p95 ms    $/case     $ total
claude-haiku-4-5       94.4%   1120      1480      ~$0.0031   ~$0.0094
gpt-5-mini             88.9%   740       910       $0.0009    $0.0028

acc = fields correct / fields scored. ~ marks a cost estimated from the fallback price row.

Model routing is by id: anything starting with claude calls Anthropic, everything else calls OpenAI. Point at a gateway with --base-url. Responses cache by (model, task, case), so rescoring after a rule change is free and a prompt change invalidates only what it should. --json emits the full per-cell report for charting or for shipping to a dashboard.

A task directory holds task.json and a cases/ directory:

my-task/
  task.json
  cases/
    case-001.json
    case-002.json

task.json:

{
  "name": "invoice-field-extraction",
  "models": ["gpt-5-mini", "claude-haiku-4-5"],
  "prompt": "Extract the following fields...\n\nFields:\n{fields}\n\nDocument:\n{input}",
  "fields": [
    { "name": "invoice_number", "rule": "identifier" },
    { "name": "total", "rule": "numeric", "tolerance": 0.01 }
  ]
}

{fields} and {input} are filled per case. Each case:

{
  "id": "case-001",
  "input": "…the document text…",
  "expected": { "invoice_number": "INV-4471", "total": "1285.50" },
  "accept": { "vendor": ["Meridian Supply", "MERIDIAN SUPPLY CO."] }
}

Cost is the one number the framework computes rather than measures, from token usage the API reports and a price table in lib/pricing.mjs. The shipped prices are illustrative round numbers so the column is never empty; a model priced from the fallback row is marked with ~. Replace the table with your providers' current prices before trusting a dollar figure.

A benchmark answers one question: was it right on the cases we wrote. Users answer a different one: was it right on the case that just happened. The feedback module collects that second signal from inside a real app with almost no integration. A thumbs up. A thumbs down. On a thumbs down, one line about what went wrong.

Run the endpoint:

node bin/feedback-server.mjs --dir ./feedback-data --port 8790

Drop the widget into any page:

<script src="http://localhost:8790/feedback/widget.js"></script>
<div id="fb"></div>
<script>
  Feedback.mount(document.getElementById("fb"), {
    feature: "invoice-extraction",
    endpoint: "http://localhost:8790",
    context: { input: documentText, output: extractedFields },
  });
</script>

Every vote lands in an append-only feedback.jsonl with whatever context you passed, and stats.json keeps running counts per feature. A thumbs-down without a comment is rejected, by the widget and again by the server: "wrong" teaches nothing, "it said March but the document says May" is a future test case. Endpoints: /feedback/stats, /feedback/export (JSONL), and /feedback/review, which returns every thumbs-down reshaped as a candidate case with the input, the model's output, the user's complaint, and an empty expected for a reviewer to fill.

Feedback is stored apart from the evaluation cases on purpose. Ground truth is curated. Feedback is raw. Mixing them is how a benchmark quietly starts measuring whatever users happened to complain about.

Where the data goes. Three destinations, in order of how much volume they need:

1. Prompt engineering. Read the thumbs-down comments. Ten of them usually name the same two failure modes, and that is a prompt change you can make this afternoon and re-run the grid against.
2. New cases. The review queue is a pipeline from real failure to ground truth: a reviewer fills in expected, the candidate becomes a case, and the evaluation set grows from what actually broke instead of what you guessed might.
3. RL and preference tuning. Each vote is stored as (input, output, vote, reason), which is the preference-pair shape RLHF, DPO, and their relatives train on. With enough volume, per-feature vote rates become a reward signal and the comments become the rationale. This module collects that signal; it does not train on it, and the gap between the two is where the caution lives.

Honest limits. Thumbs are sparse (most users never vote), biased (people vote when annoyed), and gameable. Never train directly on raw votes without human review of a sample. Track the rate per feature over time; a change in the rate after a deploy is the signal, the absolute number rarely is. And treat the comments as the valuable part, which is why the widget insists on them.

node test/run.mjs
node test/feedback.test.mjs

The first runs the whole grid against a scripted mock model, so it needs no API key: it checks every scoring rule, the latency percentile, the cost math, and that a flawed set of answers scores strictly between 0 and 1. The second exercises the feedback store's rules and the HTTP surface end to end on an ephemeral port.

Apache-2.0. Built by Dreamers Inc.