An evaluation harness for LLM-assisted structured extraction. It answers, with numbers, the question every extraction pipeline eventually faces: is it actually getting better?
Extraction systems fail quietly. A prompt tweak improves course titles and silently degrades credit parsing; a model upgrade looks better on the pages you spot-checked and worse on the ones you didn't. Without a frozen reference dataset and per-field metrics, "it looks better" is the best you can say — and it is not enough to ship on.
extraction-eval provides the missing measurement layer:
- Frozen gold-set management — datasets are keyed by content hash, so a re-fetched or reprocessed document stays in the same train/eval split forever. Splits can't leak. Dataset manifests are version-pinned, so every score is traceable to the exact gold data that produced it.
- Field-level scoring — precision, recall, and F1 per field and per entity type, with exact, normalized, and fuzzy matching modes and per-field confidence buckets. You learn which fields regressed, not just that a run "did worse."
- Deterministic replay — model responses are recorded once and replayed offline, so evaluation runs are reproducible, free, and CI-safe. No live API calls in your test suite.
- LLM-as-judge adapter — optional judge-model scoring for fields where string matching is too blunt (long descriptions, paraphrased content), behind a provider-agnostic interface with an OpenAI reference implementation. Judge prompts ship with the package — a judge you can't inspect is a judge you can't trust.
- Cost accounting — token and dollar cost per run, and cost-per-accurate-field, so model comparisons weigh quality against spend instead of ignoring it.
npm install
npm run demoThe demo compares four models — two price classes across two providers (gpt-oss-20b and Claude Haiku 4.5 in the cheap class, gpt-oss-120b and Claude Sonnet 5 above them), all recorded through one Bedrock endpoint speaking two protocols — on real academic-catalog pages (Air University, Naval Postgraduate School, Haskell Indian Nations University — all public domain) plus FEMA course listings, against hand-validated gold data. The Haskell pages are two-column PDF text where course descriptions straddle the column break — the layout that quietly breaks extraction. Scoring covers the fields catalog extraction lives on: subject code, course number, title, description, credit values and ranges, credit type, prerequisites.
The comparison is config, not code: EXTRACT_MODELS takes any model list, and open-weight models sit in the same scorecard as frontier ones. On this corpus the recorded run has a concrete story to tell: the open-weight pair wins on cost-per-accurate-field, Claude Sonnet 5 wins on the hard two-column layout (F1 0.91 vs 0.81), and the numbers — not the vendor — make the choice.
Output: per-model scorecards, a per-field F1 comparison with cost-per-accurate-field, and an LLM-judge illustration for long-text fields. Everything replays from recorded cassettes — zero live API calls. Add -- --json for machine-readable output, or npm run report for a self-contained HTML version (docs/report.html). See examples/demo for corpus provenance.
The design distills lessons from production extraction systems processing six-figure document volumes:
- Freeze the gold set before you tune. If the eval data can shift under you, every improvement is suspect. Content-hash keying makes splits immutable by construction.
- Score fields, not documents. Document-level accuracy hides exactly the regressions that matter.
- Replay, don't re-call. Live-API tests are non-deterministic and expensive; recorded responses make evals a pure function.
- Judge with a visible prompt. LLM-as-judge is useful and dangerous; parity checks against human-labeled samples keep it honest.
- Count the cost. A model that is 1% more accurate at 8x the price is usually the wrong model.
Implemented and tested: gold-set management, field-level scoring (exact / normalized / numeric / fuzzy / judge), record/replay, cost accounting, OpenAI and Anthropic extraction adapters (Chat Completions and Messages protocols, routed by model id), the OpenAI judge adapter, the worked demo (federal public-domain corpus, four-model two-provider comparison), and the HTML report.
Next, in order: Anthropic judge adapter (the judge itself currently runs on one provider), judge-parity utilities (spot-checking verdicts against human-labeled samples), regression mode (same model, changed prompt), ingest-strategy comparison (the demo deliberately feeds models naive pdftotext -layout output; scoring the same corpus through a column-aware extractor — a pattern from our production pipeline — would measure how much accuracy the ingest step itself buys), a promptfoo adapter for teams already running promptfoo suites, and evaluating agentic extraction runs.