Phase 2: evals harness + multi-provider lineup (Anthropic, OpenAI, Gemini)

.github/workflows/ci.yml

@@ -16,6 +16,7 @@ jobs:
           node-version: 22
           cache: pnpm
       - run: pnpm install --frozen-lockfile
+      # evals imports extractkit's built types, so build first
+      - run: pnpm build
       - run: pnpm typecheck
       - run: pnpm test
-      - run: pnpm build

.gitignore

@@ -7,3 +7,4 @@ dist/
 *.log
 .turbo/
 coverage/
+.data/

CLAUDE.md

@@ -4,7 +4,7 @@ extractkit is an open-source TypeScript document-extraction engine: Zod schema +
 
 ## Current Phase
 
-**Phase 1 (core library) shipped; Phase 2 (evals) is next.** `packages/core` is implemented and tested against mock models — it has not been validated against live providers yet; that happens as Phase 2 stands up. `packages/evals` and `apps/playground` do not exist yet. Keep README/ROADMAP/docs in sync with what actually ships. Requesting the DocILE dataset token (see ROADMAP Phase 0) is a pending human action that gates the Phase 2 data pull.
+**Phase 2 (evals) harness shipped; live runs and the DocILE half are pending.** `packages/core` and the `packages/evals` harness are implemented and tested against mock models. The 25 CORD-v2 receipts are pinned in `packages/evals/data/manifest.json`; the 25 DocILE invoices are blocked on the dataset token (ROADMAP Phase 0, pending human action) — loader and curation script are ready. The eval lineup spans Anthropic, OpenAI, and Google Gemini (`packages/evals/src/models.ts`); a run includes every provider whose API key is set, or the subset named in `EVAL_PROVIDERS`. The first live eval run (needs at least one provider key — `ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, or `GOOGLE_GENERATIVE_AI_API_KEY`) doubles as core's first live-provider validation and fills the benchmark page + README table via `pnpm report`. `apps/playground` does not exist yet. Keep README/ROADMAP/docs in sync with what actually ships.
 
 ## Planned Architecture
 

README.md

@@ -11,11 +11,19 @@ TypeScript has structured-output libraries (instructor-js, AI SDK `generateObjec
 ## v1
 
 - **Core library** (`packages/core`, shipped) — Zod schema + PDF/image → validated JSON with per-field `{ value, confidence, page, bbox }`. Provider-agnostic via the Vercel AI SDK. Document validation, typed failure handling, repair retries, streaming, and cost tracking built in. [Usage docs →](./packages/core/README.md)
-- **Eval harness** (next) — public benchmark on ~50 real documents (invoices/receipts): field accuracy per model, grounding accuracy, cost per 1k docs.
+- **Eval harness** (`packages/evals`, harness shipped) — public benchmark on ~50 pinned real documents (CORD-v2 receipts + DocILE invoices): field accuracy per model, grounding accuracy, cost per 1k docs. Fully reproducible — documents pinned by checksum, reports generated only from recorded runs. [Reproduce it →](./packages/evals/README.md)
 - **Playground** (planned) — drag-drop a document, watch fields extract; hover a JSON field to highlight its source region on the page.
 
 See [ROADMAP.md](./ROADMAP.md) for the build plan.
 
+## Benchmark
+
+<!-- benchmark:start -->
+
+*No results published yet — the first live eval run is pending. Numbers will appear here only from recorded runs; see [`packages/evals`](./packages/evals) to reproduce.*
+
+<!-- benchmark:end -->
+
 ## Scope
 
 General-purpose business documents: invoices, receipts, contracts.

ROADMAP.md

@@ -16,9 +16,11 @@ Tested against mock models only so far; first live-provider validation happens w
 
 ## Phase 2 — Evals (`packages/evals`)
 
-- [ ] Harness: accuracy per field per model, grounding accuracy (predicted vs. ground-truth bbox), cost per 1k docs
-- [ ] Benchmark page generated from real runs (engineering target: >90% field accuracy on the invoice set)
-- [ ] Benchmark table in README
+- [x] Harness: accuracy per field per model, grounding accuracy (predicted vs. ground-truth bbox), cost per 1k docs — tested against mock models; report generation from recorded runs
+- [x] Receipt half pinned: 25 CORD-v2 test docs curated by mapping-consistency checks, pinned by id + SHA-256 in `packages/evals/data/manifest.json`
+- [ ] Invoice half pinned: blocked on the DocILE token (Phase 0 human action); curation script is ready
+- [ ] First live eval run (needs a provider key — `ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, and/or `GOOGLE_GENERATIVE_AI_API_KEY`) → benchmark page generated from real runs (engineering target: >90% field accuracy on the invoice set)
+- [ ] Benchmark table in README (markers in place; filled by `pnpm report` from a recorded run)
 
 ## Phase 3 — Playground (`apps/playground`)
 

packages/core/README.md

@@ -10,7 +10,7 @@
 npm install extractkit ai zod
 ```
 
-`ai` (Vercel AI SDK v7) and `zod` (v4) are peer dependencies. Bring any AI SDK provider, e.g. `@ai-sdk/anthropic`.
+`ai` (Vercel AI SDK v7) and `zod` (v4) are peer dependencies. Bring any AI SDK provider — e.g. `@ai-sdk/anthropic`, `@ai-sdk/openai`, or `@ai-sdk/google` — and pass its model to `extract`.
 
 ## Quickstart
 

packages/evals/README.md

@@ -0,0 +1,43 @@
+# @extractkit/evals
+
+The extractkit benchmark harness: per-field value accuracy, grounding accuracy (predicted vs. annotated bounding box), and cost per 1k documents, measured on ~50 pinned public documents — 25 receipts from [CORD-v2](https://huggingface.co/datasets/naver-clova-ix/cord-v2) (NAVER CLOVA, CC BY 4.0) and 25 invoices from [DocILE](https://docile.rossum.ai/) (Rossum). Dataset rationale: [docs/benchmark-dataset.md](../../docs/benchmark-dataset.md).
+
+Documents are never vendored. [`data/manifest.json`](./data/manifest.json) pins each document by id + SHA-256, and the fetch script pulls them from their canonical hosts and verifies every checksum, so every published number is reproducible on exactly the same bytes.
+
+## Reproducing the benchmark
+
+```sh
+pnpm install && pnpm build          # from the repo root; evals imports core's build
+
+cd packages/evals
+pnpm fetch-data                     # CORD parquet (~230 MB) into .data/, checksum-verified
+                                    # set DOCILE_TOKEN to also fetch the DocILE invoice half
+export ANTHROPIC_API_KEY=...        # and/or OPENAI_API_KEY / GOOGLE_GENERATIVE_AI_API_KEY
+pnpm run-eval                       # extract every pinned doc with every model → results/run-<ts>.json
+pnpm report [results/run-....json]  # regenerate docs/benchmark.md + the README table
+```
+
+Without `DOCILE_TOKEN` the receipt half still runs; the invoice half needs a free token from [docile.rossum.ai](https://docile.rossum.ai/) (DocILE terms prohibit redistributing the documents, so every runner requests their own).
+
+### Choosing providers
+
+The lineup (`src/models.ts`) spans three providers — **Anthropic**, **OpenAI**, and **Google Gemini** — with three vision-capable tiers each and their public list pricing:
+
+| Provider | Key | Models |
+| --- | --- | --- |
+| Anthropic | `ANTHROPIC_API_KEY` | `claude-opus-4-8`, `claude-sonnet-5`, `claude-haiku-4-5` |
+| OpenAI | `OPENAI_API_KEY` | `gpt-5.6-sol`, `gpt-5.6-luna`, `gpt-5.4-mini` |
+| Google | `GOOGLE_GENERATIVE_AI_API_KEY` | `gemini-2.5-pro`, `gemini-2.5-flash`, `gemini-2.5-flash-lite` |
+
+By default the run includes every provider whose API key is set, so exporting only `OPENAI_API_KEY` benchmarks OpenAI alone — a cheaper way to iterate than the full Anthropic lineup. Pin the selection explicitly with `EVAL_PROVIDERS` (comma-separated, e.g. `EVAL_PROVIDERS=openai,google`); each named provider must have its key set.
+
+`pnpm pin` re-runs curation and rewrites the manifest — only needed when changing the selection, not to reproduce a run.
+
+## How scoring works
+
+- **Ground truth mapping** (`src/datasets/`): each dataset's native labels are mapped onto the demo Zod schemas in `src/schemas.ts` (receipt ← CORD `gt_parse` + `valid_line`; invoice ← DocILE KILE/LIR fieldtypes). Documents whose annotations can't be mapped confidently are rejected at pin time (`CordMappingError` / `DocileMappingError`) rather than silently mis-scored; the mapping code is in-repo so it can be audited.
+- **Value accuracy**: normalized comparison per field kind — whitespace/case for text, digits-and-sign for amounts ("24,000" ≡ "24.000" ≡ "24000"), lenient numeric parse for quantities ("2.00" ≡ "2"). A field the document doesn't carry counts as correct only when the model returns null. Line items align by printed order; extra predicted items are reported separately as hallucinations.
+- **Grounding**: among fields with a correct value and an annotated region, best IoU between the predicted bbox and any acceptable ground-truth region on that page; hit@0.5 is the headline number, missing bbox or wrong page scores 0.
+- **Cost**: measured token usage priced at the per-MTok list prices in `src/models.ts`.
+
+Raw per-field results for every run are serialized to `results/`, and reports are generated only from those records — no hand-entered numbers.

packages/evals/data/manifest.json

@@ -0,0 +1,142 @@
+{
+  "version": 1,
+  "cord": {
+    "dataset": "naver-clova-ix/cord-v2",
+    "revision": "7f0115a4b758a71d6473b8d085751692da2fef98",
+    "file": "data/test-00000-of-00001-9c204eb3f4e11791.parquet",
+    "sha256": "51c65f1788faff392abe2a0b55b023eb23e9be551c509138eaa3a832514224e7",
+    "split": "test",
+    "docs": [
+      {
+        "rowIndex": 0,
+        "imageId": 0,
+        "imageSha256": "8612d04b70f430f3aef07fbbd5200e382dcc4152b344cc2eff9f735f05a257c8"
+      },
+      {
+        "rowIndex": 1,
+        "imageId": 1,
+        "imageSha256": "5852cbe48df03646524e717e987eb3d0ac329a52fd192070b5fda1222a053789"
+      },
+      {
+        "rowIndex": 2,
+        "imageId": 2,
+        "imageSha256": "571ac6af26e75de8c06747a8c536c778f19a4562879090813571eddc74603d34"
+      },
+      {
+        "rowIndex": 3,
+        "imageId": 3,
+        "imageSha256": "8f73f0c4803ea8815040a50e3389d4f3fa15e8c28cb6d50ea9854f6f6c22bd63"
+      },
+      {
+        "rowIndex": 4,
+        "imageId": 4,
+        "imageSha256": "8f3eee7068c96e86cdb2e4b5c53085cb5e1439462edd55c373548cb1962801ad"
+      },
+      {
+        "rowIndex": 5,
+        "imageId": 5,
+        "imageSha256": "0b63b666b54ea615d1d05733ee2c6fc24f20e1a6c77b333872fa80fff28b2c48"
+      },
+      {
+        "rowIndex": 6,
+        "imageId": 6,
+        "imageSha256": "1c091eb7436b3e5a813eb6492c0350ef4eb14e3fdad78d42da35df669c757292"
+      },
+      {
+        "rowIndex": 7,
+        "imageId": 7,
+        "imageSha256": "810d3545f4105b19757c4b8b35934a477cde8cd1875302d9092315d2f960ef93"
+      },
+      {
+        "rowIndex": 8,
+        "imageId": 8,
+        "imageSha256": "58716693a6c0733358e66e64d7064a5a845424280bb55f9b5fa137b1af8c92fe"
+      },
+      {
+        "rowIndex": 9,
+        "imageId": 9,
+        "imageSha256": "0411c4d9beb9be65165fdb70c2b60f8bbf9035cc9661be2b8d4c73bbfcff7972"
+      },
+      {
+        "rowIndex": 10,
+        "imageId": 10,
+        "imageSha256": "2a5d18c199bdbb75010a54430429cf694b35bdca8bf560d1d0db1fbb090dedb8"
+      },
+      {
+        "rowIndex": 11,
+        "imageId": 11,
+        "imageSha256": "d92479ab212713e7a65e56aacc5e68c0713597476608cbc394d997971043010b"
+      },
+      {
+        "rowIndex": 12,
+        "imageId": 12,
+        "imageSha256": "b5f8a3a9a5431b8e38a8e5a16011d6a5a457f7ae4c4ef31da77e589c0b1ad882"
+      },
+      {
+        "rowIndex": 14,
+        "imageId": 14,
+        "imageSha256": "9dcf1f9f86c909b762767cb07d1cc0db30773d556154ce42e0210f65270ee39d"
+      },
+      {
+        "rowIndex": 15,
+        "imageId": 15,
+        "imageSha256": "f1cd67a7d67d04ba8d5b685553b1b5addbf2039d5954ab08c92139c0ad88a796"
+      },
+      {
+        "rowIndex": 16,
+        "imageId": 16,
+        "imageSha256": "eef326565b3945217a69afd84c893afdffd03a2f487943661c2423e6d6ec0d7d"
+      },
+      {
+        "rowIndex": 17,
+        "imageId": 17,
+        "imageSha256": "4322ab5ec4f0b1365e08af0c31288e65e8cbd2c8eb5241adfb329244acbbb30b"
+      },
+      {
+        "rowIndex": 18,
+        "imageId": 18,
+        "imageSha256": "88c35bbd2e2a634e89f2c5bfd30f7b530b1082c1f4e6e7cb3cb913f0b46c3868"
+      },
+      {
+        "rowIndex": 19,
+        "imageId": 19,
+        "imageSha256": "b076921ed83bdf900cab539f219cac8dddfaa4a99c13f7d66d7303d9f8c4c8c7"
+      },
+      {
+        "rowIndex": 20,
+        "imageId": 20,
+        "imageSha256": "6213439292eb4c2ad7e04afab6bb5bc7d1d5925d18dc392c36f233d394ca7c19"
+      },
+      {
+        "rowIndex": 29,
+        "imageId": 29,
+        "imageSha256": "1f065c268473d727282005e57441c55c3249afe0da54f3bc08823d0cbe730610"
+      },
+      {
+        "rowIndex": 30,
+        "imageId": 30,
+        "imageSha256": "52879816c28ebc3a7bc777c31685fb71f36bc4d1f7d7dc9c0b743fc5aced84de"
+      },
+      {
+        "rowIndex": 35,
+        "imageId": 35,
+        "imageSha256": "453a4ad5aadf63b2a7efdfa0915d45fd72c426254ab8f7c84d5621d56445794c"
+      },
+      {
+        "rowIndex": 38,
+        "imageId": 38,
+        "imageSha256": "b7afc72ae79e1c0b23b46c83ae89031bb08ce5aec5e532e7a6f74b9511dc7fda"
+      },
+      {
+        "rowIndex": 41,
+        "imageId": 41,
+        "imageSha256": "994b983a7f51670ca67b35bbaf9a11c8852abf1711651448d4ce39f8f8119d6f"
+      }
+    ]
+  },
+  "docile": {
+    "archive": "annotated-trainval",
+    "split": "val",
+    "docs": []
+  }
+}

packages/evals/package.json

@@ -0,0 +1,33 @@
+{
+  "name": "@extractkit/evals",
+  "private": true,
+  "version": "0.0.0",
+  "description": "Benchmark harness for extractkit: per-field value accuracy, grounding accuracy, and cost per 1k docs on pinned public invoices and receipts.",
+  "type": "module",
+  "engines": {
+    "node": ">=20.19"
+  },
+  "scripts": {
+    "fetch-data": "tsx scripts/fetch.ts",
+    "pin": "tsx scripts/pin.ts",
+    "run-eval": "tsx scripts/run.ts",
+    "report": "tsx scripts/report.ts",
+    "test": "vitest run",
+    "typecheck": "tsc --noEmit"
+  },
+  "dependencies": {
+    "@ai-sdk/anthropic": "^4.0.10",
+    "@ai-sdk/google": "^4.0.12",
+    "@ai-sdk/openai": "^4.0.11",
+    "ai": "^7.0.16",
+    "extractkit": "workspace:*",
+    "hyparquet": "^1.14.0",
+    "zod": "^4.4.3"
+  },
+  "devDependencies": {
+    "@types/node": "^26.1.0",
+    "tsx": "^4.19.2",
+    "typescript": "^6.0.3",
+    "vitest": "^4.1.10"
+  }
+}

packages/evals/scripts/fetch.ts

@@ -0,0 +1,82 @@
+// Fetches the pinned benchmark data from its canonical hosts into .data/.
+// CORD-v2 (ungated) always; DocILE only when DOCILE_TOKEN is set.
+import { spawn } from 'node:child_process';
+import { createWriteStream } from 'node:fs';
+import { access, mkdir, readFile, rm } from 'node:fs/promises';
+import { Readable } from 'node:stream';
+import { pipeline } from 'node:stream/promises';
+import { loadManifest, sha256Hex } from '../src/manifest.js';
+import { CORD_PARQUET_PATH, DATA_DIR, DOCILE_ROOT } from './paths.js';
+
+async function exists(path: string): Promise<boolean> {
+  try {
+    await access(path);
+    return true;
+  } catch {
+    return false;
+  }
+}
+
+async function download(url: string, dest: string): Promise<void> {
+  const res = await fetch(url);
+  if (!res.ok || res.body === null) throw new Error(`GET ${url} failed: ${res.status} ${res.statusText}`);
+  await pipeline(Readable.fromWeb(res.body as import('node:stream/web').ReadableStream), createWriteStream(dest));
+}
+
+async function fetchCord(): Promise<void> {
+  const { manifest } = await loadManifest();
+  const { dataset, revision, file, sha256 } = manifest.cord;
+  if (await exists(CORD_PARQUET_PATH)) {
+    const digest = sha256Hex(await readFile(CORD_PARQUET_PATH));
+    if (digest === sha256) {
+      console.log(`cord: ${file} already cached and verified`);
+      return;
+    }
+    console.log('cord: cached parquet has wrong checksum, re-downloading');
+    await rm(CORD_PARQUET_PATH);
+  }
+  const url = `https://huggingface.co/datasets/${dataset}/resolve/${revision}/${file}`;
+  console.log(`cord: downloading ${url}`);
+  await download(url, CORD_PARQUET_PATH);
+  const digest = sha256Hex(await readFile(CORD_PARQUET_PATH));
+  if (digest !== sha256) {
+    await rm(CORD_PARQUET_PATH);
+    throw new Error(`cord: downloaded parquet sha256 ${digest} does not match pinned ${sha256}`);
+  }
+  console.log('cord: downloaded and verified');
+}
+
+async function fetchDocile(): Promise<void> {
+  const { manifest } = await loadManifest();
+  if (await exists(`${DOCILE_ROOT}/annotations`)) {
+    console.log('docile: dataset already present');
+    return;
+  }
+  const token = process.env['DOCILE_TOKEN'];
+  if (token === undefined) {
+    console.log(
+      'docile: skipped — set DOCILE_TOKEN to fetch the invoice half.\n' +
+        '  Request a free token at https://docile.rossum.ai/ (the CORD receipt half works without it).',
+    );
+    return;
+  }
+  const archive = manifest.docile.archive;
+  const zipPath = `${DOCILE_ROOT}.zip`;
+  // Download URL shape from rossumai/docile download_dataset.sh; the token is
+  // a path segment, so never log the URL.
+  console.log(`docile: downloading ${archive}.zip (several GB, this takes a while)`);
+  await download(`https://docile-dataset-rossum.s3.eu-west-1.amazonaws.com/${token}/${archive}.zip`, zipPath);
+  await mkdir(DOCILE_ROOT, { recursive: true });
+  console.log('docile: extracting');
+  await new Promise<void>((resolve, reject) => {
+    const child = spawn('unzip', ['-quo', zipPath, '-d', DOCILE_ROOT], { stdio: 'inherit' });
+    child.on('error', reject);
+    child.on('exit', (code) => (code === 0 ? resolve() : reject(new Error(`unzip exited with ${code}`))));
+  });
+  await rm(zipPath);
+  console.log('docile: done');
+}
+
+await mkdir(DATA_DIR, { recursive: true });
+await fetchCord();
+await fetchDocile();

packages/evals/scripts/paths.ts

@@ -0,0 +1,9 @@
+import { fileURLToPath } from 'node:url';
+
+/** Local cache for fetched benchmark data; gitignored, never vendored. */
+export const DATA_DIR = fileURLToPath(new URL('../.data', import.meta.url));
+export const CORD_PARQUET_PATH = fileURLToPath(new URL('../.data/cord-test.parquet', import.meta.url));
+export const DOCILE_ROOT = fileURLToPath(new URL('../.data/docile', import.meta.url));
+export const RESULTS_DIR = fileURLToPath(new URL('../results', import.meta.url));
+export const BENCHMARK_PAGE_PATH = fileURLToPath(new URL('../../../docs/benchmark.md', import.meta.url));
+export const README_PATH = fileURLToPath(new URL('../../../README.md', import.meta.url));