Phase 0 + Phase 1: benchmark dataset decision and core library

.github/workflows/ci.yml

@@ -0,0 +1,21 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+jobs:
+  check:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: pnpm/action-setup@v4
+      - uses: actions/setup-node@v4
+        with:
+          node-version: 22
+          cache: pnpm
+      - run: pnpm install --frozen-lockfile
+      - run: pnpm typecheck
+      - run: pnpm test
+      - run: pnpm build

CLAUDE.md

@@ -4,7 +4,7 @@ extractkit is an open-source TypeScript document-extraction engine: Zod schema +
 
 ## Current Phase
 
-**Pre-development — spec and planning only.** This repo intentionally contains no code yet. Do not scaffold packages, configs, or app code unless explicitly asked to start building. Until then, work happens in the md files (spec refinement, roadmap, API design sketches).
+**Phase 1 (core library) shipped; Phase 2 (evals) is next.** `packages/core` is implemented and tested against mock models — it has not been validated against live providers yet; that happens as Phase 2 stands up. `packages/evals` and `apps/playground` do not exist yet. Keep README/ROADMAP/docs in sync with what actually ships. Requesting the DocILE dataset token (see ROADMAP Phase 0) is a pending human action that gates the Phase 2 data pull.
 
 ## Planned Architecture
 

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 extractkit contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -2,17 +2,17 @@
 
 **Extraction you can audit.** Define a Zod schema, feed it a PDF or image, get back schema-validated JSON where every field carries provenance — the page and bounding box it came from — plus a confidence score.
 
-> **Status: pre-development.** This repo currently holds the project spec and roadmap only. No code yet.
+> **Status: in development.** The core library ([`packages/core`](./packages/core)) is implemented and tested against mock models; the eval benchmark and playground are next. See [ROADMAP.md](./ROADMAP.md).
 
 ## Why
 
 TypeScript has structured-output libraries (instructor-js, AI SDK `generateObject`) and document parsers (LiteParse), but nothing that does the full pipeline: **document in → grounded, validated, auditable JSON out** — with a public eval benchmark so the accuracy claims are numbers, not adjectives.
 
-## Planned v1
+## v1
 
-- **Core library** — Zod schema + PDF/image → validated JSON with per-field `{ value, confidence, page, bbox }`. Provider-agnostic via the Vercel AI SDK. OCR-failure handling, retries, streaming, cost tracking built in.
-- **Eval harness** — public benchmark on ~50 real documents (invoices/receipts): field accuracy per model, cost per 1k docs.
-- **Playground** — drag-drop a document, watch fields extract; hover a JSON field to highlight its source region on the page.
+- **Core library** (`packages/core`, shipped) — Zod schema + PDF/image → validated JSON with per-field `{ value, confidence, page, bbox }`. Provider-agnostic via the Vercel AI SDK. Document validation, typed failure handling, repair retries, streaming, and cost tracking built in. [Usage docs →](./packages/core/README.md)
+- **Eval harness** (next) — public benchmark on ~50 real documents (invoices/receipts): field accuracy per model, grounding accuracy, cost per 1k docs.
+- **Playground** (planned) — drag-drop a document, watch fields extract; hover a JSON field to highlight its source region on the page.
 
 See [ROADMAP.md](./ROADMAP.md) for the build plan.
 
@@ -22,4 +22,4 @@ General-purpose business documents: invoices, receipts, contracts.
 
 ## License
 
-MIT (to be added with first code).
+[MIT](./LICENSE)

ROADMAP.md

@@ -2,18 +2,21 @@
 
 ## Phase 0 — Pre-flight
 
-- [ ] Benchmark dataset chosen (~50 public invoices/receipts; candidates: SROIE, CORD, RVL-CDIP subsets)
+- [x] Benchmark dataset chosen: **25 DocILE invoices + 25 CORD-v2 receipts**, pinned by doc ID + checksum, fetched from canonical hosts (never vendored). Decision record: [docs/benchmark-dataset.md](./docs/benchmark-dataset.md)
+- [ ] Request DocILE access token at [docile.rossum.ai](https://docile.rossum.ai/) — human action; blocks the Phase 2 data pull (CORD half is ungated)
 
 ## Phase 1 — Core library (`packages/core`)
 
-- [ ] Zod schema → PDF/image → validated JSON with per-field `{ value, confidence, page, bbox }`
-- [ ] Provider-agnostic model layer (Vercel AI SDK)
-- [ ] OCR-failure handling, retries, streaming, cost tracking
-- [ ] Vitest suite covering the failure paths, not just happy path
+- [x] Zod schema → PDF/image → validated JSON with per-field `{ value, confidence, page, bbox }`
+- [x] Provider-agnostic model layer (Vercel AI SDK v7; models passed in, no provider coupling)
+- [x] OCR-failure handling, retries, streaming, cost tracking
+- [x] Vitest suite covering the failure paths, not just happy path (53 tests, mock models)
+
+Tested against mock models only so far; first live-provider validation happens when Phase 2 eval runs stand up.
 
 ## Phase 2 — Evals (`packages/evals`)
 
-- [ ] Harness: accuracy per field per model, cost per 1k docs
+- [ ] Harness: accuracy per field per model, grounding accuracy (predicted vs. ground-truth bbox), cost per 1k docs
 - [ ] Benchmark page generated from real runs (engineering target: >90% field accuracy on the invoice set)
 - [ ] Benchmark table in README
 

docs/benchmark-dataset.md

@@ -0,0 +1,61 @@
+# Benchmark dataset — Phase 0 decision record
+
+**Decision:** the v1 benchmark set is **~50 documents: 25 invoices from DocILE + 25 receipts from CORD-v2**, pinned by document ID + checksum and fetched from the canonical hosts at eval time. Documents are never vendored into the repo.
+
+Decided 2026-07-07. Feeds `packages/evals` (Phase 2). The exact pinned ID list is produced during Phase 2 curation and may shift a few documents while staying at ~50.
+
+## Requirements
+
+The benchmark exists to back the core claim — grounded, auditable extraction — with real numbers. That set four hard requirements:
+
+1. **Field-level ground truth** (values), so per-field accuracy is measurable.
+2. **Bounding-box ground truth per field**, so provenance can be scored (predicted bbox vs. annotated bbox), not just values. Without this, the auditability wedge is unmeasurable.
+3. **Domain match:** real invoices and receipts — the locked v1 scope.
+4. **Workable licensing:** running and publishing benchmark results must be clearly fine, and others must be able to reproduce the runs without us redistributing documents.
+
+## Candidates
+
+| Dataset | Domain | Released docs | Field ground truth | Field bboxes | License / access | Verdict |
+|---|---|---|---|---|---|---|
+| **DocILE** | Real business documents (invoices, orders, etc.) from public sources | 6,680 annotated (+100k synthetic, ~1M unlabeled) | 55 field classes + line items (KILE/LIR tasks) | Yes — bbox + page per field | Research-oriented terms; free token via access-request form; tooling is MIT | **Selected — invoice half** |
+| **CORD-v2** | Indonesian shop/restaurant receipts (photos) | 1,000 (800 train / 100 dev / 100 test) | 30 semantic classes in 5 superclasses, line grouping | Yes — quad boxes | CC BY 4.0, official NAVER repo on Hugging Face, ungated | **Selected — receipt half** |
+| SROIE (ICDAR 2019) | Malaysian scanned receipts | ~1,000 (626 train / 347 test) | 4 fields only (company, date, address, total) | No — KIE values are not linked to boxes (word boxes exist separately in Tasks 1/2) | No explicit dataset license; portal registration; mirrors of unclear standing | Rejected — can't score grounding, schema too shallow, murky license |
+| RVL-CDIP | 16-class grayscale document scans | 400k | None — one document-type label per image | No | Legacy IIT-CDIP terms | Rejected — classification dataset, no extraction ground truth |
+
+RVL-CDIP was listed as a roadmap candidate but is a document-*classification* dataset; it has no field annotations at all, so it cannot evaluate extraction.
+
+## Composition (~50 docs)
+
+- **25 invoices — DocILE annotated set.** Real business documents; curate for invoice-type docs with line items, mixing single- and multi-page layouts and diverse vendors. This is the hard case core targets (PDF, tables, multi-page).
+- **25 receipts — CORD-v2 test split** (100 docs; we pin 25). Photographed receipts stress OCR quality: skew, crumple, low contrast. Line items present.
+- The selection lives in-repo as **document IDs + content checksums** plus a fetch script: CORD-v2 from Hugging Face directly; DocILE via its downloader with a user-supplied token (e.g. `DOCILE_TOKEN`). Runners without a DocILE token can still run the receipt half.
+- `packages/evals` maps each dataset's native labels onto the extractkit demo Zod schemas (invoice, receipt): vendor/merchant, date, currency, totals, tax, line items `{ description, qty, unitPrice, amount }`. The mapping code ships in the harness, so the mapping itself is auditable.
+
+## Metrics this choice enables (Phase 2 sketch)
+
+- **Per-field value accuracy** per model (normalized comparison) — already on the roadmap.
+- **Grounding accuracy:** predicted bbox vs. ground-truth bbox overlap. This is the metric no value-only dataset (SROIE) could support, and the reason both selected datasets have field-level boxes.
+- **Cost per 1k docs** from tracked token usage — already on the roadmap.
+- Candidate, not committed: confidence calibration (reported confidence vs. observed correctness).
+
+## Licensing & redistribution posture
+
+- **CORD-v2** is CC BY 4.0 — redistribution with attribution would be allowed, but we still don't vendor images; the repo stores IDs, checksums, and mapping code only. Attribution to NAVER CLOVA goes on the benchmark page.
+- **DocILE** sits behind a free access-request token under research-oriented terms. We never redistribute its documents or annotations; each benchmark runner requests their own token. What we publish is per-field accuracy numbers — standard benchmark practice, and DocILE itself operates a public leaderboard.
+- If a host disappears, the pinned IDs + checksums still document exactly what every published number was computed on.
+
+## Caveats (state these on the benchmark page)
+
+- **Training contamination:** both datasets are public and widely cited; frontier models have likely seen them. The benchmark is a *comparative* measurement across models/pipelines under identical conditions, not an absolute capability claim.
+- **Locale skew:** CORD receipts are Indonesian (Latin script). Fine for v1 scope; noted so the numbers aren't read as universal.
+- **Manual step:** DocILE token acquisition (form at docile.rossum.ai) must happen before Phase 2 data work starts.
+
+## Sources
+
+- CORD repo + license: https://github.com/clovaai/cord
+- CORD-v2 hosting/splits: https://huggingface.co/datasets/naver-clova-ix/cord-v2
+- DocILE benchmark repo: https://github.com/rossumai/docile
+- DocILE dataset access + composition: https://docile.rossum.ai/
+- DocILE benchmark paper (55 classes, 6.7k annotated): https://arxiv.org/abs/2302.05658
+- SROIE challenge: https://rrc.cvc.uab.es/?ch=13 and https://arxiv.org/abs/2103.10213
+- RVL-CDIP: https://huggingface.co/datasets/aharley/rvl_cdip

package.json

@@ -0,0 +1,13 @@
+{
+  "name": "extractkit-monorepo",
+  "private": true,
+  "packageManager": "pnpm@10.17.0",
+  "engines": {
+    "node": ">=20.19"
+  },
+  "scripts": {
+    "build": "pnpm -r build",
+    "test": "pnpm -r test",
+    "typecheck": "pnpm -r typecheck"
+  }
+}

packages/core/README.md

@@ -0,0 +1,102 @@
+# extractkit
+
+**Extraction you can audit.** Define a Zod schema, feed it a PDF or image, get back schema-validated JSON where every field carries provenance — the page and bounding box it came from — plus a confidence score.
+
+> Pre-1.0. The API may change until the public benchmark ships. Part of the [extractkit monorepo](https://github.com/RATCHAW/extractkit).
+
+## Install
+
+```sh
+npm install extractkit ai zod
+```
+
+`ai` (Vercel AI SDK v7) and `zod` (v4) are peer dependencies. Bring any AI SDK provider, e.g. `@ai-sdk/anthropic`.
+
+## Quickstart
+
+```ts
+import { anthropic } from '@ai-sdk/anthropic';
+import { extract } from 'extractkit';
+import { readFile } from 'node:fs/promises';
+import { z } from 'zod';
+
+const invoice = z.object({
+  vendor: z.string().describe('Legal name of the issuing company'),
+  invoiceNumber: z.string(),
+  issueDate: z.iso.date(),
+  currency: z.string().describe('ISO 4217 code, e.g. USD'),
+  total: z.number(),
+  lineItems: z.array(
+    z.object({
+      description: z.string(),
+      quantity: z.number(),
+      unitPrice: z.number(),
+      amount: z.number(),
+    }),
+  ),
+});
+
+const result = await extract({
+  schema: invoice,
+  document: { data: await readFile('invoice.pdf') },
+  model: anthropic('claude-sonnet-5'),
+});
+
+result.data.total;           // 1250.5 — validated against the schema
+result.fields.total;         // { value: 1250.5, confidence: 0.97, page: 0, bbox: { x0: 0.72, y0: 0.81, x1: 0.9, y1: 0.84 } }
+result.usage;                // { inputTokens, outputTokens, totalTokens, modelCalls, costUSD }
+result.issues;               // non-fatal anomalies, e.g. dropped implausible provenance
+```
+
+Every leaf of your schema comes back twice: the plain value in `data` (validated, typed `z.output` of your schema) and an `ExtractedField` in `fields` with:
+
+- `page` — 0-based page index (always 0 for images), or `null` when the model couldn't locate the value.
+- `bbox` — `{ x0, y0, x1, y1 }` normalized to page width/height (0–1, origin top-left), or `null`. Reported regions are validated: out-of-range pages and degenerate boxes are dropped and noted in `result.issues`.
+- `confidence` — the model's self-reported probability (0–1) that the value is exactly right. It is **not calibrated**; treat it as a ranking signal. The eval harness in this repo measures how well it tracks reality per model.
+
+## Streaming
+
+```ts
+import { streamExtract } from 'extractkit';
+
+const stream = streamExtract({ schema: invoice, document, model });
+
+for await (const event of stream) {
+  // fires as each field's provenance wrapper completes
+  console.log(event.path, event.field.value);
+}
+
+const result = await stream.result; // same validated result extract() returns
+```
+
+Field events are emitted before final validation (the finished `result` is authoritative), and streaming does not run repair attempts — use `extract()` when robustness matters more than latency.
+
+## Documents
+
+`document.data` accepts `Uint8Array`, `ArrayBuffer`, or a base64 string. PDF, PNG, JPEG, and WebP are supported; the type is sniffed from the file signature, and a declared `mediaType` must match it. PDFs are checked before any tokens are spent: encrypted files, empty files, and unparseable files fail fast with typed errors, and the real page count is used to validate the model's page references.
+
+## Schemas
+
+Supported: `z.object`, `z.array`, `z.string` (including formats like `z.iso.date()`), `z.number`, `z.boolean`, `z.enum`, `z.literal`, plus `.optional()`, `.nullable()`, `.describe()`, and `.refine()`. Field descriptions are forwarded to the model — use them.
+
+Rejected by design, with an `UnsupportedSchemaError` naming the offending path: `.transform()`, `.default()`, `.catch()`, and `z.date()`. A transformed or defaulted value no longer maps to a region of the document, which breaks the provenance guarantee. Transform after extraction instead.
+
+## Failure handling
+
+| Error | `code` | Meaning |
+| --- | --- | --- |
+| `UnsupportedSchemaError` | `SCHEMA_UNSUPPORTED` | Schema contains an unsupported type; thrown before any model call |
+| `DocumentError` | `UNSUPPORTED_MEDIA_TYPE`, `MEDIA_TYPE_MISMATCH`, `ENCRYPTED_DOCUMENT`, `INVALID_DOCUMENT` | Input rejected before any model call |
+| `DocumentUnreadableError` | `DOCUMENT_UNREADABLE` | The model reported the document as blank/illegible/not a document; carries its stated `issues` |
+| `MissingRequiredFieldsError` | `MISSING_REQUIRED_FIELDS` | Document was readable but required fields weren't found; carries `missingPaths` and the `partial` extraction |
+| `ExtractionFailedError` | `EXTRACTION_FAILED` | Model kept returning invalid output after all repair attempts; carries `attempts`, `usage`, and the raw text |
+
+All extend `ExtractKitError`. Transport and provider errors from the AI SDK propagate unwrapped, with the SDK's own `maxRetries` applied. When the model returns malformed or schema-violating output, extractkit re-prompts it with the specific violations (`maxRepairAttempts`, default 1); token usage is accumulated across attempts so costs stay visible.
+
+## Cost tracking
+
+`result.usage` always carries token counts and the number of model calls. Pass `pricing: { inputPerMTokUSD, outputPerMTokUSD }` to get `costUSD`; without it, `costUSD` is `null` — extractkit ships no built-in price table, so costs are never silently wrong.
+
+## License
+
+MIT

packages/core/package.json

@@ -0,0 +1,55 @@
+{
+  "name": "extractkit",
+  "version": "0.1.0",
+  "description": "Schema-first document extraction with per-field provenance: Zod schema + PDF/image in, validated JSON with page, bbox, and confidence out.",
+  "license": "MIT",
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/RATCHAW/extractkit.git",
+    "directory": "packages/core"
+  },
+  "keywords": [
+    "document-extraction",
+    "ocr",
+    "zod",
+    "structured-output",
+    "pdf",
+    "invoice",
+    "receipt",
+    "ai-sdk"
+  ],
+  "type": "module",
+  "sideEffects": false,
+  "engines": {
+    "node": ">=20.19"
+  },
+  "exports": {
+    ".": {
+      "types": "./dist/index.d.mts",
+      "default": "./dist/index.mjs"
+    }
+  },
+  "files": [
+    "dist"
+  ],
+  "scripts": {
+    "build": "tsdown",
+    "test": "vitest run",
+    "typecheck": "tsc --noEmit"
+  },
+  "dependencies": {
+    "pdf-lib": "^1.17.1"
+  },
+  "peerDependencies": {
+    "ai": "^7.0.0",
+    "zod": "^4.0.0"
+  },
+  "devDependencies": {
+    "@types/node": "^26.1.0",
+    "ai": "^7.0.16",
+    "tsdown": "^0.22.3",
+    "typescript": "^6.0.3",
+    "vitest": "^4.1.10",
+    "zod": "^4.4.3"
+  }
+}