Testing and Benchmarks

Testing and Benchmarks

About 100 test files in tests/, a database-backed integration suite, a real-OCR suite, a real-local-model suite, an end-to-end smoke test run against the container stack on every pull request, and a labelled corpus with a benchmark harness for the architecture's central claim.

---

1. The commands

pnpm test        # unit, adversarial and security suites (+ OCR once the pinned model is cached)
pnpm test:db     # Postgres-backed integration suites (needs TEST_DATABASE_URL)
pnpm smoke       # every format, against a running instance
pnpm smoke --ai  # the model pass too, against an instance with a verified model
pnpm bench       # extraction and export timed over large documents
pnpm corpus:score · pnpm bench:models · pnpm bench:charts   # detection quality and cost (§6)

---

2. The suites that matter most

tests/security.test.ts: twenty-one recovery attempts

Takes each exported artifact and tries to get the sensitive value back out: PDF text extraction, raw byte scans with glyph spacing stripped, annotation objects, every OOXML part, hidden sheets and rows, cached formula results, the shared-string table, sampled pixels, EXIF. All twenty-one must fail.

tests/adversarial.test.ts: every format, three assertions

Through the real export path, for every exportable format:

1. the accepted value is gone, read in that format's own terms (PPTX notes/layout/master, EML headers/nested parts/filenames, RTF group splits and escapes);
2. a rejected value is still there, because a redactor that empties the file passes the first test and is useless;
3. the artifact still opens as what it claims to be.

A guard test keeps the format register honest: a new format cannot be added without an adversarial case.

tests/export.test.ts

End-to-end redaction per format, including that a suggestion never accepted is still present in the output (invariant 2 as an assertion).

Detection and the model layer

Suite	Checks
detectors.test.ts	an order number is not a card; a date is a birth date only when labelled; g-flag regexes reset lastIndex
analysis-orchestration.test.ts	analyzeDocument and image analysis driven by tests/helpers/scripted-provider.ts, a real LanguageModelV4 answering from a script: chunking, concurrency, dedupe, the locate-or-discard rule, and the "provider failed, keep going" path, with only the network removed (#30, since 1.6.1)
ai-degradation.test.ts, ai-spend.test.ts, service-limits.test.ts	degraded events, the spend cap, pacing and retry classification
providers.test.ts, ai-providers.test.ts, ai-cloud-providers.test.ts, ai-compatible.test.ts, ai-catalog.test.ts, ai-probe-errors.test.ts, ai-subscription.test.ts, ai-login-flow.test.ts, ai-status.test.ts, setup-ai.test.ts	every provider protocol against local HTTP fixtures: wire requests, structured output, images, auth, discovery, disabled choices, capability invalidation, per-model prices, error explanations, the ChatGPT sign-in flow. No live account.
ollama-contract.test.ts	a real Ollama + Qwen3-VL 4B contract, opt-in (ANONIFY_OLLAMA_TESTS=1) and run weekly in CI
vision.test.ts	the 0–1000 grid and region validation

OCR against the real engine (since 1.6.1, #33)

tests/image-ocr.test.ts runs real Tesseract over tests/fixtures/ocr/account.png and checks every box against ink boxes measured from the fixture's own pixels, then exports a redaction and reads the result back. OCR spans are what both the canvas and the exporter place boxes from, so a test that checked only the words would pass with every box in the wrong place. tests/pdf-ocr.test.ts does the same end to end for scanned PDFs. The model is pinned by URL and hash (tests/fixtures/ocr/model.json). The suites skip where it is not cached, and CI's OCR job caches it and requires it.

Streaming (1.6.0 / 1.7.0)

chunked.test.ts, sealed-storage.test.ts, normalized-pages.test.ts, zip-reader.test.ts, xlsx-stream.test.ts, ooxml-stream.test.ts, pdf-ranged.test.ts, eml-scan.test.ts (the scanner against the old parser over 3,000 generated and mutated messages, fed in pieces from 1 byte to 1 MiB), text-stream.test.ts, mbox-scan.test.ts, ranged-parts.test.ts, streamed-downloads.test.ts. The rule throughout: every streamed reader must produce exactly what its whole-file counterpart would. See Streaming IO §10.

Review features (1.9.0)

patterns.test.ts (RE2 compile rules and budgets), search.test.ts, search-offsets.test.tsx (the viewers' data-offset markers match page text), words.test.ts (wordAt token boundaries), shortcuts.test.ts (nothing fires while typing; nothing destructive on a bare key), hush-agent.test.ts, hush-learn.test.ts, hush-markdown.test.tsx (no fetchable links or images), session.test.ts (cookie renewal), store.test.ts / editor-state.test.ts (undo saves only what it changed).

Everything else, by area

Formats: pdf-extract, pdf-geometry (the GHSA fix), docx-extract, docx-redact, xlsx, pptx, delimited, text, rtf, eml, eml-markdown, eml-viewer, eml-attachments, mbox, mbox-rebuild, image, formats, geometry. Export: methods, export-options, report, value-matcher, batch-layout, batch, batch-concurrency. Platform: signed-url, rate-limit, rate-limit-config, quota-config, limits, retention, purge, cleanup-sweep, storage, failure, workflow-events, vercel-limits, public-url, byte-size, env-file, setup-finish, setup-tty, changelog. Benchmarks: corpus, score, render, bench.

---

3. tests/integration/: the database-backed suites

Properties of the row rather than of a function, where a fake would agree with whatever the code did:

Suite	Checks
cleanup.integration	storage is deleted before the row, and the row is kept when storage fails
quota.integration	a charge is atomic under concurrency and lands exactly once across a retried step
attachments.integration	attachment expansion, idempotence, substitution back into the message
mailbox.integration	mailbox expansion end to end
batch-download.integration	a mailbox uploaded, expanded, reviewed, exported, rebuilt and re-split: same count, byte-equal messages, no accepted value or source envelope sender anywhere; a left-out message named; a tampered message refused
rules.integration	rules at every scope, one-transaction writes, budgets
hush-tools.integration, hush-allowance.integration	Hush's tools against real rows; the per-visitor allowance checked between steps

export TEST_DATABASE_URL=postgresql://anonify:anonify@localhost:5432/anonify_test
DATABASE_URL="$TEST_DATABASE_URL" pnpm db:migrate:deploy
pnpm test:db

Never point it at DATABASE_URL, because these suites create and delete rows. Unset means they skip, and pnpm test still passes.

---

4. pnpm smoke: not a unit test

It drives a running instance over HTTP. For every supported format it uploads a synthetic document, waits for the pipeline, accepts every suggestion, exports, downloads, and opens the result the way an adversary would: re-parsing a CSV as a grid, re-parsing a message with an independent MIME library, unzipping a deck to read its notes and master, sampling an image's pixels. It also reads the delivered report bytes. --only=eml,pptx narrows it, and any URL works (pnpm smoke https://your-deployment).

pnpm smoke --ai adds the model pass: it asserts that calls were made and accounted for, that none failed, and that every suggestion quotes the document. It does not assert what a small model finds.

---

5. What CI runs

.github/workflows/ci.yml (a Decide job skips work already done):

Job	What it does
Lint · Typecheck · Build	in parallel
Test	pnpm test, plus "Corpus holds no real identifiers" (corpus:check)
OCR	restores or fetches the pinned model, runs the OCR suites
Migrations	applies every migration to an empty database, and fails if schema and migrations disagree
Database tests	pnpm test:db against a Postgres service
Compose stack	builds the images, re-runs bucket init (idempotence), runs smoke:storage through RustFS, then pnpm smoke for every format
Changelog	fails a version-moving PR without a changelog.d/ fragment

Also: local-model.yml (weekly, and on PRs touching the provider layer) runs the Ollama contract and a full Compose install with Qwen3-VL through pnpm ai verify and pnpm smoke --ai. release.yml cuts releases from labels. sweep.yml is the Hobby expiry backstop.

CI runs without a hosted AI key, so the suite passes on deterministic detection alone. That is deliberate: detection is an assist, and losing it must never cost the user the document.

---

6. Benchmarks: measuring the claim

The architecture makes a falsifiable claim:

Running deterministic detectors first, asking a model only the contextual question, and expanding repeated values by local search costs dramatically fewer tokens than sending the document to a model, at comparable or better quality.

It could be wrong. As of 1.9.x, the tooling to test it exists (#57, #59, #55, all closed; tracking #144). Full detail: benchmarks/README.md.

The corpus (pnpm corpus:*, since 1.9.1)

• Synthetic and labelled by construction. A cheap model writes each document from a seeded spec, and the labels come from how it was generated, not from someone reading it afterwards. The model is reached through the Claude Code or Codex CLI you are already signed in to (no API key read), in an empty temp directory.
• Distribution, exactly: 600 specs (150 dev, 450 test); 50/35/15% short/medium/long; PII density none/low/medium/high at 10/30/40/20%; locales en-US 45%, en-GB 25%, de-DE, fr-FR, es-ES, en-IN 7.5% each; twelve document types, 50 each; hard negatives; rare categories over-weighted.
• No real identifiers. Every email, phone, URL and IP comes from a range reserved for fiction or documentation, as does every card and ID wherever such a range exists. CI fails if a committed file holds a value outside those ranges. Documents mentioning the operator's own identity (Claude Code/Codex account, git name and email, OS user, --deny) are rejected.
• Cross-family validation (corpus:validate): a model from another family reviews the labels, and disagreements are queued for a person.
• Status: 534 of 600 documents committed (synthetic-v1.tar.gz), not yet human-reviewed.
• Rendering (corpus:render) to txt, eml, pdf (DejaVu Sans), docx, csv or xlsx, deterministically (the same bytes on any machine), so extraction can be measured through each format.

Scoring (pnpm corpus:score)

Each document goes through analyzeDocument, the same function an upload goes through.

• Recall (covered), the primary number: every character of the value is under a redaction. Partly covered is a leak, so it counts as missed.
• Strict: covered by redactions of the right category. Precision: the share of redactions touching a labelled value (touching a hard negative is a false positive).
• Weighted cost: a missed government ID, bank account, financial value or API key costs 10; a missed person, address, date of birth, email, phone or customer ID costs 5; other misses 2; a false positive 1.
• Agreement between runs: observed agreement, Cohen's kappa, character Jaccard.

The deterministic baseline (test split, patterns only): 20.1% covered recall at 94.1% precision. It finds no names (the model's job), and none of the SSNs, because plausibleSsn rejects the reserved 987 block the corpus uses. Through each format:

Format	Documents	Recall (covered)	Precision	Labelled text extracted
txt	363	19.0%	92.9%	100%
pdf	329	18.3%	92.5%	100%
docx	240	18.1%	92.0%	100%
eml	35	28.8%	96.8%	100%
csv	31 of 32	41.0%	99.1%	99.8%
xlsx	32 of 62	38.4%	97.0%	99.8%

Each row covers a different set of documents, so compare a format with txt over the same --ids, not down the table. The 30 XLSX failures are a known app bug in loadWorkbook (a small buffer's backing pool is sliced at the wrong offset, so small workbooks are intermittently misread as corrupt zips). See Troubleshooting.

Models (pnpm bench:models, since 1.9.2)

Three phases per model, each checkpointed and resumable:

Phase	Measures
deterministic-first	the pipeline as shipped: per-category quality, tokens and cost per document, time per stage, and what patterns, model and local search each contributed
model-only	the same model with the pattern pass switched off: the tokens and quality the patterns save, or do not
throughput	documents per minute at concurrency 1/2/4/8, and where the provider starts cutting the model pass short

Each model is verified before its first phase (since 1.9.3). Results go one file per model under benchmarks/results/models/…, so adding a model never changes another's results or chart colour. pnpm bench:charts draws the five charts from CONTRIBUTING §4 plus two for #55, in light and dark, colour-blind-validated, and rewrites the README sections. --check fails if they are stale. It costs real money: try --limit 20 first.

No model run has been published yet. Running Haiku 4.5 and Sonnet 5 and writing up what it found is #153, and it is the single most useful contribution available right now.

What the corpus does not cover

Faces; handwriting and scan noise (OCR has its own fixtures); non-Latin scripts; real-world category distribution (categories are deliberately over-represented, so precision here is not precision on your documents); and the generator's own blind spots (score with at least one model from another family).

---

7. How to write a test here

A change to redaction behaviour needs a test that reads the produced artifact, not one that asserts a function was called. Build a real file, redact it, then try to get the value back out. Several bugs were found only this way:

Bug	How it hid
sharp silently ignoring a second resize	pixelation kept full detail; the function was called, and the pixels were unchanged
undo snapshots aliasing an Immer draft	undo restored the state it was meant to replace
pdf.js detaching a buffer	the next reader got nothing

tests/fixtures.ts builds synthetic files of each format. Never commit a document containing somebody's actual personal data.

---

8. Remaining gaps

Gap	Issue
Browser end-to-end tests of the editor (canvas, inspector, export dialog)	#35
A published model benchmark	#153, #144
Human review of the corpus labels	#144
A screen-reader pass and a contrast audit	#42

---

See also

• Contributing · Security Model · Roadmap · Good First Issues · CLI Reference