feat(reports): scaffold confusion matrix infrastructure for PatentBench (Stage 1 of 6)

[rhahn28]

Status: DRAFT — Stage 1 of 6 on this branch

This PR lands the scaffold for publishing per-task-type confusion matrices alongside PatentBench benchmark runs. No live data is fetched, no matrices are published yet. Additional commits on this branch will push through Stages 2–6 before it goes out of draft.

Why confusion matrices

PatentBench currently publishes scalar accuracy per task type (e.g. action_classification: 92.7%). Scalar accuracy hides which class confusions drive the errors. For paralegal OA extraction, mis-classifying Final as Non-Final is a different kind of failure than mis-classifying Ex Parte Quayle as Non-Final — one silently costs an applicant appeal rights, the other is an annoyance. A confusion matrix makes this visible.

What landed in this commit

Module	Responsibility
patentbench/reports/confusion.py	Single-pass build_confusion_matrix() — cell values and cell traces are incremented in lockstep. Post-hoc trace construction is banned because it decouples the trace from the value (the class of bug that produces arithmetically-consistent but test-id-wrong artifacts).
patentbench/reports/ground_truth.py	load_ground_truth() refuses rows that lack peds_source provenance, and refuses rows tagged source: "llm" or source: "abigail". The system under test cannot label its own truth data.
patentbench/reports/peds_client.py	USPTO PEDS client with 1 req/s throttle, 3× retry on 429/5xx with backoff, strict dict-only JSON handling (no silent fallback).
patentbench/reports/build_confusion.py	CLI: run.json + truth.json → deterministic matrix.json + human-readable matrix.md.
patentbench/reports/verify_confusion.py	Independent verifier. Reconstructs the matrix from source and fails loud on any drift in cells, traces, SHA-256, labels, or totals.
tests/test_reports_confusion.py	15 invariant and negative-case tests.
.gitattributes	Pins LF on artifact files so cross-platform checkouts do not invalidate committed SHA-256 values.

Invariants enforced by code + CI

• sum(cells) + unparseable + quarantined == total. Asserted inline in build_confusion_matrix and re-checked independently in verify_confusion.
• Every non-zero cell has exactly cell_value test IDs in its cell_trace. Empty traces on non-zero cells fail CI.
• Labels are ASCII-sorted for byte-identical output across operating systems.
• hallucinated_labels lists any class the model predicted that never appears in ground truth.
• Source run file SHA-256 and ground-truth file SHA-256 are baked into every artifact. Verifier fails on drift.
• Parse order in raw_response: fenced \``json … ```, then first balanced-brace substring, then None. evalandexec` are forbidden.

Tamper test (verified locally)

$ python -m patentbench.reports.verify_confusion /tmp/out.json
VERIFIED: /tmp/out.json

# mutate matrix[0][0] 1 -> 99
$ python -m patentbench.reports.verify_confusion /tmp/out.json
Confusion matrix verification FAILED:
  - matrix values differ
exit=1

Test summary

• 52 pass (37 pre-existing + 15 new)
• ruff: clean
• mypy: clean

Why this is draft

Stages 2–6 still pending. Stage 4 onward touches live PEDS and live abigail.app, which produces data that lands in this repo — I am holding those steps until the project owner confirms the open questions below.

Open questions for the owner

1. PEDS endpoint = https://ped.uspto.gov/api/queries?
2. Use the existing data/real_oa/benchmark_cases.jsonl 82-case set as v1 universe, or a fresh pull?
3. Filter v1 to published applications only (PEDS-authoritative claims), or quarantine unpublished cases and document them?
4. Scope v1 paralegal_oa_extraction ground truth to fields PEDS provides cleanly (action_type, mail_date, examiner) and defer rejection-type truth to a separate PR where a USPTO-registered attorney has hand-labeled 20–30 cases?

Once these are answered, I'll push commits to this branch for Stages 2–6 and mark ready for review.

Stage plan

1. ✅ Scaffold + tests (this commit)
2. PEDS client integration tests (mocked)
3. Ground-truth fetcher script (one-shot) + seeded data/ground_truth/*.json
4. Fresh ABIGAIL eval on Paralegal subset (run via patentbench/harness.py against abigail.app, stored under data/benchmark_runs/abigail/<iso>.json)
5. Generate matrix artifacts under reports/confusion_matrices/ + update README.md
6. Mark ready for review

Non-goals for this branch

• No changes to pre-existing benchmark data (data/benchmark_results*.json).
• No changes to pre-existing adapters or scripts.
• No production DB migrations anywhere.
• No Admin-tier matrices in v1 (deferred — Admin is deterministic and must be 100%).
• No SME (human-attorney) matrices (deferred — SME annotation data does not yet exist).

[rhahn28]

Stage 2 landed: first real confusion matrix committed

Commit: daf4e42 — feat(reports): first real confusion matrix - action_classification

The pipeline now runs end-to-end on real data:

• Ground truth: data/ground_truth/action_classification.json — 82 rows, all sourced from USPTO PEDS prosecution_events, each carrying verifiable peds_source lineage (application number, retrieved_at, field path, SHA-256 of the events list). Zero LLM-produced labels.
• Benchmark run: data/benchmark_results_sonnet.json (ABIGAIL v3 Variant B, 2026-03-21).
• Artifact: reports/confusion_matrices/abigail/action_classification.md + .json.
• Verification: python -m patentbench.reports.verify_confusion reports/confusion_matrices/abigail/action_classification.json → VERIFIED.

Result

5 observed truth classes using the NF{0|1}-F{0|1}-A{0|1} projection:

truth \ predicted	NF0-F0-A0	NF1-F0-A0	NF1-F0-A1	NF1-F1-A0	NF1-F1-A1
NF0-F0-A0	1	0	0	0	0
NF1-F0-A0	0	30	0	0	0
NF1-F0-A1	0	0	7	0	0
NF1-F1-A0	0	0	0	33	0
NF1-F1-A1	0	0	0	0	11

Perfect diagonal on the 82-case sample. Zero unparseable, zero off-diagonal, zero hallucinated classes.

Scope note on the remaining stages

Stages 3-6 as originally planned targeted paralegal_oa_extraction and paralegal_clm_extraction. Those task types do not exist in the current benchmark dataset — the dataset covers admin-tier tasks (deadline_calculation, action_classification, fee_computation, timeline_analysis). To build matrices for paralegal tasks we need:

1. A PEDS pull that extracts claim structures (independent/dependent counts) and rejection-type metadata (which claims, which references, per rejection).
2. New benchmark cases generated against those extractions.
3. A fresh harness run against abigail.app emitting the structured output the paralegal extractors expect.

The scaffold already supports all of that; only the data does not exist yet. Splitting the remaining work into a follow-up issue rather than keeping this PR open indefinitely.

Tests / lint

• pytest tests/test_reports_confusion.py — 19 passed (15 prior + 4 new for action_classification).
• ruff — clean.
• mypy patentbench/reports/ — clean.