Evaluation

Offline evaluation

OCR Foundry does not include an OCR recognizer. evaluate scores ID-matched UTF-8 JSONL reference and prediction files; benchmark freezes, verifies and compares offline suites. Run your recognizer outside this package and provide one prediction per reference ID, including an empty prediction for failed OCR.

ocr-foundry evaluate --reference reference.jsonl --predictions predictions.jsonl \
  --normalization NFC

Raw codepoint CER and exact-line accuracy are always reported; --normalization NFC adds separate normalized metrics. Whitespace WER is opt-in (--word-tokenization whitespace) and is not linguistic segmentation for Khmer or other unspaced scripts. See the evaluation reference for schemas, metrics and the heldout-data protocol.

Tracked frozen suite

From the repository root, use the existing lock for the 103-line, six-document real_doc_v1 release:

ocr-foundry benchmark verify --lock benchmarks/real_doc_v1/benchmark.lock.json
ocr-foundry benchmark compare --lock benchmarks/real_doc_v1/benchmark.lock.json \
  --baseline tesseract-5.5.4 --format markdown

The tracked predictions carry a Tesseract 5.5.4 baseline (raw CER 0.1671, exact-line accuracy 0.1748). One declared document is a clean digital vector render. The tracked source files are text descriptors and the .png crop paths hold placeholder text bytes, not images you can feed to OCR. verify checks the lock against those bytes, not image authenticity, inference or the manifest's real_document provenance claim. This result is a check of the tracked reference and prediction text, not independently rerunnable OCR on the tracked crops.

freeze refuses to overwrite the committed lock. Use an unused temporary path and remove benchmark.lock.json from the copy only:

cp -a benchmarks/real_doc_v1 /tmp/real_doc_v1
rm /tmp/real_doc_v1/benchmark.lock.json
ocr-foundry benchmark freeze --manifest /tmp/real_doc_v1/manifest.json
ocr-foundry benchmark verify --lock /tmp/real_doc_v1/benchmark.lock.json

To compare your own OCR runs, acquire rights-reviewed real crops, produce predictions externally and freeze a suite with the new assets and manifest. The benchmark schema and commands specify IDs, statuses, split isolation, fingerprints and baseline selection.

Separate Model Version 6 report

The multilingual roadmap reports a distinct 757-real-crop Model Version 6 continuation-training experiment. Its pretrained-model baseline was 52.21% CER on that cohort; calibrated synthetic continuation Arm C reported 50.02% CER at epoch 5. Its crops, predictions and checkpoints are not tracked here. Those numbers cannot be compared to the Tesseract baseline from the 103-line suite because the models and cohorts differ. For calibrated generation settings, see Configuration.