Proposal: a workbench for reviewing disagreements between OCR backends

[jabrailkhalil]

I would like to build a companion workbench for comparing two OCRmyPDF plugin backends and reviewing the text they disagree on. I would use the existing plugin interfaces and keep the first output as a reviewed text sidecar.

I would start with ten pages, align the two outputs where possible, and show a queue of disagreements with their page context. A reviewer could accept one version or enter a correction, and the export would preserve the decision and source references.

I would handle mismatched word boundaries and uncertain alignment explicitly. I would not promise safe reconstruction of every PDF from edited OCR text. Would this fit as an ecosystem tool, and which plugin/output interfaces would you recommend for a first implementation?

I previously proposed Qwen Code's Android work, adjusted the design with its maintainers, and contributed the merged initial shell/PWA. I would use the same small-first approach here.

[drb995]

Two public data points that bear on the design, since "review the disagreements" is the part I would want evidence for before building it. Yesterday the maintainer of a journalists' document tool ran RapidOCR and Tesseract 5 on the same 17-page scanned court order and found they fail in opposite ways: RapidOCR turned three whole clauses into nonsense ("eeo o o n o e y ons") that Tesseract read cleanly, while Tesseract corrupted the identifiers RapidOCR got right ("LSO# 69993I" became "699931", the postcode "M5K 1K7" became "MSK IK/"), and their own conclusion was to re-OCR the garbled lines with the second engine or run both and choose per line, rather than swap one default for another: tomcardoso/watchdog#809. On what the review queue should be built from, a consultancy's public spike on 185 paper-filed IRS 990-EZ scans (4,070 fields scored against the IRS's own extract) compared two queues of the same size, 78 fields: the single reader's 78 lowest-confidence fields caught 0 of its 27 errors, and the 78 fields where a second reader disagreed caught 18 of them (https://github.com/fulton-ring/agent-content-demos/tree/main/scanned-990ez-extraction). Both are single runs on small sets, so a direction rather than a measurement, but they point the same way: the disagreement set is small, it lands on the lines that matter, and the plausible-but-wrong read is the one a confidence score does not surface. One thing to plan for from the same 990-EZ spike: the pair still agreed on 9 wrong values, so the queue needs a "both wrong, here is the correction" outcome beside "accept A" and "accept B".

[jabrailkhalil]

I'll include an explicit "both wrong; enter a correction" outcome and retain both raw readings, the correction, and page/region provenance. I'd evaluate queue recall and missed errors against independently transcribed ground truth, including cases where both engines agree. The disagreement queue is a prioritization signal, so those shared errors need a separate check.

[jbarlow83]

This is really a separate application. There's a little streamlit script to compare two different runs of ocrmypdf if that's of use.

Also speaking from experience here, you'll really want to take an OcrElement tree and convert it to some other format more suitable for comparison. You'll need to compare the data to the data, comparing text in similar on page positions, rather than the sidecar which has mostly discarded positional data. A pure text diff won't do.

[jabrailkhalil]

Thanks, I found misc/ocrmypdf_compare.py and the OcrElement model. I’ll keep PDF comparison in a separate application, preserving page geometry and both readings.

My ClicknTranslate work covers screen OCR and translation. For PDF review, I’ll start with split/merged words, displaced text and a “both wrong, enter a correction” outcome. I’ll compare missed errors against independently transcribed ground truth, including engine-agreement cases.

[jabrailkhalil]

I have a runnable positional comparison lab with inspectable fixtures. I use OCRmyPDF 17.13.0's public OcrElement models, preserving word boxes, line/tree paths and both raw readings. Split/merged and ambiguous groups stay reviewable instead of being forced into a text-only pairing.

The four-region reference includes A=B0, B=B0, gold=80. That shared error stays outside the automatic disagreement queue, so error-region recall is 2/3, not 2/2. I can record an explicit 80 correction from the agreement/audit view; it retains both original readings and does not retroactively count the error as detected.

This first slice uses synthetic observations and literal independent references. I have not run live OCR or a real-PDF benchmark, and the frame mapping is caller-supplied.

I ran 24 tests with 118 subtests and nine CLI cases on each of Linux Python 3.11/3.14 and Windows Python 3.12. Four deliberately broken variants fail their intended assertions. The exact results also verify the installed upstream model's source hash.