Repository navigation
OCR experimental
DiffLens can pull text out of scanned PDFs — pages with no text layer — through an optional OCR path. It is labeled experimental for a reason: quality is honestly mediocre, and this page explains exactly when it runs, what you need to install, and what to expect.
The OCR path engages only when all of the following hold:
- The document is a PDF (other formats never OCR).
- More than 30% of its pages have an empty text layer — the same threshold as the "scanned PDF suspected" extraction warning.
- The OCR preference is not switched off in Settings (default is auto).
- A user-installed Tesseract CLI with the Vietnamese traineddata (
vie) is detected on the machine.
If any condition fails, extraction simply proceeds without OCR — it never fails, never blocks, and at most adds one honest warning.
DiffLens detects, but never bundles Tesseract (Apache-2.0 since v4). Install it yourself:
# Debian / Ubuntu
sudo apt-get install tesseract-ocr tesseract-ocr-vie
# macOS (Homebrew) — vie traineddata must be added manually
brew install tesseract
brew install tesseract-lang # includes vie
# Windows — run the UB-Mannheim installer, tick "Vietnamese"
# https://github.com/UB-Mannheim/tesseract/wikiDetection checks PATH first (PATH order = preference), then known install locations per OS (C:\Program Files\Tesseract-OCR, /opt/homebrew/bin, /usr/local/bin, /usr/bin). A binary is considered usable only if tesseract --list-langs reports vie; a found-but-broken install degrades honestly to "no languages". The Settings screen and the Downloads screen both show an OCR availability card with the resolved binary path and language list — the same snapshot the ocr_status command returns.
- Textless pages are rasterized at 300 DPI through the pinned PDFium loader.
- Each page is OCR'd in a separate Tesseract invocation with the
vie+englanguage pair, under a hard 90-second per-page timeout (processes are killed and reaped on timeout). - At most 50 pages per job; beyond that the job stops with a warning naming the cap.
- Results are parsed from Tesseract's TSV output and rebuilt into lines by word geometry, then merged into the document as ordinary blocks — diff, alignment and heuristics work on them unchanged.
Expect mistakes. Tesseract's Vietnamese recognition on real-world scans is weak by modern standards — diacritic errors (ơ/ớ/ờ, đ/d confusion) are common, and quality degrades further on low-resolution or skewed scans. Community benchmarks put Tesseract's character error rate on Vietnamese text around ~18%, versus roughly ~4–5% for modern detector+recognizer pipelines (PP-OCRv5-class ONNX models). Those numbers are order-of-magnitude approximations, but the gap is the explicit motivation for the planned v2 improvement: a native ONNX-based OCR engine (PaddleOCR-class det+cls+rec via the same ort runtime DiffLens already uses for embeddings), which would remove the external Tesseract dependency and dramatically improve Vietnamese accuracy.
Because of this, OCR output is honestly labeled everywhere:
- Every OCR-derived block carries
anchor: "ocr"provenance, visible in the audit trail — you see exactly which blocks came from a machine's best guess. - The document gains warnings like "N pages … extracted via OCR (experimental)" plus the mean confidence across OCR'd pages.
- Low-confidence results are flagged rather than silently presented as fact.
Settings → OCR preference switch → off. This maps to the run_diff command's ocr_enabled flag: false disables the path entirely — no detection, no OCR warnings, no subprocess. The default (auto) means "only when a scanned PDF is detected and Tesseract + vie are available", so on a machine without Tesseract nothing ever happens.
CI exercises the OCR path end-to-end on every push: the core job installs tesseract-ocr + tesseract-ocr-vie on the runner, asserts vie is listed, and runs the OCR integration tests (ocr_tests, marked #[ignore] locally precisely because they need the real binary). The test log must contain the ocr-ci-ok marker — CI fails if the tests silently skip.