ocrtool

Turns Google Cloud Document AI OCR output into either a searchable "sandwich" PDF (default) or a plain-text e-text (--mode text, see Text mode).

A sandwich PDF keeps the original scanned page image intact and adds an invisible text layer on top, so the file looks the same but text can be selected, copied, and searched in any PDF viewer.

How it works

1. Google Document AI OCR runs on a scanned PDF and produces JSON output containing the extracted text and per-token bounding boxes (normalizedVertices in 0–1 page coordinates).
2. ocrtool reads the source PDF and the JSON shards, converts each token's bounding box from normalized image coordinates (top-left origin, y-down) to PDF points (bottom-left origin, y-up), and appends an invisible text content stream (3 Tr render mode) to each page.
3. The output PDF is visually identical to the input but fully searchable.

Usage

ocrtool <shard.json>... --pdf <PDF> --output <PDF> [--page <N>] [--web-optimize] [--dpi <N>] [--jpeg-quality <N>]
ocrtool <shard.json>... --mode text --output <TXT> [--page <N>] [--keep-headers] [--page-markers]

JSON shard files are given as plain positional arguments — a shell glob like *.json works — in any order. Each must be named <name>-<N>.json; the tool sorts them by that embedded index and aborts before doing anything if the set is inconsistent (see Shard ordering and validation).

Flag	Description	Mode
--mode	overlay (default, sandwich PDF) or text (plain-text e-text, see Text mode)	both
--pdf	Source scanned PDF	overlay (required)
--output	Output path — a PDF in overlay mode, a .txt file in text mode	both
--page	Process only this page number (1-indexed spread); omit to process all pages	both
--web-optimize	Downscale page images and re-encode as JPEG for web delivery (default: off, images untouched, see below)	overlay
--dpi	Target DPI for --web-optimize (default 150)	overlay
--jpeg-quality	JPEG quality 1-100 for --web-optimize (default 80)	overlay
--keep-headers	Don't strip detected running headers/page numbers (default: strip)	text
--page-markers	Insert a [Page N (L/R)] line at each physical-page boundary (default: off)	text

Web optimize mode

Source scans are typically print-resolution (300-600 DPI) and can run hundreds of megabytes to over a gigabyte — far more than needed once OCR is done and the goal is reading on screen rather than commercial printing. --web-optimize downscales every page image to --dpi (default 150) and re-encodes it as JPEG at --jpeg-quality (default 80). It's off by default — without it, page images are passed through completely untouched, whatever format they're already in.

Only two source formats are decoded: 1-bit CCITT Group 4 (CCITTFaxDecode, K < 0 — what scanners and Document AI produce) and images already stored as JPEG (DCTDecode, which still get downscaled if oversized — useful for scan sets that mix bitonal text pages with color plates/covers). Anything else (JBIG2, Group 3, raw/Flate-packed samples, CMYK, indexed color, ...) is left untouched with a warning, as is any image already at or below the target DPI. Image processing runs on a thread pool sized to the machine, since each image is independent of the others — see CLAUDE.md for details.

Single page (proof of concept)

ocrtool shard-0.json --pdf scan.pdf --output out.pdf --page 1

All pages, multiple shards

ocrtool shard-*.json --pdf scan.pdf --output out.pdf

Shards can be listed in any order — the tool sorts them by the index embedded in the filename itself.

Shard ordering and validation

Every shard filename must look like <name>-<N>.json (a shared prefix plus its 0-indexed shard number, e.g. scan-0.json … scan-62.json) so the tool can order and account for them without relying on the order they were listed on the command line. Before touching the PDF or writing any output, it verifies:

• every filename matches that pattern and all share the same <name> prefix (catches shards from two different documents being mixed together)
• no shard index is used by more than one file (doubled shards)
• the indices form one contiguous run from 0 (no missing shards)

and, as each file is actually parsed, that its own shardInfo.shardIndex / shardInfo.shardCount agree with what the filename implies. Any mismatch aborts the run with an error describing exactly what's wrong, before any output file is written.

Text mode

--mode text skips the PDF entirely and converts the JSON shards straight into a single plain-text e-text file, in the plain-text convention used by Projekti Lönnrot (the Finnish-language sibling of Project Gutenberg): UTF-8, CRLF line endings, original OCR line breaks and hyphenation kept exactly as scanned (no reflow/dehyphenation). Paragraphs are joined by a blank line only where the source actually has one — measured from each paragraph's vertical position, not assumed — so a tightly-set character list or a densely-typeset page of dialogue doesn't get blank lines inserted that aren't in the original. Where a paragraph's first line is indented in the source (again measured, not assumed), the output keeps that indent — which is what makes a page of unspaced dialogue still readable as separate speaker turns. See Usage for the flags.

Two-page spreads

Many scans photograph a book as two-page spreads (one landscape image per physical left+right page pair) — detected from each page's aspect ratio. Text mode splits each spread into two physical pages and emits left before right, using a dynamically-detected gutter position (not a fixed midpoint) so it keeps working on skewed or off-center scans. See CLAUDE.md for the algorithm.

Diagnostics

A summary is printed to stderr as the file is written: stripped running headers/page numbers, detected page-skew, uncertain left/right splits, and any physical page with no extracted content. The process exits non-zero if two or more consecutive physical pages come up empty — a strong signal something went wrong (a missing shard, a misdetected spread, etc.) rather than a legitimately blank page (e.g. the blank verso of a title page, which is common and only produces a warning).

Document AI JSON format

The tool expects the JSON produced by Document AI's Document OCR processor. Key fields used:

Field	Description
text	Full extracted text for the shard
shardInfo.textOffset	Character offset of this shard's text within the full document
pages[].pageNumber	1-indexed absolute page number (matched against PDF pages)
pages[].tokens[].layout.boundingPoly.normalizedVertices	Four vertices, coordinates in [0, 1]
pages[].tokens[].layout.textAnchor.textSegments[].startIndex/endIndex	Unicode codepoint offsets into the shard's text
pages[].tokens[].layout.orientation	PAGE_UP tokens are overlaid; rotated tokens are currently skipped
pages[].paragraphs[].layout	Text mode only: same shape as a token's layout, one per paragraph
pages[].dimension.width/height	Text mode only: aspect ratio used to detect two-page spreads

startIndex is omitted when 0. All integer fields that arrive as JSON strings (int64) are handled automatically.

Building

Requires Rust with Cargo. The crate pins lopdf = "=0.29.0" and image = "=0.24.9" to stay compatible with Cargo < 1.76.

cargo build --release

The binary is written to target/release/ocrtool.

Performance

On a 625-page, 63-shard document (~700 MB of JSON input):

• ~1.5 seconds end-to-end (release build)
• ~221 000 tokens overlaid

Each shard is parsed and dropped before the next is read, so peak heap usage is roughly one shard (~11 MB) plus the PDF in memory.

--web-optimize on that same 625-page document (212 MB, print-resolution scans) takes about 15 seconds on a 20-core machine — parallelized across a thread pool, since decode/resize/JPEG-encode are pure-Rust codecs with no SIMD and slow enough per image to matter serially. Output was 98 MB.

Known limitations

• WinAnsiEncoding (Helvetica): covers Latin-1 plus the Windows-1252 extended range (€, †, ‡, –, —, •, ", ", …, ™, and others). Characters outside WinAnsiEncoding are replaced with ? in the PDF overlay — one common source, an OCR script mixup on a stylized all-caps word (Greek letters that are visual homoglyphs of Latin ones, e.g. Κ/K, Ο/O), is corrected before this happens rather than turning into ?; genuinely non-Latin text (Cyrillic, CJK, etc.) still is.
• --web-optimize only decodes CCITT Group 4 (K < 0) and already-JPEG (DCTDecode) page images. JBIG2, Group 3, raw/Flate-packed samples, CMYK, and indexed color are left as-is (with a warning) rather than converted. DPI is computed assuming each image fills its whole page — true of every source PDF this tool targets — not from the content stream's actual placement matrix.
• Text mode header/page-number stripping is heuristic (recurring-text frequency for running headers, sequence-matching for page numbers). It's been validated against a real scanned book but can misfire on layouts very different from that sample; --keep-headers disables it.
• Shard filenames must end in -<N>.json with a consistent prefix; the tool orders and validates them itself (see Shard ordering and validation) regardless of command-line order.

License

MIT