Documents and OCR

Documents and OCR

The doc-extract service turns non-HTML bytes into text, markdown, tables and records. It ships in the bundle and docker compose up starts it. Without it a crawl still runs — it just skips every PDF, office file and scanned page it reaches, silently.

What it reads

Input	How	OCR?
PDF with a text layer	pdfplumber → markdown + tables; pypdf reads a tagged PDF's declared headings	no
PDF scanned page	PDFium raster → RapidOCR, that page only	yes
.docx / .xlsx / .pptx	python-docx / openpyxl / python-pptx	no
JSON / CSV	stdlib → records	no
Image or screenshot	RapidOCR	yes
HTML served from a .pdf URL	trafilatura → readability → markdownify	no

Markdown comes in two tiers. Roughly four documents in five are tagged and carry their own logical structure — those are read exactly. The rest fall back to inferring headings from typography.

OCR is the last resort. Any recoverable text layer is read directly, which is both exact and free. OCR is RapidOCR (ONNX PP-OCRv5): CPU-only, no torch, and the model weights ship inside the wheel, so it works fully offline and air-gapped.

It makes no outbound calls

The service never fetches a URL — callers POST bytes they already hold, so it has no SSRF surface. It carries no analytics or error-reporting client, and document bytes are processed in memory, not written to disk.

How it is wired

Your agents call it directly with bytes they fetched. The coordinator never touches it — it only hands each agent the address and secret when the agent connects, which is why a co-located agent needs no configuration at all.

agent ──fetched bytes──► doc-extract ──► {markdown, text, tables, records, ocr}

Agents on other machines

DOC_EXTRACT_URL defaults to http://127.0.0.1:8092, which a remote agent cannot reach. The failure is silent: the agent skips non-HTML content exactly as if the service did not exist.

If any agent runs elsewhere:

1. Give the service a route those agents can reach — usually a second server block on the same reverse proxy, with TLS terminated there.
2. Set DOC_EXTRACT_URL in .env to that address. Every connect command generated from then on carries it.
3. Keep DOC_EXTRACT_SECRET — it is the only thing in front of the service once it is reachable over a network. Never publish port 8092 unprotected.

Check what agents are being handed:

curl -s https://writ.example.com/api/fleet/connect-info \
  -H "Authorization: Bearer <token>" | grep doc_extract

Settings

Variable	Default	What it does
DOC_EXTRACT_SECRET	—	Shared secret. Required in production.
DOC_EXTRACT_URL	http://127.0.0.1:8092	Address as agents see it. Blank disables the lane.
DOC_EXTRACT_OCR_MODE	auto	auto | off | force.
DOC_EXTRACT_OCR_DPI	200	Raster DPI before OCR. Higher reads better, costs more.
DOC_EXTRACT_OCR_MAX_PAGES	100	Most pages one request may OCR.
DOC_EXTRACT_MAX_BYTES	33554432 (32 MiB)	Hard cap on one request body.
DOC_EXTRACT_TIMEOUT_S	60	Wall-clock budget for one extraction.
DOC_EXTRACT_TIER	light	light ships everywhere. rich also tries docling for complex tables and pulls torch — install it yourself.

The last three bound the real risk here, which is not SSRF but resource exhaustion from hostile input: a 500 MB "document" or a 5,000-page scan.

Running without it

docker compose up -d coordinator     # start the coordinator alone

and set DOC_EXTRACT_URL= (empty). Crawls then skip non-HTML content — a no-op, never an error.

Licensing

Every dependency is permissive: PDFium via pypdfium2 (BSD-3-Clause / Apache-2.0), pdfplumber over pdfminer.six (MIT), pypdf (BSD-3-Clause), RapidOCR (Apache-2.0). PyMuPDF was deliberately removed — it is AGPL-3.0, and §13 obliges a network service to offer the corresponding source of the combined work.