💡 Vision-Native Document Intake: Fix Journey 3's Scanned-PDF Failure

[github-actions[bot]]

Summary

Replace TalkTerm's planned text-only PDF pipeline with Claude's native multimodal PDF ingestion so that scanned, image-based, and handwritten documents work on the first try instead of surfacing an error. Journey 3 in the PRD literally describes an image-based PDF failing and the avatar asking the user to provide a text version — a scenario that, as of early 2026, is no longer necessary. The Claude API already handles scanned PDFs end-to-end via its multimodal pipeline.

Market Signal

Claude's PDF support now processes every page as both an image and a text stream, merging results for maximum fidelity. The public Claude API docs list PDF as a first-class input type; 2026 developer writeups report reliable OCR on printed, scanned, and handwritten content including layout-aware table extraction. Comparison pieces in 2026 position Claude against GPT-4V and Gemini on invoice and receipt extraction and find Claude competitive or leading on layout understanding. Meanwhile, document-AI startups like Graphlit are pitching "document infrastructure" as a layer above raw LLM calls — evidence that the primitive capability is now commodity. The industry has moved past OCR-as-a-separate-step; TalkTerm's PRD has not.

User Signal

This is the only idea in this batch that comes directly from the PRD's own text. Journey 3, in full:

Mary encounters an issue — one PDF is image-based and can't be parsed. Instead of a stack trace or cryptic error, Mary says: "I was able to read two of the three documents, but Competitor_C_pricing.pdf is a scanned image I can't read directly. Would you like me to proceed with the two I have, or can you provide a text version of the third?"

Journey 3 is framed as a success story for graceful error handling. But the error is only necessary because the planned implementation treats PDF ingestion as text extraction. With vision-native ingestion, Mary reads the third PDF, finishes the comparison, and Journey 3 becomes a success story for not failing at all. FR16 requires "input documents (PDF, TXT, DOCX files) to the agent for use in workflow processing" — it does not mandate a text-only pipeline. The PRD can be strengthened simply by choosing the right adapter on day one.

Technical Opportunity

The planned architecture has no implementation yet (the repo is pre-implementation), which makes this the cheapest possible moment to pick the right approach. The AgentBackend port and claude-sdk-backend.ts anti-corruption layer are where document handling will live. Sending PDFs as multimodal input to the Claude API is a few lines of SDK code — the work is (a) plumbing file drag-and-drop and file dialog intake (FR16 already specifies this UI), (b) chunking large PDFs to fit context windows, and (c) surfacing per-page extraction confidence in the audit trail (FR27-29) for user review. Image-based and handwritten content is handled for free by the multimodal model. The "explicit opt-in for image PDFs" guard recommended by Anthropic's docs maps onto TalkTerm's existing FR20 confirm-plan pattern.

Assessment

Dimension	Score	Rationale
Feasibility	high	Pure implementation choice — pick multimodal over text-only at day zero. Roughly a day of SDK wiring plus test coverage. Cheaper than the alternative because it removes the image-PDF failure branch entirely.
Impact	medium	Fixes a documented Journey failure mode and removes a class of user-facing errors. Impact is ceiling-bounded because it's a defensive improvement, not a new capability.
Urgency	medium	The cost of deferring grows with every line of text-only PDF code written. The cheapest moment to make this decision is now, before src/ exists.

Adversarial Review

Strongest objection: Vision input is more expensive per token than text input. For BYOK users whose API bill the PRD is explicitly trying to minimize (see #55 Effort Modes), routing every PDF through vision blows up cost on documents that didn't need it.

Rebuttal: The proposal is vision-capable, not vision-first. The intake pipeline should detect whether a PDF has an extractable text layer and use the cheap text path when available; multimodal is the fallback for image-based pages, not the default for all pages. That preserves cost for the common case while eliminating the failure mode. The cost delta of multimodal-for-fallback is small relative to the UX cost of Journey 3's failure message, and it aligns cleanly with the proposed #55 Effort Modes — Economy mode skips the fallback, Power/Turbo enable it.

Suggested Next Step

ADR: draft an architecture decision record for "PDF intake pipeline" that documents the text-layer-first / multimodal-fallback strategy, references the Claude PDF docs, and updates Journey 3 in the PRD to describe the successful-read case. Low-ceremony, high-leverage — ideally landed before the first storage or intake code is written.

---

Proposed by the BMAD Analyst (Mary) on 2026-04-06.
Workflow run