Bulk import: PDF / image / plain-text recipe files

[thebigjoe1]

Summary

Bulk import recipes from PDF, image, and plain-text files (TXT, RTF, MD), matching the import surface Tandoor offers today. Hybrid approach: built-in heuristic parser first (no AI required), AI fallback only when heuristic confidence is low.

Why

• Real ask from multiple users (Lemmy launch thread, selfh.st planning).
• Common comparison point against Tandoor, which has this in for years. Closes the only meaningful migration gap surfaced in launch comments.
• Built-in heuristic means the feature stands on its own without forcing AI on people. AI fallback delivers clean results on weird inputs without users needing to hand-edit.

Architecture decision: hybrid (not AI-only)

How peer apps handle this:

App	Approach
Tandoor	Rule-based heuristic plus Tesseract OCR. No AI required.
Mealie	Rule-based ingredient parser (small CRF model), optional OpenAI on top.
Paprika	Closed-source, heuristic-based.

CookTrace mirrors that pattern. Built-in by default, AI as an optional quality boost.

Staged plan

Phase 1: single-file paste import (PDF, RTF, TXT, MD)

Same UX as the existing Photo Import button, accepts more file types.

Text extraction layer:

• PDF: pdf-parse returns text.
• RTF: minimal inline RTF stripper (no new dep).
• TXT, MD: passthrough.
• Empty-extraction PDFs (scanned, no text layer): error with "use Photo Import per page" guidance.

Parsing layer (the hybrid bit):

• Built-in heuristic parser: section detection, per-line ingredient parsing, step normalization. Emits a confidence score.
• AI fallback: only fires when heuristic confidence is low AND the user has AI configured.
• No-AI users with low-confidence parses get the heuristic result anyway, with a "Best effort, review carefully" badge.

Camera shortcut on the picker screen so the mobile-snap-a-cookbook-page flow stays one tap.

Phase 2: bulk import (multi-file, folder picker, ZIP)

Drop zone + thumbnail picker workflow for "I have 50 recipes to import."

UI:

• Multi-file drop or select.
• Folder picker via webkitdirectory.
• ZIP support: drop a zip of mixed PDF / RTF / TXT / MD and the server expands it.
• Picker grid: per-file extracted name, confidence pill, checkbox.
• Bulk select all / select clean only / clear, plus per-row "Try AI on this one" for low-confidence rows when AI is configured.

Server:

• POST /api/recipes/bulk-scan: accepts multipart with N files or a single ZIP. Extracts text and runs the Phase 1 heuristic parser on each. Caches parsed previews under .import-cache/bulk-<uuid>/.
• POST /api/recipes/bulk-commit: saves selected previews, deletes the cache directory.
• POST /api/recipes/bulk-cancel: explicit cleanup endpoint.
• GET /api/recipes/bulk-text/:uuid/:fileId: returns cached extracted text for the per-row AI fallback.

Cleanup guarantee:

• Multer storage is memory-only. Original ZIP and PDF buffers never touch disk.
• Only parsed JSON previews land in .import-cache/bulk-<uuid>/. Original file bytes are dropped after extraction.
• Cache directory is deleted on commit, on explicit cancel, and on the existing TTL sweep (1 hour, now handles directories too).

Phase 3: cookbook-style multi-recipe PDFs

A single PDF containing many recipes (digital cookbooks). Trace AI reads the whole document and emits one create_recipe tool call per recipe identified.

AI-only by design. Heuristic boundary detection for cookbook layouts is too brittle to ship as a primary path; cookbook documents are inherently the most variable input shape.

UI:

• New menu entry "Import Cookbook" alongside Import from File and Bulk Import.
• Picks a single PDF, server extracts text via the existing /extract-and-parse endpoint.
• Shows extracted text size and page count before kicking off the AI run so the user can confirm before spending AI tokens on a large book.
• "Extract Recipes" calls Trace AI with a multi-recipe system prompt. The AI emits one create_recipe call per recipe; the existing tool handler saves each one straight to the user's library.
• Progress UI counts up as recipes are saved. On AI completion the dialog shows the full list of saved recipes; each row links to its detail page.
• Partial-success handling: if the AI errors mid-run after saving N recipes, those N are kept and a clear message is surfaced.

Server: no new endpoints. Reuses the existing /extract-and-parse for text, and the existing POST /api/recipes/ (called by the create_recipe tool handler) for saves.

Phase 3.5: image-only / scanned cookbook PDFs (deferred)

The Phase 3 flow needs a text layer in the PDF. Scanned old cookbooks (where the PDF is just a bag of page images) yield empty extraction and the dialog surfaces a clear hint pointing the user at single-page Photo Import.

True scanned-cookbook support needs either a pre-render step (PDF page → image) plus AI vision, or native AI-document-block support (Claude / Gemini accept PDF as a document content type). Both are non-trivial and deserve their own scoped follow-up issue once there's user demand.

Non-goals (initially)

• No folder-on-server scan via filesystem path. Security blast radius is too large for the value. Drop zone or upload only.
• No recipe-scrapers-style site-specific extractors per file format. The heuristic parser handles the variability for the common case; AI handles the weird cases.
• No persistent storage of the user's original files. The parsed JSON preview is the only thing cached, and it's deleted on commit or after the TTL.

Tradeoffs

• Adds one server dependency (pdf-parse, pure-JS, around 200 KB).
• Heuristic parser is brittle on edge cases (multi-unit ingredients, unusual layouts, garbled OCR). That's exactly what the AI fallback is for.
• Phase 3 cookbook import is AI-only — there's no clean way to detect cookbook boundaries heuristically across the range of layouts that actually exist in the wild.
• AI parsing cost is per-file (or per-cookbook) when invoked. Local-model users (Ollama, LM Studio) pay nothing. Cloud users only pay when the heuristic flunked or when they explicitly used Cookbook Mode.
• Existing IMPORT_ZIP_MAX_MB env-var cap covers the upload side. Bulk picker can apply per-file size limits.

[thebigjoe1]

Included in RC2