Importer perf: capture plan in tasks/

[ajslater]

Summary

Meta-plan + 6 sub-plans for taking a 600k-comic bulk import faster without sacrificing correctness. Plan-only — no code changes. Mirrors the #621 / #625 plan-capture pattern.

• Surface map (62 files, ~5k LOC across 8 phase subdirs)
• Confirmed perf hot spots with file:line references
• Sub-plan ordering by impact-per-LOC, surgical wins first, structural refactor last

Sub-plans (in suggested ship order)

1. 01-link-batching.md — link/prepare.py:105-135 fires ~6.6M SELECTs for a 600k import. Fix: name→pk dict per model. Drops to ~14 SELECTs total.
2. 02-create-fk-batching.md — create/foreign_keys.py:46-75 fires ~2M SELECTs via per-row field_model.objects.get(). Fix: pre-fetch parent FK pk-maps, pass via <field>_id=<pk> shortcut.
3. 03-query-prune.md — moved/comics.py:60-68 per-comic Folder N+1 (1.2M → 2), narrowed prefetch_related/select_related to fields actually referenced, hoisted status_controller.update() out of inner loops.
4. 05-sqlite-tuning.md — phase-level transaction.atomic (no atomic exists anywhere in scribe currently), importer-scoped PRAGMAs (cache_size=512MB, wal_autocheckpoint=0), connection_created signal handler, post-import PRAGMA optimize + wal_checkpoint(TRUNCATE).
5. 06-comicbox-side.md — comicbox-repo PRs: to_dict(keys=...) field projection, CIX-first probe order, filesystem-mtime pre-filter to skip archive opens entirely on unchanged files.
6. 04-streaming-pipeline.md — split into reference-data phase (all-at-once, ~800MB) + chunked comic ingestion (5-10k per chunk, ~30-60MB peak). JSONL spill file. Resume-from-watermark. Memory cap drops from ~10GB peak to <1.5GB. Biggest refactor; ships last.

Test plan

• Review 00-meta.md for accuracy of surface map and findings
• Confirm sub-plan ordering matches your priorities
• Flag any sub-plan that needs more detail or has a wrong premise
• Approve as planning artifact (no code changes); implementation lands in follow-up PRs per sub-plan

🤖 Generated with Claude Code

[ajslater]

Revisions in 331860aa from review feedback:

04-streaming-pipeline.md

• Dynamic chunk size via codex/librarian/memory.py:get_mem_limit() (already handles cgroups2/cgroups1 + psutil.virtual_memory() fallback). Floor 1000, ceiling 50k, default 25% mem budget. Indicative table: ~50k on a 4GB Pi, ~16k on a 1GB Pi, ~4k on a 256MB container, floor on anything smaller.
• Spill file moves to ROOT_CACHE_PATH / "import_cache.jsonl" (settings/__init__.py:523). Documented why /tmp is wrong (tmpfs OOM risk on Pi) and why ROOT_CACHE_PATH is right (persistent across reboots, same volume as DB, already managed).

06-comicbox-side.md

• Withdrew Improvement C entirely. After reading comicbox/box/sources.py:155-174 and comicbox/box/load.py:118-141, comicbox already does the optimal thing: walks namelist(), matches filenames against FILENAME_FORMAT_MAP, and only loads/parses formats that are actually present — each SourceData carries the pre-selected fmt so _load_metadata skips the try-every-format fallback. Probe ordering on top of that would risk dropping MetronInfo/CoMet/CBI data that codex officially supports.
• Reworded the worker-compute-waste framing to clarify the remaining waste is within each format (parsing keys codex doesn't read), not across formats. Improvement A (to_dict(keys=...) field projection) still stands.
• Cleaned up cross-references to the dropped improvement in commit shape, test plan, and correctness invariants.