Importer perf: chunk per-comic phases to bound peak memory

[ajslater]

Summary

Implements tasks/importer-perf/04-streaming-pipeline.md from PR #627. Final entry in the importer perf series after #628 / #629 / #630 / #631 / #632 / #633.

After looking at the actual code post-#628–#633, the plan's elaborate "two-pass spill-file" design turned out to be unnecessary. A simple chunk loop around the per-comic phases (read → query → create_and_update → link) works because every FK bulk_create already uses update_conflicts=True — a Publisher / Imprint / Series / Volume created by chunk N is visible to chunk N+1 as an idempotent UPSERT, not a duplicate-key error. No reference-data preflight pass needed; no spill file needed.

Three commits, each independently reviewable:

55518dc1 — setdefault for cross-chunk metadata accumulators

Switches FIS, FTS_UPDATE, FTS_CREATE, FTS_CREATED_M2MS, FTS_UPDATED_M2MS, FTS_EXISTING_M2MS from self.metadata[KEY] = {} to self.metadata.setdefault(KEY, {}) so a second pass over a different chunk doesn't erase the first chunk's contents. Per-pass-only keys keep their direct = {} reset. No behavior change in the single-pass pipeline.

6bae6955 — counts use += not =

The three counts create_and_update writes (tags, covers, comic) were assigned with =, which would overwrite each chunk's running total. Switched to +=. Counts.__init__ zeros all attrs so single-pass semantics are unchanged (0 + N == N).

9fd3b039 — chunked apply()

Wraps read → query → create_and_update → link in a chunk loop. Pre-phases (init_apply, move_and_modify_dirs) and post-phases (fail_imports, delete, full_text_search) still run once.

• Chunk size derived from codex.librarian.memory.get_mem_limit() (cgroups + psutil fallback) times IMPORTER_CHUNK_MEM_FRACTION (default 0.25, configurable). Floor 1000, ceiling 50000.
  • 4 GiB host: ~50k chunks (capped)
  • 1 GiB host: ~16k chunks
  • 256 MiB container: ~4k chunks
  • 64 MiB cgroup: floor ~1000
• Path partitioning: files_created | files_modified is sorted (deterministic chunk boundaries for log diffing + future resume work), sliced into chunks of chunk_size, then the original created vs modified split is restored per chunk via set intersection.

Why this design beats the original plan

The plan proposed a two-pass architecture with a JSONL spill file (~5 PRs A→E). After the surgical N+1 wins from #628–#633, the actual remaining work is small enough that:

• Reference-data accumulation across chunks is safe via UPSERT (no preflight pass needed)
• Memory pressure comes from LINK_FKS / LINK_M2MS / CREATE_COMICS holding all 600k comics' data — chunking those is the actual fix
• A spill file would just defer the same data through disk; not needed when chunks fit in RAM by construction

This is one PR, ~120 LOC across 6 files, instead of 5 PRs and ~1500 LOC.

Test plan

• make fix clean
• make lint-python clean (0 errors, 0 warnings)
• pytest tests/importer/ tests/test_search_fts.py — 7 passed
• Round-trip test: import a 1k-comic library; assert resulting Comic / FK / M2M row state is identical to a pre-PR baseline
• Memory ceiling test: import a 100k-comic library under tracemalloc; expect peak RSS < 1.5 GiB
• Chunk boundary test: fixture with paths sharing FKs across chunks; verify the FKs are deduplicated correctly via UPSERT
• Wall clock: 100k-comic dev fixture; chunked pipeline should be within 10% of all-at-once wall clock (memory savings shouldn't cost time)

🤖 Generated with Claude Code