Importer perf: phase-level transaction.atomic + scoped SQLite PRAGMAs

[ajslater]

Summary

Implements tasks/importer-perf/05-sqlite-tuning.md from PR #627. Fourth in the importer perf series after #628 (link prepare), #629 (create FKs), #630 (query-prune).

Two independently revertable commits:

ce82a8d0 — wrap create/link phases in transaction.atomic

The scribe daemon previously had no transaction.atomic anywhere — every bulk_create / bulk_update ran in autocommit. For a fresh 600k-comic import that's ~2,300 fsyncs across create_and_update + link (50 FK creates + 600 comic creates + 200 M2M link batches + 1,500 comic updates). At SATA-SSD fsync costs that's ~15-25s of pure wait time the SQL itself doesn't need.

transaction.atomic wraps both phases so all batches commit on one fsync each. The existing abort_event checks stay inside; abort returns out cleanly (Django commits on normal exit, only rolls back on uncaught exception). The codex daemon already serializes writers via db_write_lock, so the long-write transaction doesn't starve other writers; readers under WAL never block on a writer regardless of transaction length.

d6f6aca0 — importer-scoped PRAGMAs + post-import checkpoint/optimize

New pragmas.py module exposes an importer_pragmas() context manager that wraps the entire apply() run:

• Bumps cache_size to IMPORTER_SQLITE_CACHE_KB (default 512 MiB, configurable via importer.sqlite_cache_kb). The 600k-comic working set fits, eliminating page-cache misses across the link-phase prune walk.
• Sets wal_autocheckpoint=0 to defer WAL checkpoints during the import. The default 1000-page autocheckpoint fires hundreds of times per large import, stalling the writer each time.
• Hooks the connection_created signal so a mid-import reconnect (CONN_MAX_AGE recycle, pool grow) re-applies the override instead of silently inheriting the steady-state values.
• On exit: restores the steady-state PRAGMAs, runs PRAGMA wal_checkpoint(TRUNCATE) (otherwise the deferred frames stay resident until a reader transaction crosses them), and runs PRAGMA optimize so the planner sees fresh statistics post-import.

Expected speedup

Order-of-magnitude estimate for a fresh 600k-comic import:

Optimization	Time saved
Phase-level atomic() (2,300 fsyncs → 5)	~15-25 s
512 MiB cache (eliminates link-phase page misses)	~30-90 s
wal_autocheckpoint=0 (no mid-import checkpoint stalls)	~5-15 s
Combined	~1-2 minutes off a 30-min import

Cumulative with the surgical-N+1 wins from #628/#629/#630.

Test plan

• make fix clean
• make lint-python clean (0 errors, 0 warnings)
• pytest tests/importer/ tests/test_search_fts.py — 7 passed
• WAL size check: after a 1k-comic test import, Path('codex.sqlite3-wal').stat().st_size < 4 KiB (one frame post-truncate)
• Wall-clock measurement on a real-scale fixture once one is available

🤖 Generated with Claude Code