Ingestion concurrency + scale review (phases 1-2); fixes held for db-reliability

[BigSimmo]

Summary

Phases 1-2 of the ingestion-concurrency/scale review. Docs only — no behavior changes yet. Phase-3 fixes are deliberately held until the db-reliability branch merges; fix commits will be added to this PR afterwards.

Phase 1 — docs/ingestion-state-machine.md

Full state model of documents × ingestion_jobs × indexing_v3_agent_jobs × index generations: legal composite states, writer × transition matrix (worker / edge agent / API routes / ops scripts), crash-window analysis, and a verified violations registry.

Method: 7 scoped race-hunter agents (one per writer × transition group), 24 consolidated claims, each re-derived by an independent adversarial verifier (exact reaching schedule or killing guard required). 24/24 confirmed, 5 narrowed, 0 refuted. Two facts checked directly against the live catalog.

Highest-severity confirmed findings (7 need no concurrency at all):

• R11 — aborted DELETEs leave failed ledger rows with live paths; cleanup:storage then permanently deletes a live document's PDF + images
• R15 — retrying a failed (or completed) job of an indexed document sets it queued, so the next claim's reset_document_index destroys the live committed index
• R3 — double-build metadata repoint leaves an "indexed" doc retrieval-empty, then abandoned-generation cleanup deletes the surviving rows
• R4/R19/R23 — generation-blind resets (worker reclaim / queue recovery / cleanup RPC) amputate a live build; commit stamps full client-side counts over the partial row set (silent partial clinical index)
• R1 — no lease heartbeat: every >45-min job is reclaimed while alive; no write anywhere is fenced by locked_by
• R12 — every successful reindex permanently strands the previous generation's image objects (only the DELETE route ever writes the storage ledger)
• R22 — routine queue recovery silently force-completes every queued reindex of indexed docs
• R24d/c — enrichment writers (edge agent / route mode:'enrichment' / worker inline) are mutually unguarded: completed/"good" docs with zero artifacts, duplicate index units, detached cards; repair_strict_enrichment_gate_batch is invoked by nothing
• R24e — schema.sql drift: fresh envs have a fatal ingestion_job_stages.job_id FK and a claim RPC that never seeds agent jobs (live verified unaffected)

Phase 2 — docs/scale-readiness-review.md

Read-only live profiling (explain_retrieval_rpc + direct EXPLAIN ANALYZE; one SET LOCAL experiment inside a rolled-back transaction). At 10× (~20k docs):

• F1 — match_document_table_facts_text already takes 6.75 s (unindexable 5-column trigram similarity() OR-disjunct); ~linear growth; called 3× in parallel per request
• F2 — document_index_units has no vector index and no vector arm — semantic recall for visual/enrichment evidence is structurally zero
• F3 — hnsw.ef_search=40 silently caps every vector arm below its LIMIT (measured 40 vs 72 rows at ef=200)
• F4 — ~5 GB of HNSW on a 256 MB-buffer instance (~50 GB at 10×)
• F6 — 10-14 RPC fan-out per cold request vs max_connections 60

Ranked mitigation list in the doc; retrieval-affecting changes must pass the content_mrr@10 eval gate before defaults move.

Phase 3 (pending)

Fix backlog is ranked in the state-machine doc §8 (janitor guard, retry-route demotion fix, lease heartbeat + locked_by fences, metadata merge, recovery hardening, storage reconciliation, …). Held until db-reliability merges; each fix will be the smallest safe change, via committed migrations + code, gated on verify:cheap, check:indexing, reindex:health.

🤖 Generated with Claude Code

[supabase]

This pull request has been ignored for the connected project sjrfecxgysukkwxsowpy because there are no changes detected in supabase directory. You can change this behaviour in Project Integrations Settings ↗︎.

---

Preview Branches by Supabase.
Learn more about Supabase Branching ↗︎.

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.