Partial index for metadata_attempt_at drain query (closes #659)

[jakebromberg]

Summary

Migration 0070 adds flowsheet_metadata_attempt_pending_idx — a partial B-tree on flowsheet(id) covering the metadata_attempt_at IS NULL tail. Required by both #638 (historical drain) and #639 Phase 2 (recurring drift-repair sweep). Without it each batch seq-scans 2.6M+ rows just to find the NULL slice, and #639 Phase 2's recurring cadence amplifies the cost.

CREATE INDEX IF NOT EXISTS "flowsheet_metadata_attempt_pending_idx"
  ON "wxyc_schema"."flowsheet" USING btree ("id")
  WHERE "entry_type" = 'track'
    AND "artist_name" IS NOT NULL
    AND "metadata_attempt_at" IS NULL;

The metadata_attempt_at column shipped in #658, so the partial predicate is now valid against prod.

Production runbook (please run before merging)

Drizzle wraps each migration in a transaction, which means the in-migration CREATE INDEX runs not as CONCURRENTLY and acquires a ShareLock on flowsheet for the build duration — that's the failure mode this index is supposed to prevent. The established workaround (per 0057, 0068) is: build out-of-band on prod first via CREATE INDEX CONCURRENTLY, then merge this PR so the migration's IF NOT EXISTS makes apply a no-op.

ssh wxyc-ec2
psql "$DATABASE_URL" -c "
  CREATE INDEX CONCURRENTLY \"flowsheet_metadata_attempt_pending_idx\"
    ON \"wxyc_schema\".\"flowsheet\" USING btree (\"id\")
    WHERE \"entry_type\" = 'track'
      AND \"artist_name\" IS NOT NULL
      AND \"metadata_attempt_at\" IS NULL;
"

Expected build window: a few minutes, no AccessExclusiveLock, no INSERT pause.

After the CONCURRENTLY build completes, verify and merge:

SELECT indexname FROM pg_indexes
 WHERE schemaname='wxyc_schema'
   AND tablename='flowsheet'
   AND indexname='flowsheet_metadata_attempt_pending_idx';
-- expect: 1 row

Shape choices (also documented in the migration comment)

• Single-column (id) is enough. The drain orders by id and pages by id > $lastId. Heap lookup for the row body is cheap given the small post-filter row count.
• add_time not in the predicate. The now() - interval '60 seconds' filter exists to dodge the legacy mirror's eventual-consistency window on freshly-inserted rows (see Re-emit liveFs SSE event after metadata enrichment UPDATE lands #628), not for selectivity. Adding it would inflate the index without improving plans.
• Partial keeps the index small. ~1.86M NULL rows pre-drain → ~50 MB. Long-term the residual stays small and the index shrinks accordingly.

CLAUDE.md update

The new CREATE INDEX IF NOT EXISTS paragraph documents the established pattern (0068 set it; 0070 follows). Tightens the prior "DDL exactly as drizzle-kit emitted" guidance from #658, which was over-broad — IF NOT EXISTS on CREATE INDEX is the only sanctioned hand-edit on the DDL itself, scoped to migrations whose deploy runbook involves a CONCURRENTLY pre-build.

Tests

tests/unit/database/schema.flowsheet-metadata-attempt-pending-idx.test.ts — 6 assertions modelled on the 0057 test:

• migration exists at the journal-pointed path
• DDL has IF NOT EXISTS (idempotent against prod-prebuilt index)
• DDL targets flowsheet(id)
• WHERE predicate covers all three filter clauses (entry_type='track', artist_name IS NOT NULL, metadata_attempt_at IS NULL)
• DDL does not use CREATE INDEX CONCURRENTLY (drizzle txn-wrap incompatible). The negative regex is anchored on non-comment lines so the runbook's CREATE INDEX CONCURRENTLY example in the comment block doesn't false-positive.
• schema.ts declares the index so drizzle-kit drift detection sees it.

TDD: all six failed first, then the schema + migration brought them green.

Test plan

• npm run typecheck clean
• npm run lint 0 errors
• npm run format:check clean
• npm run lint:migrations passes (1 historical allowlisted warning, unchanged)
• npm run test:unit 1362 / 1362 passing (the 6 new assertions plus everything else)
• CI passes
• Pre-merge: run the CREATE INDEX CONCURRENTLY ... command against prod RDS (see runbook above)
• Post-merge: Auto Build & Deploy migrate finds the index already present and skips it via IF NOT EXISTS
• Post-merge: EXPLAIN the drain query and confirm Index Scan using flowsheet_metadata_attempt_pending_idx (rather than Seq Scan on flowsheet)

Unblocks

• [A.1.a] flowsheet-metadata-backfill historical drain job #638 (historical drain) — its core query needs this index before deploy
• [A.1.b] Add metadata_attempt_at marker and flip backfill to recurring drift-repair #639 Phase 2 (recurring sweep) — same query, recurring cadence amplifies the cost

Closes #659