fix(db): index the two hot unindexed FK referencing columns

[jiashuoz]

The bug

Postgres auto-indexes the referenced side of a foreign key (the PK) but never the referencing side. Both columns below are ON DELETE SET NULL, which Postgres implements as an internal per-row trigger firing once for every deleted parent row:

UPDATE ONLY child SET fk = NULL WHERE fk = $1

Unindexed, that's a sequential scan of the entire child table — per deleted row, not per statement.

The measurement

Same database, same moment, same predicate shape, same FK action. The only difference is the index:

table	rows	pages	index on FK column	plan	cost
webhook_subscriber_deliveries	421,319	88,177	none	Seq Scan	93,443.49
webhook_events	454,487	98,432	idx_webhook_events_message_created (026)	Index Scan	8.44

~11,000× — and webhook_events is the larger table.

Why it mattered

A 1000-message delete batch was ~88M page visits. Every batch aborted on a 60s statement_timeout. Downstream, one missing index produced four symptoms that looked unrelated:

1. deleteAgent exceeded the 60s proxy budget → HTML 504
2. that HTML body failed the conformance suite's response-schema gate (spec documents application/json)
3. the slow deletes pinned both slots of maxConcurrentDestructive, so unrelated deletes 429'd
4. two of those 504s in one 5m window fired the HTTP 5xx fast burn alert

Latent in production, not staging-specific. 025's own header says message_id is ON DELETE SET NULL so the 30-day retention purge can proceed — the design depends on exactly the path that was unindexed, and the cost scales with table size.

The two migrations

106 — webhook_subscriber_deliveries.message_id, plain index. 419,593 of 424,276 rows (98.9%) are non-null, so a partial index would be within ~1% of the same size.

107 — messages.reviewed_by_user_id, partial index. Only 4,333 of 549,579 rows (0.79%) are non-null, making partial ~127× smaller. The FK check still uses it because col = $1 implies col IS NOT NULL — the same implication the planner already exploits for the partial idx_webhook_events_message_created, confirmed by EXPLAIN. Rarer path, but worse per occurrence, and deleteAccount is itself in destructiveOps, so it runs under the same cap and timeout budget — and "the account deletion timed out" is a poor answer on a data-rights path.

Both use CREATE INDEX CONCURRENTLY with e2a:no-transaction, following 059, which added an index to this same table for this same class of problem. Both carry the invalid-index recovery note 059 established.

Deliberately not in scope

The audit found 9 more single-column FKs with no leading index, all on tables that are currently empty or never-analyzed on staging:

contact_engagements.contact_id (CASCADE)      oauth_refresh_tokens.user_id (CASCADE)
oauth_auth_codes.user_id (CASCADE)            oauth_clients.created_by_user_id (SET NULL)
oauth_refresh_tokens.client_id (CASCADE)      oauth_auth_codes.client_id (CASCADE)
user_sessions.user_id (CASCADE)               sending_ramp_reservations.user_id (CASCADE)
oauth_pkce_requests.client_id (CASCADE)

I did not blanket-index these: indexes cost write throughput, and a seq scan of an empty table is free. But several are per-user tables that grow (user_sessions, oauth_refresh_tokens), and I can't size them on prod from staging. Worth a follow-up audit against prod row counts, plus a schema test with an explicit allowlist so a new unindexed FK becomes a deliberate decision rather than an accident.

Tests

go test ./internal/identity/ -run Migrat -short green; go build ./... clean. The migration runner's e2a:no-transaction single-statement requirement is satisfied by both files.

🤖 Generated with Claude Code