Skip to content

perf(db): add hot-path indexes for recovery crons, Loom imports and storage listings - #2213

Merged
richiemcilroy merged 2 commits into
mainfrom
perf/db-hot-path-indexes
Sep 3, 2026
Merged

perf(db): add hot-path indexes for recovery crons, Loom imports and storage listings#2213
richiemcilroy merged 2 commits into
mainfrom
perf/db-hot-path-indexes

Conversation

@richiemcilroy

@richiemcilroy richiemcilroy commented Sep 3, 2026

Copy link
Copy Markdown
Member

Three additive indexes plus a one-line predicate change, driven by PlanetScale Insights for cap/cap-production/main and verified with EXPLAIN on a branch restored from today's backup.

What changed

  • videos (isScreenshot, transcriptionStatus, createdAt): the two stalled-pipeline recovery crons in apps/web/lib/video-pipeline-recovery.ts full-scanned videos every run (Insights: 22.6M + 22.6M rows read over 96 runs each, p99 197ms / 240ms). After: index range scan estimated at 735 / 3,355 rows instead of 215k.
  • imported_videos (id): every imported_videos.id = ? lookup was a table scan (no index on id). Video deletion runs DELETE ... WHERE id = ? inside a transaction, which under REPEATABLE READ next-key-locks every scanned row, so a delete could block concurrent Loom imports. The Loom recovery join also did a 304-row PK range per candidate. After: single-row covering lookup.
  • storage_objects (integrationId, objectKey(191)) + listObjectsByPrefix now adds objectKey LIKE prefix% alongside the existing BINARY objectKey LIKE prefix%. The collation-aware predicate is a strict superset of the binary one, so results are unchanged, but it lets MySQL range-scan the new index. Before: every listing read all objects for the integration (Insights: 9.8M rows over 1,229 runs, p99 628ms; the largest integration has 278k objects). After: range scan on the prefix for typical owner/video/ listings; large HLS-segment prefixes keep today's plan.

Validation

  • EXPLAIN on the restored branch for the three target queries and for the hot dashboard/list queries on the same tables (caps list, user cap count, storage hash/status lookups, notifications list): target plans switched to the new indexes, every other plan unchanged.
  • Migration generated with pnpm db:generate; tsc --noEmit clean for @cap/database and @cap/web-backend; Biome clean.
  • Schema is applied via PlanetScale deploy request Custom s3 bucket backend #90 (additive, online, revertible); code and schema are independent so deploy order does not matter.

Greptile Summary

The PR adds three database indexes for recovery, imported-video, and storage-listing access paths while preserving case-sensitive storage-prefix semantics.

  • Adds indexes for stalled video-pipeline recovery queries, imported-video ID lookups, and integration-scoped object-key prefixes.
  • Adds an optimizer-friendly collation-aware prefix predicate while retaining the existing binary predicate and pagination ordering.
  • Appends the generated migration snapshot and journal entry.

Confidence Score: 5/5

The PR appears safe to merge with no actionable correctness or security issues identified.

The added indexes are compatible with the repository’s database targets and migration chain, while the storage query retains its original binary filtering and pagination semantics.

Important Files Changed

Filename Overview
packages/web-backend/src/Storage/StorageRepo.ts Adds an index-usable LIKE predicate without changing the effective case-sensitive result set or binary pagination order.
packages/database/schema.ts Declares three additive indexes that align with the generated migration artifacts.
packages/database/migrations/0042_perf_indexes.sql Creates valid MySQL 8 indexes for the three targeted query paths.
packages/database/migrations/meta/0042_snapshot.json Records the new indexes consistently, including the raw SQL prefix-index expression.
packages/database/migrations/meta/_journal.json Appends migration 0042 without modifying prior migration history.

Reviews (1): Last reviewed commit: "perf(storage): let prefix listings use t..." | Re-trigger Greptile

Context used:

@richiemcilroy
richiemcilroy merged commit 676513e into main Sep 3, 2026
27 checks passed
@richiemcilroy
richiemcilroy deleted the perf/db-hot-path-indexes branch September 3, 2026 15:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant