fix: Performance and reliability improvements for database and caching

[filthyrake]

Summary

• [Reliability] Missing Database Operation Timeouts #549: Add 30s timeout to all database retry operations to prevent queries from hanging indefinitely under database contention
• [Performance] Unbounded VOD List Count Query Causes Table Scans #544: Replace VOD list COUNT query with COUNT(*) OVER() window function for efficient O(1) pagination
• [Performance] Chat Message Pagination Count Query Is Inefficient #546: Replace chat pagination COUNT query with limit+1 pattern, eliminating O(n) table scans
• [Performance] Bitrate Cache Lock Contention Limits Throughput #545: Add sharded locks (16 shards) to bitrate cache to reduce lock contention under high load
• [Reliability] Bitrate Cache Has No Size Limit #553: Add max size limit (1000 entries) to bitrate cache to prevent unbounded memory growth

Test plan

• All db_retry tests pass (31/31)
• All modified modules import cleanly
• Verify VOD list pagination still works correctly
• Verify chat message pagination still works correctly
• Verify bitrate estimation under concurrent load

Closes #544, #545, #546, #549, #553

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR improves reliability and performance across the studio/live stack by adding database operation timeouts, optimizing pagination queries, and reducing contention/unbounded growth in the bitrate cache.

Changes:

• Add per-operation DB query timeouts and a total retry deadline in db_retry.
• Optimize VOD list pagination by using a COUNT(*) OVER() window function.
• Optimize chat message pagination by switching from COUNT queries to a limit + 1 approach.
• Reduce bitrate cache lock contention via sharded locks and cap cache growth with a max-size eviction strategy.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
api/studio_vod.py	Replaces separate COUNT query with COUNT(*) OVER() and derives total from returned rows.
api/studio_chat.py	Removes COUNT query for chat pagination, uses limit+1 to compute has_more, and changes total semantics.
api/live_schemas.py	Updates chat list schema docs to reflect cursor pagination and new total meaning.
api/live_metrics.py	Introduces sharded locks and max-size eviction for bitrate cache.
api/db_retry.py	Adds per-query timeouts and an overall retry deadline to prevent indefinite hangs.

Comments suppressed due to low confidence (1)

api/studio_chat.py:234

• The new pagination query relies on WHERE stream_id = ? AND deleted_at IS NULL plus id < before_id and ORDER BY id DESC LIMIT …. There isn’t an index matching this pattern (existing migration creates ix_chat_messages_stream_created on (stream_id, created_at)), so this can still degrade into inefficient scans on large chats. Consider adding a migration for a composite/partial index such as (stream_id, id DESC) with WHERE deleted_at IS NULL (or including deleted_at in the index) to support the new access path.

    query = (
        sa.select(
            chat_messages,
            users.c.username.label("username"),
        )
        .select_from(
            chat_messages.outerjoin(users, chat_messages.c.user_id == users.c.id)
        )
        .where(chat_messages.c.stream_id == stream["id"])
        .where(chat_messages.c.deleted_at.is_(None))  # Exclude deleted messages
        .order_by(chat_messages.c.id.desc())
        .limit(effective_limit + 1)  # Fetch one extra to detect has_more
    )

    if before_id:
        query = query.where(chat_messages.c.id < before_id)

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

This changes the meaning of the total field from “total messages matching the query” to “messages returned in this page”. That’s a backwards-incompatible API change and the field name is likely to be misinterpreted by clients. Consider either (a) renaming the field (e.g., returned/count) and keeping total semantics, or (b) making total optional/omitted and introducing a new field for per-page count, to avoid breaking existing consumers.

[Copilot]

In _enforce_cache_size_limit, non_expired filtering does if k not in keys_to_evict where keys_to_evict is a list. That makes eviction selection potentially O(n^2) as cache size grows past the limit. Consider using a set(keys_to_evict) for membership checks (and/or building non_expired from a snapshot like list(_bitrate_cache.items())) to keep eviction overhead predictable.

[Copilot]

New timeout behavior is introduced (per-attempt asyncio.wait_for(...) raising DatabaseTimeoutError, plus the new total_deadline enforcement in execute_with_retry), but the existing db-retry test suite doesn’t cover these paths. Add unit tests in tests/test_db_retry.py to validate: (1) query timeouts raise DatabaseTimeoutError, and (2) total_deadline stops retries once exceeded.

[Copilot]

total is derived from the first returned row’s total_count. When the requested page is empty (e.g., page is past the last page), rows is empty and this returns total = 0, which is incorrect and can break pagination UI logic. Consider falling back to a separate COUNT query when rows is empty (or restructure the query so total can be returned even for empty pages).

[Copilot]

This endpoint filters by videos.deleted_at IS NULL, optional videos.status, and (for non-admins) videos.owner_id OR live_streams.owner_id, then orders by videos.created_at DESC. There is no composite index covering these access patterns, so the window COUNT(*) OVER() can still require large scans/sorts on bigger datasets. Consider adding a migration for an index that matches the common filters/order (e.g., a composite/partial index involving owner_id, deleted_at, status, and created_at).