perf: make database connection pools resilient under concurrent load#30
Merged
Conversation
List and detail handlers assembled their DTOs by fanning out many related-table queries through tokio::join! (up to 14 for the full series response). Each arm acquired its own pool connection simultaneously, so a single request could hold most of the pool at once. Under concurrent load (e.g. many open tabs) this saturated the connection pool and made acquire() block for seconds, especially on SQLite where the pool is small. Add a shared-semaphore gate (db_batch::with_permit / fan_out_limiter) and wrap each join arm so at most a fixed number of queries hold a connection at a time; the rest park on the semaphore without a connection. A semaphore is used rather than a homogeneous stream because the arms return distinct types and must keep their positional bindings. Applied to the series, books, and bulk handlers' fan-out sites. The bound is currently a module constant; making it per-backend configurable is left for a follow-up. Includes unit tests for bound enforcement and result ordering.
Task workers, the scheduler, and the inventory poller shared the API's connection pool, so heavy or bursty background work (scans, analysis, thumbnail generation) could hold every connection and starve interactive requests — acute on SQLite, whose pool is small. Add Database::new_background(), which builds a separate, smaller pool over the same database (cloning the config with an overridden max_connections, reusing the existing connect/pragma setup, and running no migrations). In serve, route the in-process background subsystems to this pool while the API and request-path services keep the primary pool. The background pool is only created when workers run in-process; web-only deployments and the dedicated worker process keep a single pool. On Postgres the pool is additive to the API pool, so the server's max_connections must accommodate the total (documented; startup validation to follow). SQLite's busy_timeout default already makes writers wait rather than error. Includes a test verifying the background pool is independent but targets the same database.
…able Expose per-backend batch_fan_out and background_max_connections settings (env-overridable) so the per-request query fan-out cap and the background work pool can be tuned without recompiling. Bump the SQLite max_connections default to 64 for read headroom under WAL. The fan-out bound is resolved once at startup and read by the list and detail handlers (replacing the hardcoded default); serve reads the background pool size from config. On Postgres, warn at startup if the API and background pools together exceed the server's max_connections, since the background pool is additive when workers run in-process. Update the sample SQLite and Docker configs and the configuration and performance docs, including the API-vs-background pool split and the SQLite write-serialization caveat.
Fire many concurrent series-list requests against a deliberately small SQLite pool with a short acquire timeout and assert they all succeed. This guards the per-request query fan-out bound: if it regressed to an unbounded tokio::join!, the cross-request demand would starve the pool and the requests would fail with acquire timeouts instead of 200s. Includes an ignored PostgreSQL parity variant that runs the same workload when a test server is available. The deterministic proof that concurrency stays within the bound lives in the db_batch unit test; this test verifies the bound is wired through the real HTTP endpoints under load.
…orms Freshness is already driven by SSE entity events, which invalidate the relevant query caches when data actually changes. The global staleTime of 5s combined with refetchOnWindowFocus meant switching between many open tabs re-ran every active query, including the heavy series list — a needless load amplifier. Raise staleTime to 30s and disable refetch-on-focus, relying on the SSE stream for real-time updates. Keep refetch-on-reconnect so the client recovers any events missed while offline.
Deploying codex with
|
| Latest commit: |
be6dd58
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://d8210fcf.codex-asm.pages.dev |
| Branch Preview URL: | https://db-optimizations.codex-asm.pages.dev |
The default (non-full) /series/list built its DTOs with an unbounded futures::future::join_all over a per-series helper that ran ~6 sequential queries each, including a per-series library lookup. For a 200-series page that fanned out to ~1000 concurrent, unbounded connection acquisitions from a single request — enough to exhaust the pool on its own. Swap every such list call site to the existing batched converter, which issues a fixed handful of queries regardless of page size. On the client, every cover_updated event force-refetched the entire series/books list. During a library scan or bulk thumbnail regeneration — which emit a cover/book event per item — this produced hundreds of heavy list refetches that piled up and cancelled each other, hammering the API pool. Cover changes only affect an image URL (already cache-busted via the cover-timestamp store), so drop the list refetch on cover events entirely, and throttle the legitimate data-change invalidations so a burst coalesces into a handful of refetches instead of one per event. Updates the affected event-handler tests and strengthens the pool contention test to cover the non-full list path.
list_on_deck fetched every unread book across all of a user's eligible series into memory, then picked the first book per series, sorted, and paginated in memory — discarding ~99% of the rows and scaling with the size of the library. On large libraries this made the home page's On Deck section slow. Rework it: prune eligible series by visibility and library in Rust, then SELECT DISTINCT series_id to find which eligible series actually have an on-deck book (the total), order and paginate that series list, and load unread books only for the page's series before picking the first per series. Rows loaded now scale with page size, not library size. Also fix on-deck pagination past the first page: callers pass a 0-indexed row offset, but the repository recomputed start = offset * page_size, so later pages skipped too far and returned nothing. The two callers also disagreed — the native handler passed an offset while the Komga handler passed a 0-indexed page index. Standardize both on an offset (Komga now passes page * size) and slice with start = offset. Adds a pagination test covering multiple on-deck series.
Heavy background task processing still produced two frontend request storms after the earlier refetch-storm work. Both are client-side only. - release_announced: the SSE handler invalidated the releases inbox, per-series ledger, tracking, and the heavy series "full" query on every event, unthrottled. A release-source poll (or a source reset, which re-emits every release as announced) lands a burst of events, turning one detail-page refresh into a full/tracking/releases refetch flood. Now throttled through the same coalescing path the list uses, and the "full" invalidation is dropped entirely: a release advances tracking.latestKnownChapter, not local book counts or the upstream gap, so the Behind-by badge recomputes from the cheap tracking refetch alone. - List invalidation over-match: the throttled series/books list invalidators fired the bare ["series"] / ["books"] keys, which prefix-match every open detail query (full, tracking, aliases, releases, covers). With detail tabs open during a scan, one list refetch fanned out across all of them. Target the specific list/grid/home-section keys instead; genuinely-changed details are still refreshed by their targeted per-entity invalidations. - Task-progress poller fan-out: useTaskProgress is mounted in many places at once, and each instance ran its own poll loop plus immediate fetches on mount, multiplying the tasks/stats and processing-tasks requests. Rewrite it as a useSyncExternalStore subscriber over a single reference-counted manager that owns one poll loop and one SSE subscription, so the server sees one poller regardless of mount count. Public API and merge semantics are unchanged. Tests added for the new throttle/no-full release contract, the narrowed list invalidation, and shared-poller deduplication across mounts.
The SSE handler invalidated the bare ["series", id] key on series update/metadata events and ["series"/"books", id] on cover events. React Query matches by prefix, so each of these also invalidated the entity's independent sub-resources — tracking config, aliases, and the release ledger for a series; genres, tags, and external links for a book — none of which a content/metadata change or a cover regeneration actually touches (they have their own event types). With several detail tabs open during an analyze run or a bulk thumbnail regeneration, that turned every per-entity event into a tracking+aliases+releases (or genres+tags) refetch burst for each affected entity. Target the specific detail queries instead: series content/metadata invalidates the full DTO plus the metadata view; a cover change invalidates only the detail DTO that carries the cover-source fields. There is no bare two-segment detail query, so nothing relied on the broad prefix. The book_updated branch keeps its broad invalidation deliberately: an analyze pass does rewrite a book's genres, tags, and metadata. Updates the cover-event tests to assert the narrowed keys.
…ge refresh Audited every frontend invalidation against the actual read-query keys and fixed the remaining cases that refetch far more than they change, plus the one burst-driven refresh that wasn't throttled. - Release mutations (dismiss/acquire/patch/delete/bulk/reset) invalidated the bare ["series"] key. The series DTO carries no release-derived fields, so the only series query a release change affects is the per-series ledger panel — but the bare key, via prefix matching, also refetched full/metadata/covers/tracking/aliases/lists across every open tab. Replace with a predicate targeting only the release ledger panels (and the tracking row for ledger-wiping resets). - MediaCard analyze actions invalidated the whole ["series"]/["books"] namespace on enqueue, but analysis is async: nothing has changed yet, and the real refresh arrives via the completion SSE event. Narrow to the single entity's detail query. - The tasks settings page refetched the task list (four status fetches under the "all" filter) plus stats on every task-completion SSE event, unthrottled — a request storm when a bulk analyze or scan completes thousands of tasks. Throttle the refresh and keep the existing periodic poll as the backstop; the side effect also moves out of the state updater to keep it pure. Left untouched: per-entity invalidations that fire once per save on the entity being edited (one-shot, single entity, no burst), and the many mutation onSuccess invalidations that are already correctly scoped.
…rence data Reduce the steady stream of background requests an open tab makes when nothing is happening. - SSE anti-buffering: the entity and task-progress streams now send X-Accel-Buffering: no so reverse proxies don't buffer the response and hold back the 15s keep-alive. Buffered keep-alives let an idle-timeout close the connection, forcing the client to reconnect every minute, which in turn re-runs refetch-on-reconnect across the whole app. - Adaptive task polling: the shared task-progress poller now backs off to a slow cadence when no task is running and only polls quickly while work is in flight, instead of polling at the fast cadence forever. SSE still announces new work instantly, which flips it back to the fast cadence. Polling survives fetch errors and re-arms correctly. - Reference data caching: all tags / genres "fetch every page" lookups in the filter panels and metadata editors move to a shared hook with a long staleTime. These lists change only on an explicit edit (which already invalidates them), so they no longer re-run the multi-page sweep on every remount or reconnect. Note: tags/genres remain keyed as before so existing invalidations and all consumers still dedupe to one query.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Makes both database backends resilient under concurrent load so no single request can drain the connection pool and background work cannot starve interactive requests. Per-request query fan-out is now bounded, background subsystems run on their own pool when co-located with the API, and the new behavior is tunable per backend via config. Frontend query defaults are relaxed to stop multi-tab refetch storms.
Motivation
Under heavy multi-tab use, a single SQLite-backed server would "lose it": logs filled with 10+ second connection-acquire warnings on list endpoints. A single list request was issuing many queries at once, each taking its own pool connection, so a couple of concurrent requests could exhaust a small pool. In single-process deployments this was compounded by API traffic, task workers, the scheduler, the scanner, and pollers all competing for the same connections. The goal is resilience under load spikes and many tabs, not turning SQLite into a horizontally scalable backend (PostgreSQL remains the documented scaling path).
Changes
batch_fan_outandbackground_max_connections(withCODEX_DATABASE_{SQLITE,POSTGRES}_*env overrides); defaults are conservative on SQLite and higher on PostgreSQL to preserve parallel query latency. The SQLitemax_connectionsdefault is raised from 16 to 64 for headroom.Notes