fix(api): stop synchronous route work from stalling the whole server

[Pfannkuchensack]

Summary

Kind: fix + perf (backend, frontend, dependencies)

Users reported the backend becoming unresponsive for minutes at a time. The cause was not one bug but a chain, and reproducing it needed three conditions at once — a large library, an active gallery search, and a running generation — which is why it resisted diagnosis.

The mechanism. The gallery list and name routes were declared async def while calling synchronous SQLite services. That work therefore ran on the event loop, so for its entire duration the process served no other HTTP request and delivered no socket.io event. Users experienced this as the whole application freezing mid-generation rather than as a slow gallery. The same defect was present in 167 further route handlers.

Measured on a seeded 200k-image, 1.7 GB database (1.56 GB of it metadata blobs). The metric is the latency of an unrelated trivial request issued while a gallery name query is in flight — the queries themselves are not made faster, the loop is freed:

Scenario	before	after
name query with search term	1662 ms	4 ms
name query without search term	2298 ms	375 ms

Sample counts tell the same story more plainly: in the two-second window the probe returned 1 response before and 28 after.

What each commit does:

• fix(api): run gallery and search routes off the event loop — eight gallery/search routes and the five auth dependencies declared def so FastAPI dispatches them to the threadpool.
• perf(gallery): add a flat item-names endpoint — the name list wrapped every entry in an object carrying a kind discriminator. Building those models cost 820 ms of the 2225 ms service call at 200k items, and every consumer discarded the field, re-deriving the kind from the file extension. A new GET /v1/gallery/item_names returns a flat list; an optional created_date filter subsumes the separate by-date virtual-board route, so regular boards and virtual dates now share one endpoint, one cache and one query-args selector instead of a skipToken branch duplicated in four places. Result: 2.51 s → 1.57 s, 8.48 MB → 3.85 MB. Five legacy name endpoints keep working and are marked deprecated=True — external integrations still call them.
• perf(api): stop gzipping responses that are already compressed — Starlette's GZipMiddleware compresses every content type except text/event-stream. A 3 MB PNG cost 52 ms of event-loop time to gzip and came back at 3.01 MB, larger than it went in; a 12 MB PNG cost 210 ms. With auto-switch on, that lands after every generated image. Lowering compresslevel is not an alternative — on incompressible input level 1 costs 51 ms against level 9's 52 ms.
• feat(queue): add lightweight item summaries endpoint — from @JPPhoto's optimize-queue-return-data, adopted with the route declared def. The queue list fetched full SessionQueueItem objects carrying the complete GraphExecutionState: measured against a real 65-item queue database, 50 items cost 56 ms to parse and 9 ms to re-serialize — ~66 ms of event-loop time plus ~1.8 MB per request, for fields the list never renders. It also replaces an N+1 (one query per requested id, each taking the process-wide database lock).
• build: unpin FastAPI and move to 0.141.1 — see Merge Plan.
• perf(api): run every synchronous route handler off the event loop — the remaining 167 handlers, identified by AST rather than by hand.

Related Issues / Discussions

• Adopts the queue-summaries work from @JPPhoto's optimize-queue-return-data branch.
• The auth-guard repair protects the invariant introduced after GH [Security] InvokeAI: unauthenticated arbitrary-directory enumeration via /models/scan_folder in multi-user mode #9365.
• Touches gallery_default.py, which PR perf(db): use a covering index and anti-join for the gallery image names query #9385 also rewrites — see Merge Plan.

QA Instructions

Automated. Two new guards, both verified to fail before the fix and pass after:

• tests/app/routers/test_no_blocking_async_routes.py parses every router module and fails if any handler is async def without awaiting. Reintroduce async on any handler to see it name the offender.
• tests/app/routers/test_event_loop_blocking.py stubs a service call to block for one second and asserts an unrelated route still answers during it. Six routes covered, GET and POST.

Plus tests/app/routers/test_gallery_item_names.py (7 tests, two of which compare the new endpoint against the deprecated one so ordering and counts cannot drift while both are served) and tests/app/api/test_gzip_content_types.py (14 tests, including one asserting the real app has the middleware wired).

Full run: 2157 backend tests, 1707 frontend tests, ruff / tsc / eslint / knip / dpdm / prettier clean. Nine pre-existing failures in test_download_queue, test_model_install and test_load_api are network-dependent and reproduce identically on an unmodified tree.

Manual, to see the effect. Needs a large library — a few hundred MB of image metadata is enough; the effect scales with SUM(LENGTH(metadata)) FROM images.

1. Start a batch of ~10 images with auto-switch enabled.
2. While it runs, type a term into the gallery search box.
3. Before: the UI freezes — progress bar stops, thumbnails stop loading, the queue stops updating. After: the search itself is still slow, but everything else keeps responding.

To measure rather than eyeball it: fire GET /api/v1/gallery/item_names?search_term=… and poll GET /api/v1/app/version concurrently, recording the latency of the second. Note that a benchmark of the search endpoint alone shows no improvement — that is the wrong instrument here.

Deprecated routes. GET /v1/gallery/items/names, /v1/images/names, /v1/videos/names and both /v1/virtual_boards/by_date/{date}/*_names still return their original shapes; only the OpenAPI deprecated flag changed.

Media responses no longer carry Content-Encoding: gzip. Confirm images and videos still load, including behind a reverse proxy.

Merge Plan

The FastAPI bump needs attention. pyproject.toml moves from fastapi==0.118.3 to >=0.141.1,<0.142; contributors must re-sync (uv sync) after pulling. The old pin carried a comment guessing the OpenAPI crash on 0.119 was "probably Invoke's [bug], because we are doing something unusual with AnyInvocation". It was not: fastapi/_compat/v2.py assumed every field mapping carries a $ref. Upstream fixed it in 0.124.0 with no change needed here.

Two later FastAPI changes break silently and are handled — both are worth a reviewer's attention:

• 0.130 emits contentMediaType instead of format: binary for file uploads. typegen.js mapped only the latter to Blob, so upload call sites would have started typing their File argument as string. Caught only because tsc happened to fail.
• 0.141 keeps an included router as a single node in app.routes instead of copying its routes into it. The default-deny auth guard walked app.routes for APIRoute instances and found 2 of 197 — passing while inspecting almost nothing. It now walks iter_route_contexts (the traversal FastAPI's own OpenAPI generation uses) and asserts a floor on the route count so going blind fails loudly. Only the allowlist-staleness assertion caught this; "fixing" it by trimming PUBLIC_ROUTES would have killed the guard.

Conflicts with PR #9385, which rewrites _build_half in gallery_default.py. This PR adds a shared _query_name_rows in the same file. Whichever merges second needs a manual pass. Unrelated note for that PR's own review: it introduces INDEXED BY hints into the shared query builder, which is SQLite-only syntax.

Not in scope, deliberately: the metadata LIKE '%…%' full scan (six sites, unindexable by construction) is being addressed differently in v7; the SQLite single-connection/global-lock design is tracked separately; making compresslevel configurable is written up with measurements but not implemented — level 9 costs 5.5× the CPU of level 1 for 0.4 percentage points of output size.

Suggested split: if the FastAPI commit would rather be reviewed on its own, it is self-contained and the auth-guard finding deserves its own title.

Checklist

• The PR has a short but descriptive title, suitable for a changelog
• Tests added / updated (if applicable)
• ❗Changes to a redux slice have a corresponding migration — n/a, no slice shape changed
• Documentation added / updated (if applicable) — docs/contributing/blocking-work-in-api-routes
• Updated What's New copy (if doing a release after this PR)

[keturn]

I was just bumbling around yesterday trying to figure out Invoke's approach to concurrency, as I realized the FastAPI handlers are async but most of our code (including the BaseInvocation API) very much is not.

perf(api): run every synchronous route handler off the event loop

Wait, what? This is just removing "async" from 167 existing "async def"? Is that…?

Okay, the explanation for how this works is at FastAPI, Concurrency and async / await: FastAPI automatically kicks any non-async function to a thread pool.

In an async-first application, "parse every router module and fail if any handler is async def without awaiting" is really not the heuristic you want to use. You want to be async by default and only introduce the complexity of thread switching if you're doing a blocking operation. Especially in Python, where threads don't actually get you multi-core parallel execution.

The aforementioned FastAPI docs back me up on that:

In these cases, it's better to use async def unless your path operation functions use code that performs blocking I/O.

However… in a codebase where most of the code is written synchronously and people aren't used to thinking about whether they're about to call a blocking function?

As a former Twisted developer, it hurts me to even think it, but no-async-by-default might be the right call, I guess?

We're not trying to optimize requests/second throughput, as the number of users and frequency of requests on any one InvokeAI sever is actually pretty low. We're trying to reduce the chances of someone accidentally making a commit that blocks the server's event loop.

So. I can't say I'm a fan, but I guess I understand why you might want to do it that way.

I still feel tempted to argue for some kind of "it doesn't use await but that doesn't mean it's blocking" escape hatch, but I'm not confident I could really back it up with benchmarks. Probably the only high-volume route we have is whatever gets used for serving thumbnails, and hopefully that's backed by some StaticFile handler. (I guess that might not be true if that backend is pluggable to S3 storage or whatever.)

[keturn]

I guess the hazard of "kick it to the thread pool by default" is then all your code has to be thread-safe by default. Which I don't think is an easier/safer assumption to make than knowing if your code is blocking.

i.e. does sending all sqlite-related activity to a general-purpose thread-pool mean we have multiple threads opening and writing to the same sqlite database at once? Is that a thing we can assume it's safe to do?

[Pfannkuchensack]

i.e. does sending all sqlite-related activity to a general-purpose thread-pool mean we have multiple threads opening and writing to the same sqlite database at once? Is that a thing we can assume it's safe to do?

Worth answering precisely, because the premise is slightly off in a way that matters.

We never open a connection per thread. There is exactly one sqlite3.Connection, created once at
startup. Every runtime access goes through SqliteDatabase.transaction(), which holds a process-wide
threading.RLock for the duration. The only code touching _conn outside it is startup-only (PRAGMAs,
VACUUM, the migrator), before the server accepts requests. So access is serialized by the lock, not
merely permitted by SQLite.

And this predates the PR. Multiple threads already write to that database on every generation:
SessionProcessor runs _process in a thread per worker — several concurrently with multi-GPU — calling
set_queue_item_session / complete_queue_item; DownloadQueueService and ModelInstallService have
their own threads; SlidingWindowTokenMiddleware already did its user lookup via run_in_threadpool.
The RLock around a single connection is the design that makes that work, and it has been carrying that
load for a long time. This PR doesn't introduce multi-threaded DB access — it stops the request layer
from pretending it's exempt.

What does change, and I don't want to gloss over it: two async def handlers that never awaited were
implicitly serialized against each other on the loop. As def they can genuinely overlap. Each
transaction() is still atomic; a handler spanning two transactions is not — but it never was, since
the worker threads could always interleave between them. What's gone is atomicity relative to other
handlers, which was an accident of the dispatch model rather than something designed for. (I checked
for module-level mutable state in the routers, which is what loop-serialization would have protected:
there is none.) So the residual risk is a handler doing multi-step read-modify-write that relied on no
other handler running in between — if you know of one, worth flagging.

On the general point, I'd argue the two assumptions aren't symmetric. "Does this handler block?" is
mechanically checkable, and the PR adds test_no_blocking_async_routes.py which parses every router and
fails if a handler is async def without awaiting — before that existed, the assumption was violated 167
times unnoticed, and the cost was the server going unresponsive for minutes. "Is this thread-safe?" isn't
checkable, but it was already required by the threads above. The PR doesn't add that requirement; it
removes a false belief that the request layer was exempt from it.

[Pfannkuchensack]

I was just bumbling around yesterday trying to figure out Invoke's approach to concurrency, as I realized the FastAPI handlers are async but most of our code (including the BaseInvocation API) very much is not.

perf(api): run every synchronous route handler off the event loop

Wait, what? This is just removing "async" from 167 existing "async def"? Is that…?

Okay, the explanation for how this works is at FastAPI, Concurrency and async / await: FastAPI automatically kicks any non-async function to a thread pool.

In an async-first application, "parse every router module and fail if any handler is async def without awaiting" is really not the heuristic you want to use. You want to be async by default and only introduce the complexity of thread switching if you're doing a blocking operation. Especially in Python, where threads don't actually get you multi-core parallel execution.

The aforementioned FastAPI docs back me up on that:

In these cases, it's better to use async def unless your path operation functions use code that performs blocking I/O.

However… in a codebase where most of the code is written synchronously and people aren't used to thinking about whether they're about to call a blocking function?

As a former Twisted developer, it hurts me to even think it, but no-async-by-default might be the right call, I guess?

We're not trying to optimize requests/second throughput, as the number of users and frequency of requests on any one InvokeAI sever is actually pretty low. We're trying to reduce the chances of someone accidentally making a commit that blocks the server's event loop.

So. I can't say I'm a fan, but I guess I understand why you might want to do it that way.

I still feel tempted to argue for some kind of "it doesn't use await but that doesn't mean it's blocking" escape hatch, but I'm not confident I could really back it up with benchmarks. Probably the only high-volume route we have is whatever gets used for serving thumbnails, and hopefully that's backed by some StaticFile handler. (I guess that might not be true if that backend is pluggable to S3 storage or whatever.)

Fair challenge, and the FastAPI docs quote is right — but note its condition: "unless your path
operation functions use code that performs blocking I/O." That's not an edge case here, it's
essentially every handler. The rule lands on def for us by its own terms rather than in spite of them.

Two things I could measure rather than argue:

The thread hop costs ~156 µs. Trivial handler, in-process, 600 requests: 0.320 ms as async def,
0.477 ms as def. Real, but it buys back multi-second event-loop stalls — the gallery name query was
holding the loop for 1.7–2.3 s per request before this.

The high-volume route you're hoping is a StaticFile isn't one. get_image_thumbnail is a regular
handler doing open(path, "rb"); StaticFiles only serves the UI bundle and /static. So the hottest
route in the app is precisely one that performs blocking I/O — and as you guessed, with a pluggable S3
backend it becomes blocking network I/O, which is worse. That's the one route where I'd most want it
off the loop.

One clarification on the GIL point: for pure-Python CPU work you're right that threads buy nothing. But
the calls we're moving release it — sqlite3.execute drops the GIL for the duration of the query, as
does file I/O. So for this particular workload the threads do overlap rather than just take turns.

On the escape hatch — you're right that "doesn't await" ≠ "doesn't block", and the guard is deliberately
blunt. I'd happily make it an allowlist with a justification per entry, the same shape as PUBLIC_ROUTES
in test_model_manager_authorization.py: async def allowed, but only as a conscious, reviewed
exception rather than the default. That keeps the property you want (async where it genuinely helps)
without going back to a rule nobody can check. If that sounds reasonable I'll add it — I just didn't
want to build the exception mechanism before there was a concrete handler that needed it.