fix(gateway): serialize local-model calls — fix oMLX "model is busy" chat failures

[SecureCloudGroup]

The bug (from your screenshot + logs)

Chat failed with "couldn't reach the model" then, on retry, oMLX's "Model gemma-4… is busy; cannot reload runtime settings variant until active requests finish". Root cause: a local model server (Ollama/MLX/oMLX) serves one request at a time and errors on any overlap, but the app sent it concurrent requests:

• a big local model (gemma-4 26B) exceeded the 60s interactive timeout, so the app abandoned the request while oMLX kept running it → the retry collided with it;
• the background auto-reindex embeds via bge-m3 on the same server every tick, which can collide with a foreground chat.

The fix

• Serialize every local-provider gateway call through a process-wide BoundedSemaphore(1) so oMLX never sees overlap. Cloud providers are exempt (they parallelize fine).
  • A Semaphore, deliberately — not a Lock/RLock. The streaming path holds it across the SSE generator, which Starlette drives across different threadpool workers; a Lock/RLock raises RuntimeError on the cross-thread release and would wedge the lock forever. A Semaphore's release isn't owner-bound.
• Background auto-reindex yields: a non-blocking local_available() peek skips the tick while a chat holds the model, so your chat is never delayed behind a backfill.
• Interactive timeout 60s → 180s: gemma-4 26B can take over a minute to cold-load + generate; abandoning it early caused the retry collision. Cloud models never approach this.
• chat_stream now surfaces a stream timeout as a clean 504 ("model may still be loading") instead of "gateway unreachable".

Rigor (this one bit me, so I over-verified)

Two adversarial review passes. The first reproduced a critical bug in my initial RLock version — cross-thread release → RuntimeError → permanently wedged local-model subsystem. I switched to a Semaphore and re-reviewed: 0 findings (checked for the non-re-entrant deadlock risk, double-release, leaks, and residual collisions). Tests cover: local serialization (1 concurrent local call vs. many cloud), the cross-thread release the old RLock couldn't do (direct regression test), local_available gating, and the interactive-timeout wiring. Full backend suite green (656), ruff clean.

Known, accepted edge (bounded, correct-result)

If you fire a chat at the exact moment a scheduled turn is mid-generation on the same local model, the chat waits for it (up to the scheduled turn's ~300s ceiling) rather than colliding — a correct-but-slower result. Rare for a single user; I can add a "waiting…" stream heartbeat if you ever hit it.