v0.6.0 — Phase B: Admission Bounding, Batch Queueing, Multi-Model Process Split

InferenceX v0.6.0 — Phase B: Admission Bounding, Batch Queueing, Multi-Model Process Split

What changed

Bounded per-model admission wait (B4, rescope-admission-control)

Admission control used to reject a request outright the moment a model hit its max_num_seqs ceiling, even when vLLM's own scheduler would have queued and served it a moment later. A live test against vLLM 0.22.1 confirmed it: 48 requests fired at once, four times over the ceiling, and every single one completed with no errors, worst case just over a second. vLLM was never going to reject these. The admission gate was.

So Gate 1 now waits instead of rejecting outright. It's a per-model asyncio.BoundedSemaphore, acquired with a timeout (INFERENCE_X_ADMISSION_WAIT_S, five seconds by default). If a slot opens before the timeout, the request goes through exactly as before. If it doesn't, the client still gets the same 429 and Retry-After they'd have gotten instantly under the old behavior. Nothing about the response contract changed, only how long it's willing to wait first.

Batch priority queueing (B5, add-batch-priority-queueing)

B4's wait applied the same short timeout to every request regardless of priority. That's fine for interactive traffic but wrong for anything running as a batch, where a five-second wait is far too short and a hard failure is worse than waiting. Batch requests now get their own timeout, thirty seconds by default (INFERENCE_X_BATCH_ADMISSION_WAIT_S), on the same semaphore, same FIFO order. Interactive requests are untouched by this change; their code path never runs the new logic.

There's also a cap now on how many batch requests can be queued per model at once (INFERENCE_X_MAX_QUEUED_BATCH_MULTIPLIER, default 8x the model's max_num_seqs). Past that cap, a batch request gets rejected immediately rather than piling onto an already-long queue. That cap is what makes the longer wait safe to ship without an unbounded queue depth as the tradeoff.

One model per OS process (B6, split-multi-model-serving)

The old EnginePool could load more than one model into a single process, which is how the playground's compare mode worked. It came at a real cost: every model sharing that process ran with enforce_eager forced on (no CUDA graphs), max_model_len silently clamped to 2048, and gpu_memory_utilization sized by heuristics tuned against past failures rather than measured VRAM. All of that applied for the whole session, not just while two models were actually being compared.

INFERENCE_X_LOADED_MODELS resolving to more than one distinct model is now a startup error. Each server process serves exactly one model. Comparing two models means running two processes, and the playground now does this automatically, one process per model, each on its own port. playground/client.py's dual-process compare mode already worked this way and needed no changes.

Fixing this turned up a second bug: the free-VRAM probe itself was wrong. It read torch.cuda.mem_get_info(), which only reflects what the calling process's own CUDA context can see. That's stale the moment a sibling process is also holding VRAM. The probe now shells out to nvidia-smi first, which reports the actual device-wide free memory, and falls back to the old method only when nvidia-smi isn't available (CI, no GPU).

Also

• docs/DECISIONS.md gains DEC-059, DEC-060, and DEC-061 for B6, B4, and B5 respectively.
• docs/PHASE-A-ARCHITECTURE.md §10 marks B4, B5, and B6 complete.
• All three OpenSpec changes are archived under openspec/changes/archive/2026-08-10-*.

---

Test coverage

551 unit tests, 1 xfailed (unchanged, DEC-051 N1, still correctly deferred to Phase C3). Up from 528 at v0.5.0.

---

Upgrade notes

The breaking change here is B6: if you were setting INFERENCE_X_LOADED_MODELS to more than one model to load them into a single process, that now fails at startup with a clear error instead of silently doing it. Run one process per model instead — make playground and make playground-compare already do this for you, and playground/README.md documents the manual pattern if you're not going through those.

Three new environment variables, all optional with sane defaults:

• INFERENCE_X_ADMISSION_WAIT_S (default 5) — interactive admission wait bound
• INFERENCE_X_BATCH_ADMISSION_WAIT_S (default 30) — batch admission wait bound
• INFERENCE_X_MAX_QUEUED_BATCH_MULTIPLIER (default 8) — batch queue depth cap, as a multiple of the model's max_num_seqs

Nothing else changes shape. Existing clients that never set priority: batch see no behavior difference beyond the (usually beneficial) admission-wait change in B4.

---

Known limitations

• admission_wait_s's default was calibrated against opt-125m, a small fast model. It hasn't been validated against a larger, slower model — a live smoke test against one is a named, non-blocking follow-up.
• B5's queue-depth cap and wait-time defaults are sanity-checked against 9-16 concurrent requests in testing, not against the ~200-request scale the downstream batch-evaluation use case actually runs at.
• Crash isolation between sibling engine processes under B6's process-per-model model is untested — what happens to process A if process B's engine core crashes hasn't been verified either way.
• The original root cause of the free-VRAM probe's staleness on this platform (why torch.cuda.mem_get_info() behaves this way here specifically) remains unidentified; the fix works regardless, but the underlying platform behavior wasn't chased down.
• Authentication is still not implemented (local-only, DEC-DEFER-01).
• Batch-invariant / cross-run determinism is still out of scope until Phase C3 (DEC-051 N1).

Full Changelog: v0.5.0...v0.6.0