v0.6.0 — Phase B: Admission Bounding, Batch Queueing, Multi-Model Process Split

InferenceX v0.6.0 — Phase B: Admission Bounding, Batch Queueing, Multi-Model Process Split

What changed

Bounded per-model admission wait (B4, rescope-admission-control)

Admission control used to reject a request outright the moment a model hit its max_num_seqs ceiling, even when vLLM's own scheduler would have queued and served it a moment later. A live test against vLLM 0.22.1 confirmed it: 48 requests fired at once, four times over the ceiling, and every single one completed with no errors, worst case just over a second. vLLM was never going to reject these. The admission gate was.

So Gate 1 now waits instead of rejecting outright. It's a per-model asyncio.BoundedSemaphore, acquired with a timeout (INFERENCE_X_ADMISSION_WAIT_S, five seconds by default). If a slot opens before the timeout, the request goes through exactly as before. If it doesn't, the client still gets the same 429 and Retry-After they'd have gotten instantly under the old behavior. Nothing about the response contract changed, only how long it's willing to wait first.

Batch priority queueing (B5, add-batch-priority-queueing)

B4's wait applied the same short timeout to every request regardless of priority. That's fine for interactive traffic but wrong for anything running as a batch, where a five-second wait is far too short and a hard failure is worse than waiting. Batch requests now get their own timeout, thirty seconds by default (INFERENCE_X_BATCH_ADMISSION_WAIT_S), on the same semaphore, same FIFO order. Interactive requests are untouched by this change; their code path never runs the new logic.

There's also a cap now on how many batch requests can be queued per model at once (INFERENCE_X_MAX_QUEUED_BATCH_MULTIPLIER, default 8x the model's max_num_seqs). Past that cap, a batch request gets rejected immediately rather than piling onto an already-long queue. That cap is what makes the longer wait safe to ship without an unbounded queue depth as the tradeoff.

One model per OS process (B6, split-multi-model-serving)

The old EnginePool could load more than one model into a single process, which is how the playground's compare mode worked. It came at a real cost: every model sharing that process ran with enforce_eager forced on (no CUDA graphs), max_model_len silently clamped to 2048, and gpu_memory_utilization sized by heuristics tuned against past failures rather than measured VRAM. All of that applied for the whole session, not just while two models were actually being compared.

INFERENCE_X_LOADED_MODELS resolving to more than one distinct model is now a startup error. Each server process serves exactly one model. Comparing two models means running two processes, and the playground now does this automatically, one process per model, each on its own port. playground/client.py's dual-process compare mode already worked this way and needed no changes.

Fixing this turned up a second bug: the free-VRAM probe itself was wrong. It read torch.cuda.mem_get_info(), which only reflects what the calling process's own CUDA context can see. That's stale the moment a sibling process is also holding VRAM. The probe now shells out to nvidia-smi first, which reports the actual device-wide free memory, and falls back to the old method only when nvidia-smi isn't available (CI, no GPU).

Also

• docs/DECISIONS.md gains DEC-059, DEC-060, and DEC-061 for B6, B4, and B5 respectively.
• docs/PHASE-A-ARCHITECTURE.md §10 marks B4, B5, and B6 complete.
• All three OpenSpec changes are archived under openspec/changes/archive/2026-08-10-*.

---

Test coverage

551 unit tests, 1 xfailed (unchanged, DEC-051 N1, still correctly deferred to Phase C3). Up from 528 at v0.5.0.

---

Upgrade notes

The breaking change here is B6: if you were setting INFERENCE_X_LOADED_MODELS to more than one model to load them into a single process, that now fails at startup with a clear error instead of silently doing it. Run one process per model instead — make playground and make playground-compare already do this for you, and playground/README.md documents the manual pattern if you're not going through those.

Three new environment variables, all optional with sane defaults:

• INFERENCE_X_ADMISSION_WAIT_S (default 5) — interactive admission wait bound
• INFERENCE_X_BATCH_ADMISSION_WAIT_S (default 30) — batch admission wait bound
• INFERENCE_X_MAX_QUEUED_BATCH_MULTIPLIER (default 8) — batch queue depth cap, as a multiple of the model's max_num_seqs

Nothing else changes shape. Existing clients that never set priority: batch see no behavior difference beyond the (usually beneficial) admission-wait change in B4.

---

Known limitations

• admission_wait_s's default was calibrated against opt-125m, a small fast model. It hasn't been validated against a larger, slower model — a live smoke test against one is a named, non-blocking follow-up.
• B5's queue-depth cap and wait-time defaults are sanity-checked against 9-16 concurrent requests in testing, not against the ~200-request scale the downstream batch-evaluation use case actually runs at.
• Crash isolation between sibling engine processes under B6's process-per-model model is untested — what happens to process A if process B's engine core crashes hasn't been verified either way.
• The original root cause of the free-VRAM probe's staleness on this platform (why torch.cuda.mem_get_info() behaves this way here specifically) remains unidentified; the fix works regardless, but the underlying platform behavior wasn't chased down.
• Authentication is still not implemented (local-only, DEC-DEFER-01).
• Batch-invariant / cross-run determinism is still out of scope until Phase C3 (DEC-051 N1).

Full Changelog: v0.5.0...v0.6.0

v0.5.0 — Per-Request Engine Timing

InferenceX v0.5.0 — Per-Request Engine Timing

What changed

Per-request engine timing (B3, expose-per-request-engine-timing)

Chat completions now optionally carry an engine-sourced timing object on the non-streaming response and on the streaming terminal event: queue_time_ms, prefill_time_ms, decode_time_ms, and inference_time_ms. These intervals are derived from the engine's own internal request-state clock (a single clock domain), not from InferenceX wall-clock measurements at the HTTP boundary. That gives clients a decomposition they cannot reconstruct themselves — when the scheduler admitted the request versus how long prefill and decode actually took. timing is omitted (null) when the engine cannot supply metrics for a request; InferenceX never invents estimates. Streaming and non-streaming share the same derivation path through derive_terminal_metadata, so both surfaces stay consistent. Process-wide GET /metrics (B2) and HTTP-boundary GET /v1/metrics are unchanged — this release adds a third, per-response surface rather than merging the existing two.

On the streaming path, timing rides the same opt-in as usage rather than introducing a second flag: it appears only on the terminal event, and only when the client sets stream_options: {"include_usage": true}. A streaming client that doesn't set that flag sees no timing — the same absence-by-default posture usage already established in v0.3.0.

Also

• docs/PHASE-A-ARCHITECTURE.md §10 marks B3 complete.
• Adds the cut-release skill documenting this repo's semver-driven release cut pattern.

---

Test coverage

528 unit tests, 1 xfailed (DEC-051 N1, correctly deferred to Phase C3). (+9 from v0.3.0)

---

Upgrade notes

No config migration required. timing is a strictly additive, optional response field — existing clients that don't read it are unaffected, and no existing field changed type or became required.

If you call the API directly and want per-request timing on a streamed response, opt in explicitly (the same flag that already gates usage):

{"stream": true, "stream_options": {"include_usage": true}}

make chat, make playground, make playground-compare, make client, make benchmark, and make advise all run unchanged.

---

Known limitations

• Authentication not implemented (local-only — DEC-DEFER-01)
• Rate limiting/backpressure: admission control rejects batch-tier requests with 429 under KV pressure, but no global request queue or concurrency cap exists outside per-model KV/seq gates (DEC-DEFER-02) — under architectural review as part of the upcoming B4 admission-control re-scope
• Batch-invariant / cross-run determinism is out of scope until Phase C3 (DEC-051 N1)
• BenchmarkResult.peak_vram_delta_gb and AdvisorResult.vram_gb remain as deprecated aliases; removing either requires a future ADR (DEC-057)
• runner._check_vram_budget compares measured occupancy against an analytic estimate — a known, deliberately unfixed coupling (DEC-057 §6)
• The Engine Boundary factory (engines/registry.create_engine) remains unbuilt — deferred, not blocking, per DEC-047 §5

Full Changelog: v0.4.0...v0.5.0

v0.4.0 — AsyncLLM Engine Migration & Native Prometheus Metrics

InferenceX v0.4.0 — AsyncLLM Engine Migration & Native Prometheus Metrics

What changed

AsyncLLM engine migration (B1, migrate-async-llm-engine)

VLLMEngine now runs on vLLM's AsyncLLM async engine client instead of the offline LLM class wrapped by a hand-rolled EngineDriver. Client disconnect and request timeout now trigger a real engine-side abort instead of the previous local give-up that left computation still running on the GPU (DEC-058). generate() and generate_stream() both derive from a single _stream_chunks() call site into AsyncLLM.generate(), preserving the single-source derive_terminal_metadatainvariant.is_healthy()and_log_kv_cache_stats()readAsyncLLM's flatter attribute surface directly, with no llm_engineindirection.EngineDriverand its pool-step lock are removed entirely;pool_size > 1` remains unguaranteed and unforbidden, unchanged from before.

Native engine metrics (B2, expose-native-engine-metrics)

GET /metrics is now mounted on the FastAPI app, backed directly by vLLM's own PrometheusStatLogger registry — verified empirically (not inferred) to be the default global prometheus_client.REGISTRY. This gives scrapeable visibility into vLLM's execution-internal state (KV cache utilization, running/waiting queue depth, per-phase timing) without InferenceX reimplementing any of it. Zero vLLM-specific imports land in api/, keeping the Engine Boundary (DEC-047) clean. /metrics is excluded from ObservabilityMiddleware's recording path, so scrape traffic never enters InMemoryStorage or skews GET /v1/metrics's avg_latency_ms / total_requests aggregates — the two metrics surfaces answer different questions (HTTP-boundary vs. engine-internal) and are not merged. prometheus-client is now a direct dependency rather than a transitive one. A startup WARNING is logged when pool_size > 1, naming the metric-label collision risk of multiple AsyncLLM instances' default stat loggers (recorded, not guarded against — multi-engine serving redesign is future scope).

Also

• docs/DECISIONS.md gains DEC-058, documenting the AsyncLLM migration's rationale.
• docs/PHASE-A-ARCHITECTURE.md §10 marks both B1 and B2 complete.

Full Changelog: v0.3.0...v0.4.0

v0.3.0 — Truthful Streaming Metrics, Deterministic Seeds & Effective-Request Transparency

InferenceX v0.3.0

Feature release. Closes out Phase A: streamed metrics stop being estimated, requests can pin a sampling seed, every response reports exactly what the server actually ran, and the benchmark/advisor pipeline stops reporting numbers it can't back up. Also fixes a real concurrency-capacity leak and a type-unsound engine contract found during a full pre-Phase-B architecture audit.
No API breaking changes — all new request/response fields are optional/additive with backward-compatible defaults.

---

What changed

Truthful streamed token usage (OS-2)

BaseEngine.generate_stream previously yielded bare str, so streamed completions were metered by counting whitespace-delimited words — always wrong, and wrong by an unknowable, tokenizer-dependent amount. generate_stream now yields a typed ChatStreamChunk (content / finish_reason / usage) carrying real engine-accounted usage. Set stream_options: {"include_usage": true} to get a terminal usage chunk before data: [DONE]. Every streamed token count produced before this release is superseded — pre-existing figures are not comparable and should not be mixed with new ones (see Upgrade notes).

Deterministic sampling seed (OS-3)

ChatCompletionRequest gained an optional seed: int | None field. When set, it's forwarded unchanged to vLLM's SamplingParams on both the streaming and non-streaming paths — no clamping, no -1 normalization. This does not claim end-to-end determinism (batch composition, hardware, and backend version can all still change output); it only guarantees the seed you send is the seed the sampler receives.

Effective Request transparency & strict mode (OS-4)

Every response — streamed and non-streamed — now carries:

• resolved — the request the server actually executed (post-clamping), so a client never has to guess what ran.
• warnings — every substitution the server made (type: "substituted") and every check it couldn't perform (type: "degraded"), from a closed, versioned code registry.
• strict: bool (request field, default false) — when true, converts any substitution the server would have made into a 400 rejection instead. It never changes what gets substituted, and it never rejects on a degraded warning (that would turn admission's fail-open posture fail-closed).

Benchmark suite identity (OS-5)

suite_version is now a verified SHA-256 hash of the prompt corpus, computed and checked by one shared canonicalizer (benchmarks/suite_identity.py) instead of being a stored literal nothing validated. A missing or mismatched hash fails loudly and names make suite-version instead of silently proceeding. ResultStore now filters by suite version before selecting the latest result per model, so the advisor can no longer rank across incompatible benchmark runs.

Honest advisor scoring & canonical VRAM field (OS-6)

• Removed quant_score, a constant 1.0 placeholder that inflated every model's score identically without discriminating on anything real. The advisor now scores only measured quantities — throughput, warm TTFT, VRAM headroom — reweighted to exact fractions (4/9, 1/3, 2/9) preserving the prior ratio.
• viable is now the sole viability signal; a gated model's score can no longer be confused with a merely low-scoring one.
• Renamed peak_vram_delta_gb → vram_device_occupied_gib on BenchmarkResult (it always measured device-wide occupancy, never a per-process delta — only the name was wrong). Old field names remain readable as deprecated aliases; the canonical field wins on conflict.

Admission reservation leak fix (reliability)

ChatService.stream_response's admission reservation could leak on an early SSE client disconnect (closed tab, curl Ctrl-C, an LB read timeout) — the release path sat outside the guard covering the rest of the request lifecycle. Repeated disconnects permanently ate concurrency slots until the process restarted. The guard now covers the full lifecycle from admit() onward; regression tests assert both the KV and sequence trackers return to zero after every termination path (normal completion, timeout, exception, cancellation, and disconnect before/after first token).

Engine streaming contract type fix

BaseEngine.generate_stream was declared async def ... -> AsyncGenerator[...], which type-checks as a coroutine returning an async generator — not the async generator itself every real implementation actually is. Dropped the stray async, which cleared 4 previously-suppressed mypy errors and let services.chat_service come out of the DEC-048 baseline. Zero runtime behavior change — this only makes the declared type match what already ran.

Also

• Documented the repo's agent-tooling stack (rtk proxy for git mutations, context-mode for read-only gates, token-savior for symbol lookups) and formalized the branch-from-develop / PR-into-develop workflow in AGENTS.md, CONTRIBUTING.md, and a new CLAUDE.md.
• All prior OpenSpec changes (OS-1 through OS-6, plus two pre-existing unarchived bug-fix changes) are now archived, with every promoted requirement verified byte-identical to its source change.

---

Test coverage

519 unit tests, 1 xfailed (DEC-051 N1, correctly deferred to Phase C3). (+89 from v0.2.0)

---

Upgrade notes

No config migration required.

Streamed completion-token counts recorded before this release (including any pre-existing benchmarks/results/*.json entries) are superseded, not comparable, and should not be mixed with counts recorded after — they were produced by word-counting rather than engine accounting. Regenerate any benchmark result you rely on throughput figures for:

make benchmark MODEL=<name>

If you call the API directly and want streamed usage, opt in explicitly:

{"stream": true, "stream_options": {"include_usage": true}}

Default streaming clients that don't set this now report no token figure for that request (absence, not an estimate) — a deliberate loss of default coverage in exchange for correctness.

make chat, make playground, make playground-compare, make client, make benchmark, and make advise all run unchanged — no target was renamed or removed, and the API response schema only gained optional fields.

---

Known limitations

• Authentication not implemented (local-only — DEC-DEFER-01)
• Rate limiting/backpressure: admission control rejects batch-tier requests with 429 under KV pressure, but no global request queue or concurrency cap exists outside per-model KV/seq gates (DEC-DEFER-02)
• Batch-invariant / cross-run determinism is out of scope until Phase C3 (DEC-051 N1)
• BenchmarkResult.peak_vram_delta_gb and AdvisorResult.vram_gb remain as deprecated aliases; removing either requires a future ADR (DEC-057)
• runner._check_vram_budget compares measured occupancy against an analytic estimate — a known, deliberately unfixed coupling (DEC-057 §6)
• The Engine Boundary factory (engines/registry.create_engine) remains unbuilt — deferred, not blocking, per DEC-047 §5

v0.2.0 — Admission Control, VRAM Tiers & Model-Variant Routing

InferenceX v0.2.0

Feature release. Moves InferenceX from "one model, one GPU, best-case conditions" to actual capacity management — admission control, per-GPU-class VRAM tiers, automatic model-variant selection, and a shared engine driver thread that closes a real concurrency race. Also ships two significant VRAM / throughput bug fixes and a validated ~2B-parameter model on 8 GB hardware.
No API breaking changes — all new request fields are optional with backward-compatible defaults.

---

What changed

Admission control (pre-dispatch context/KV enforcement)

Every request now passes three independent gates before it reaches the GPU, instead of failing unpredictably inside vLLM:

• Sequence-concurrency ceiling — in-flight requests against a model are capped at its resolved max_num_seqs; no clamp path, a request either gets a slot or is rejected.
• Context length — prompt_tokens + requested_output_tokens is checked against the model's context window (min of max_model_len, the VRAM tier's cap, and the request's own max_context_tokens, if set).
• KV-pool pressure — a per-model in-memory counter tracks tokens reserved by in-flight requests against the engine's real post-load KV capacity.

New request field priority: "interactive" | "batch" (default "interactive"): interactive requests get max_tokens clamped to fit available budget when possible; batch requests are rejected with a 429 + Retry-After header instead of silently truncated. All three gates degrade
to "don't block" when the underlying number is unavailable — consistent with the project's fail-open posture for advisory signals.

VRAM tiers (config/vram_tiers.yaml)

Declarative per-GPU-class capacity envelopes — 6gb / 12gb / 24gb — covering gpu_memory_utilization ceiling, max_model_len cap, max_num_seqs, batching knobs, and prefix-caching. Resolved once at startup from probed VRAM and logged; the same models.yaml now behaves correctly across a 6 GB laptop card and a 24 GB workstation card without hand-tuning per machine.

Automatic model-variant selection

Models can now declare a shared family (e.g. qwen2.5-7b) grouping the same logical model at different precisions. Setting INFERENCE_X_LOADED_MODELS=qwen2.5-7b (or INFERENCE_X_DEFAULT_MODEL) picks whichever variant's estimated weight size fits the resolved VRAM tier's budget — full precision preferred, falling back to a quantized variant. Raises a clear error (rather than silently picking the first-listed variant) when nothing fits.

Shared engine driver thread (concurrency correctness fix)

Streaming and non-streaming requests previously ran independent loops calling vLLM's synchronous step(). Under real concurrent load, a request's terminal output could be delivered to a different in-flight request's caller — passed unit tests against a mocked engine, reproducibly lost output live. EngineDriver now owns each loaded model exclusively: exactly one thread calls add_request/step() and demultiplexes every output back to the correct caller by request ID. Fixed two follow-on issues surfaced while hardening this: a late-submission race during driver failure, and (see below) a throughput regression this same design accidentally introduced.

VRAM over-allocation and throughput regression (bug fixes)

Two independent, previously-undetected bugs, both surfaced by a routine opt-125m benchmark:

• gpu_memory_utilization: auto bypassed footprint-aware sizing entirely, grabbing a flat (free_vram − buffer) / total_vram fraction regardless of model size — a 125M-parameter model could claim >85% of an 8 GB GPU. Fixed by routing "auto" through the same footprint calculation (weights + KV cache + runtime overhead) used for explicit values.
• The new EngineDriver's idle-poll wait fired on every loop iteration, including while work was already queued, capping every model's step() rate to ~20 Hz regardless of GPU speed. Fixed to only wait when genuinely idle.

Verified live on an RTX 4060 (8 GB): opt-125m peak VRAM delta 7.06 GB → 1.65 GB; throughput 16.7 tok/s → 329.3 tok/s (19.7×); p50 latency 11,030 ms → 518 ms.

AWQ weight-estimator fix (unquantized embedding/lm_head)

estimate_weight_gib() applied the quantization bytes-per-parameter ratio uniformly to every parameter. AWQ/GPTQ-style quantization leaves embedding lookups and (when untied) the output projection at full precision — for a model with untied embeddings and a 150k-word vocabulary, that's ~2 GB the old estimate missed, causing a real load failure despite a "fits" estimate. Fixed by splitting those layers out and pricing them separately.

qwen1.5-1.8b — validated ~2B-class model entry

Added and live-benchmarked on 8 GB WSL2: 73.6 tok/s, 5.71 GB peak VRAM delta, sizing itself to ~70% GPU utilization automatically. First confirmation that a real, ungated, chat-tuned ~2B model runs comfortably on this hardware class after the fixes above.

make playground compare-mode fixes

A real two-model compare pair (qwen2.5-0.5b + qwen2.5-1.5b, ~7.3 GB combined on an 8 GB card) failed startup with a negative allowed gpu_memory_utilization — a fixed per-engine-transition VRAM reservation was double-counted on top of the next engine's already-inclusive footprint estimate. Recalibrated after live-verifying both engines load and serve together. Also fixed while investigating this failure:

• The failure banner truncated at a flat 200-character limit with no regard for word boundaries, cutting the actionable "Load fewer models..." clause off mid-sentence. Now truncates at the nearest word boundary (280-char limit).
• Ctrl+C did nothing once any screen (including the failure overlay) was on top — Textual's own Screen/ModalScreen classes claim plain Ctrl+C for text-copy, silently shadowing the app's non-priority quit binding. Fixed by marking the quit binding priority=True in both the compare playground and the chat CLI.

Dead code and speculative-flexibility removal

Removed unreferenced modules (config_loader.py, prompt_formatting.py), dead config files (config/routing.yaml, config/server.yaml) and their loader method, unused MetricsService methods, an unused compiled regex, a single-implementation router ABC, and a duplicated _is_wsl() definition. Merged two structurally-identical per-model counter classes in AdmissionController into one. Replaced a hand-rolled percentile function with statistics.quantiles().

Dependency security bump

pip-audit flagged 20 known vulnerabilities across 8 packages. Bumped aiohttp 3.14.0→3.14.1, cryptography 48.0.0→49.0.0, msgpack 1.1.2→1.2.1, pydantic-settings 2.14.1→2.14.2, starlette 1.2.1→1.3.1 — cuts findings to 7 across 3 packages (torch, vllm, diskcache left pinned: no fix version available yet, and bumping independently risks breaking vLLM's CUDA/torch compatibility matrix).

huggingface-cli → hf CLI migration

huggingface-cli is deprecated upstream. All documentation and error-message references updated to the hf CLI (hf download, hf auth login).

---

Test coverage

430 unit tests, 0 failures. (+131 from v0.1.2)

---

Upgrade notes

No config migration required. If you have a config/routing.yaml or config/server.yaml from before this release, they are no longer read and can be deleted.

If you're using multi-model pools on an 8 GB (or smaller) GPU, re-verify your INFERENCE_X_LOADED_MODELS combination after upgrading — the corrected sequential-cap sizing may admit combinations that were previously (incorrectly) rejected:

INFERENCE_X_LOADED_MODELS=qwen2.5-0.5b,qwen2.5-1.5b ./scripts/dev.sh serve

Re-run benchmarks for any model you rely on sizing numbers for — the VRAM/throughput fixes changed both measured VRAM delta and throughput for every model, not just the ones re-benchmarked in this release:

make benchmark MODEL=qwen2.5-0.5b
make benchmark MODEL=qwen1.5-1.8b

---

Known limitations

• Authentication not implemented (local-only — DEC-DEFER-01)
• Rate limiting/backpressure now partially addressed: admission control rejects batch-tier requests with 429 under KV pressure (DEC-038), but no global request queue or concurrency cap exists outside per-model KV/seq gates (DEC-DEFER-02 not fully resolved)
• Streaming token counts not captured in observability middleware
• Quantization advisor score is a placeholder (always 1.0)
• qwen2.5-7b-awq's corrected weight estimate has not yet been re-validated with a live load on this hardware
• minicpm5-1b has a ~2.8 GB VRAM overhead vs. its generic footprint estimate with no confirmed root cause; a fitted correction is applied but not a diagnosed fix
• _multi_engine_overhead_gib's recalibrated 0.6 GB constant is verified for a 2-engine compare pool only — revisit if a 3+-engine pool is added
• One CVE accepted: CVE-2025-69872 in diskcache 5.6.3 (transitive vLLM dependency, no fix version available)

v0.1.2 — Adaptive VRAM Allocation & KV Cache Ceiling

InferenceX v0.1.2

Feature release. Fixes the VRAM allocation strategy that caused startup
failures on consumer GPUs and prevented running two small models side-by-side.
No API breaking changes.

---

What changed

Adaptive gpu_memory_utilization

The previous approach hardcoded gpu_memory_utilization against total VRAM.
On an 8 GB card with 6.93 GB free, 0.90 requires 7.2 GB and fails immediately.

models.yaml now uses gpu_memory_utilization: auto for all models. At startup,
InferenceX computes the correct value from what's actually free:

gpu_memory_utilization = (vram_free − buffer) / vram_total

Uses torch.cuda.mem_get_info (same allocator view as vLLM); nvml/nvidia-smi
fallback. Result is cached for the process lifetime. Buffer defaults to 0.4 GB —
override with INFERENCEX_VRAM_SAFETY_BUFFER_GB=0.5 make serve.

Startup now logs the full breakdown per model:

Loading qwen2.5-0.5b
  gpu_memory_utilization: auto → 0.82
    (6.93 GB free − 0.40 GB buffer) / 8.00 GB total [torch]
  max_model_len: 8192

max_model_len KV cache ceiling

vLLM's default max_model_len of 32768 pre-allocates ~2 GB of KV cache per
small model at startup — memory that a local single-user session never uses.
Sub-2B models are now capped at max_model_len: 8192 (~0.5 GB KV cache).
This restores dual-model side-by-side use on 6–8 GB GPUs.

Model	max_model_len	KV cache
opt-125m	8192	~0.12 GB
qwen2.5-0.5b	8192	~0.5 GB
tinyllama-chat	8192	~0.5 GB
qwen2.5-1.5b	8192	~0.5 GB
llama3-8b	4096	~1.0 GB

Benchmark provenance — max_model_len drift detection

BenchmarkResult now stores the max_model_len active during the run.
The advisor warns when a stored result was benchmarked with a different
context ceiling than the current config — prompting a re-run.

Playground server lifecycle

make chat and make playground now cleanly stop the background
uvicorn/vLLM process on exit — only when the TUI started it. Attaching
to an already-running server leaves it untouched.

---

Test coverage

299 unit tests, 0 failures. (+11 from v0.1.1)

---

Upgrade notes

Re-run benchmarks after upgrading — stored results have max_model_len: null
and will trigger advisor drift warnings until refreshed:

make benchmark MODEL=qwen2.5-0.5b
make benchmark MODEL=tinyllama-chat
make benchmark MODEL=qwen2.5-1.5b

---

Known limitations

• Authentication not implemented (local-only — DEC-DEFER-01)
• Rate limiting not implemented (DEC-DEFER-02)
• Streaming token counts not captured in observability middleware
• Quantization advisor score is a placeholder (always 1.0)
• Cold-start VRAM margin is a fixed multiplier (1.20); true peak measurement deferred (DEC-034)
• llama3-8b requires quantization to run on 8 GB GPUs (deferred — see #4)
• Multi-model sequential VRAM profiling not yet implemented (deferred — see #4)
• One CVE accepted: CVE-2025-69872 in diskcache 5.6.3 (transitive vLLM dependency)

v0.1.1 — Advisor Accuracy & Benchmark Hardening

InferenceX v0.1.1

Patch release fixing three bugs in the benchmark advisor and one in the playground loading screen. No API additions or breaking changes.

---

What changed

Model advisor — VRAM viability fixes

• peak_vram_delta_gb was always 0.0 when benchmarks ran against a pre-loaded model (max(0, free_before − free_after) returned zero when free VRAM barely moved). Fixed to vram_total − min(free_before, free_after) — captures true GPU occupancy regardless of load state
• Viability gate now applies a cold-start margin: effective_required = peak_vram_delta_gb × 1.20. Cold vLLM startup (weights + KV cache + CUDA init) peaks 15–25% above steady-state footprint; comparing warm footprint to free VRAM could mark a model VIABLE that OOMs on cold load
• Recommendation strings show the full calculation: 3.19 GB footprint × 1.20 = 3.83 GB required
• VRAM headroom scoring uses effective_required for consistency with the gate
• Override margin via INFERENCEX_COLD_START_MARGIN=1.25 make advise
• make advise hardware header now prints the active margin and override instructions

Benchmark results — hardware provenance

• Stored result JSON lacked machine provenance; advisor scores from a desktop could silently pollute rankings on a laptop
• Results now persist a hardware snapshot at run time: gpu_name, vram_total_gb, vram_free_gb, cpu_cores, ram_total_gb
• Advisor skips results whose saved GPU name or VRAM total mismatches the current machine (>0.5 GB tolerance) and prints WARNING: lines to stderr
• Legacy results without a hardware field are soft-warned but still ranked — re-run make benchmark MODEL=<name> to refresh

Playground loading screen — actionable error surfacing

• On vLLM OOM or startup failure, loading screen showed a generic fallback while the real error was buried in logs/playground-server.log
• extract_error_summary() now tails 120 lines, matches vLLM OOM and ERROR patterns explicitly, and falls back to a timeout heuristic when the last log line is still a weight-loading INFO
• Duplicate error line in LoadingScreen.set_error() removed

API

• GET /v1/benchmark/advise response now includes a warnings field (list of strings); empty array when all results are clean

---

Test coverage

288 unit tests, 0 failures. (+15 from v0.1.0)

---

Upgrade notes

Stored benchmark results from before this release have peak_vram_delta_gb: 0.0 if they were run against a pre-loaded model.
Re-run to get accurate advisor rankings:

make benchmark MODEL=qwen2.5-0.5b
make benchmark MODEL=tinyllama-chat

---

Known limitations

• Authentication not implemented (local-only deployment assumed — DEC-DEFER-01)
• Rate limiting not implemented (DEC-DEFER-02)
• Streaming token counts not captured in observability middleware (SSE responses bypass body buffering by design)
• Quantization advisor score is a placeholder (always 1.0) until INT8/FP8 benchmark data is available
• Cold-start VRAM margin is a fixed multiplier (1.20); true cold-load peak measurement in the benchmark runner is deferred (DEC-034)
• One CVE accepted: CVE-2025-69872 in diskcache 5.6.3 (transitive vLLM dependency, not directly used)

v0.1.0 — Initial release

InferenceX v0.1.0

A self-hosted LLM inference platform backed by vLLM, with an OpenAI-compatible API,
model registry, observability pipeline, interactive TUI playground, benchmark suite,
and model advisor. Runs on a single WSL2 machine with one consumer-grade GPU.

---

What's included

Core inference API

• POST /v1/chat/completions — OpenAI-compatible, non-streaming and SSE streaming
• GET /health — HTTP 200 / 503 / 500 with structured JSON status
• GET /v1/models — lists all registered models from config/models.yaml
• Layered architecture: routes → service → engine interface → vLLM adapter
• Sanitized error responses; no internal details leak to clients

Model registry and routing

• Config-driven ModelRegistry from config/models.yaml
• Chain-of-responsibility routing: ExplicitModelPolicy → DefaultModelPolicy
• Eager startup validation — misconfigured models crash the process at boot, not on first request
• 5 pre-configured models: opt-125m, qwen2.5-0.5b, tinyllama-chat, qwen2.5-1.5b, llama3-8b

Observability pipeline

• ObservabilityMiddleware records latency, token counts, and error flags — zero changes to route handlers
• In-memory ring buffer (1000 records); optional NDJSON export via INFERENCE_X_METRICS_FILE
• GET /v1/metrics summary: total requests, error count, avg and p95 latency
• Non-blocking write path (~0.14 µs/write)

Multi-model engine pool

• INFERENCE_X_LOADED_MODELS=model-a,model-b loads multiple models in a single server process
• Automatic gpu_memory_utilization weight-splitting and sequential VRAM cap validation
• Fails fast if the model combination cannot fit in available VRAM

Chat CLI (make chat)

• Claude-style multi-turn conversation interface
• Scrollable turn history with role-styled bubbles (user / assistant)
• Live token streaming via Markdown.get_stream()
• LoadingScreen with real-time server log feed during model startup
• Ctrl+N to start a new conversation, Ctrl+C to quit

Compare playground (make playground)

• Side-by-side model comparison TUI (two models, one server process)
• Dual streaming panels with synchronized rendering
• Requires INFERENCE_X_LOADED_MODELS with two models

Benchmark suite and model advisor

• Fixed 10-prompt suite (benchmarks/prompts/standard.json, SHA256-versioned)
• Measures TTFT, total latency (p50/p95/p99), throughput (tok/s), VRAM delta
• ModelAdvisor ranks models with a weighted composite score (40% throughput, 30% TTFT, 20% VRAM headroom, 10% quantization fit)
• Models that exceed available VRAM are marked viable=False and scored 0
• GET /v1/benchmark/results and GET /v1/benchmark/advise API endpoints
• make benchmark MODEL=<name> and make advise CLI targets

WSL2 / vLLM compatibility

• Automatic FlashInfer JIT disable on WSL2 (VLLM_USE_FLASHINFER_SAMPLER=0)
• CUDA_HOME resolution from bundled toolkit when system path is absent
• pin_memory=False and spawn multiprocessing enforced on WSL2
• scripts/install_vllm_patch.sh for persistent site-packages .pth hook

Security and hardening

• Default bind to 127.0.0.1
• Schema constraints: content max 32k chars, max_tokens ≤ 4096, temperature 0.0–2.0
• Streaming timeout via asyncio.wait_for (INFERENCE_X_STREAM_TIMEOUT_S, default 120s)
• SSRF guard on playground URL validator
• Rotating file log handler via config/logging.yaml

---

Test coverage

273 unit tests, 0 failures.

---

Quick start

uv sync
cp .env.example .env          # add HF_TOKEN if using gated models
make chat                     # start the chat CLI
make playground               # start the compare TUI
make benchmark MODEL=qwen2.5-0.5b
make advise

---

Known limitations

• Authentication not implemented (local-only deployment assumed — DEC-DEFER-01)
• Rate limiting not implemented (DEC-DEFER-02)
• Streaming token counts not captured in observability middleware (SSE responses bypass body buffering by design)
• Quantization advisor score is a placeholder (always 1.0) until INT8/FP8 benchmark data is available
• One CVE accepted: CVE-2025-69872 in diskcache 5.6.3 (transitive vLLM dependency, not directly used)