v0.4.0

vllm-mlx 0.4.0 finalizes the 0.4.0 line (promoting 0.4.0rc1). It brings harmony/GPT-OSS prompt rendering, omni audio-from-video fusion, a tiered SSD KV cache that now covers multimodal models, configurable Metal memory, and a long list of tool-calling, reasoning, and MLLM correctness fixes.

Highlights

• GPT-OSS / harmony prompt rendering through the official openai-harmony renderer, so prior assistant tool_calls survive multi-turn agentic conversations instead of being flattened to bracket text. Optional, gated behind --enable-auto-tool-choice + --tool-call-parser harmony|gpt-oss and the vllm-mlx[harmony] extra, with a clean fallback to apply_chat_template when the package is absent (#581).
• Omni audio-from-video fusion: omni models (e.g. Nemotron-Omni, Qwen2.5-Omni) auto-extract the audio track from a video_url and fuse it into the forward pass; caller-provided audio always wins, non-omni models are untouched (#591).
• SSD cold tier for multimodal: --ssd-cache-dir now wires the SSD prefix-cache tier onto the MLLM/VLM path, not just text models (#618).
• Configurable Metal buffer cache: MLX_BUFFER_CACHE_LIMIT env var, with a device-scaled default instead of a hardcoded 32 GB cap, fixing OOM on <=32 GB Macs (#620).
• Multi-slot system-KV cache with hit-ratio counters, extended to the pure-LLM stream_chat path (#541, #523).
• logit_bias support on the chat endpoint (#596).

Constrained decoding & tool calling

• Validate OpenAI tool function names (#582)
• Bound constrained JSON whitespace progress (#547)
• Route constrained non-stream chat through the stream path (#548)
• Qwen3-XML: parse bare <function=> without a <tool_call> wrapper (#597)
• Qwen tool parser: extract mixed XML + function calls, strip truncated markup (#583)
• Gemma 4: parse paren / fenced / unfenced tool_code fallback forms (#623)
• Gemma 4: route the streaming tool marker out of the reasoning stream (#625)
• Fail invalid response_format output before returning success (#539)
• Correct _suffix off-by-one and strip backslashes before non-ASCII in JSON output (#525)
• Fix dangling think before tool calls in templates (#494)

Reasoning

• Mistral / Ministral [THINK] reasoning parser (#599)
• Respect enable_thinking=false from chat_template_kwargs (#537)
• No-final-content watchdog metadata for thinking routes (#512)

MLLM / multimodal

• Keep assistant tool_calls and tool messages on the MLLM chat path so agent continuations differ from the fresh prompt (#611)
• Honor chat_template_kwargs (e.g. enable_thinking) on the --mllm path (#600)
• Preserve MLLM content part order (#536)
• Detect MLLM via config.json with a substring matcher fallback (#520)
• Wire MLLM assistant drafters for Gemma 4 MTP, expose MTP draft counters/stats, clarify bypass counter semantics (#507, #504, #473, #527)
• Gemma 4 text-model dispatch and build fixes: realize private lazy arrays on the build thread, eval() the derived TextModel for the fast kernel, honor per-layer quantization overrides, match mlx-vlm 0.5.0 attention signature (#614, #606, #607, #595, #564)
• Qwen 3.5/3.6: accept the current mlx-vlm attention kwargs in the batch patch (#616)
• Require mlx-vlm 0.6.2 for Step 3.7 Flash (#589)

KV cache & memory

• Preserve original bfloat16 dtype across quantized SSD spill; snapshot KV on the producer thread; spill native QuantizedKVCache and handle bf16 buffers (#612, #563, #605)
• Fix hybrid cache snapshot aliasing (#576)
• System-prompt canonicalization for OpenAI paths (#528)
• Fix sampling defaults and short prefix-cache reuse (#424)

Streaming & concurrency

• Preserve the VLLM_MLX_SIMPLE_ENGINE_LOCK_ADMISSION=wait opt-in; fail fast on an occupied SimpleEngine route (#615, #540)
• Trim Python overhead in decode, streaming, and constrained-decoding hot paths (#526)
• Thread the SpecPrefill backbone policy (#579)

Server / API

• Use the request model in registry responses (#549)
• Expose request/token counters for /v1/status parity (#585)
• Normalize empty logits processors (#594)
• Require a patched Starlette (#587)

Models

• Support XLM-RoBERTa reranker heads (#550)

Tooling & docs

• Structured issue templates and a refreshed PR template; PR merge readiness checklist (#559, #492)
• Re-enable lifecycle tests in CI; harden the bench-serve workload runner with regression tests (#532, #515, #517, #518, #519)
• Add M5 to the Apple Silicon support list; add a caffeinate troubleshooting section (#557, #604)

Install

pip install vllm-mlx==0.4.0

Harmony rendering for GPT-OSS is an optional extra:

pip install "vllm-mlx[harmony]==0.4.0"

Full changelog: v0.3.0...v0.4.0