v0.4.0rc1

This release candidate adds Gemma 4 MTP support via MLLM assistant drafters, extends the system-prompt KV cache to the pure-LLM streaming path, and trims Python overhead across decode, streaming, and constrained-decoding hot paths. It also lands the bench-serve workload runner refactor and several reasoning and tool-calling correctness fixes.

This is a release candidate. Install with pip install vllm-mlx==0.4.0rc1 --pre.

Highlights

• Gemma 4 MTP support wired through the MLLM assistant drafter path (#507), with counters and effective draft stats exposed for inspection (#473, #504).
• The system-prompt KV cache now covers the pure-LLM stream_chat path, not only the chat-completion routes (#523).
• Python overhead in decode, streaming, and constrained-decoding hot paths reduced (#526).
• bench-serve workload runner split into focused helpers across four PRs (#515, #517, #518, #519).

MLLM and multimodal

• Wire MLLM assistant drafters for Gemma 4 MTP (#507)
• Expose effective MLLM MTP draft stats (#473)
• Expose MLLM MTP draft counters (#504)
• Detect MLLM via config.json with a substring matcher as fallback (#520)

Reasoning and tool calling

• Fix dangling think tags before tool calls in chat templates (#494)
• Respect enable_thinking=false from chat_template_kwargs (#537)
• Correct _suffix off-by-one and strip backslashes before non-ASCII in JSON output (#525)
• Fix sampling defaults and short prefix-cache reuse (#424)
• Fix unexpected keyword argument 'mtp' when enable-mtp is set (#503)

KV cache and memory

• Extend the system-prompt KV cache to the pure-LLM stream_chat path (#523)

Performance

• Trim Python overhead in decode, streaming, and constrained-decoding hot paths (#526)

Server and API

• Use the request model in registry responses (#549)

bench-serve

• Split load_workload and harden the workload runner with regression tests (#515)
• Split validate_quality_checks into per-check helpers (#517)
• Split run_workload_case into request, transport, and record helpers (#518)
• Split summarize_workload_results into per-case helper (#519)

Tooling and docs

• Re-enable lifecycle tests and wire them into CI (#532)
• Add missing sampling defaults to lifecycle CLI fixtures (#558)
• Add structured issue templates and refresh PR template (#559)
• Add M5 to the Apple Silicon support list (#557)
• Document PR merge readiness checklist (#492)

Install

pip install vllm-mlx==0.4.0rc1 --pre

Full diff: v0.3.0...v0.4.0rc1