v0.1.0 — omni-modality core + Qwen3-TTS speech stack

vllm-omni-mlx 0.1.0 — the first release: the vLLM-Omni serving surface, rebuilt for batch-1 latency on Apple Silicon. One process, one model, no scheduler, no worker pool — OpenAI- and Anthropic-compatible chat APIs plus a full Qwen3-TTS speech stack on MLX.

Highlights

• Chat core (OpenAI + Anthropic): /v1/chat/completions and /v1/messages with SSE streaming over one unified request core; a cross-turn prompt cache that prefills only the new suffix (measured 8.3× faster time-to-first-token on continued conversations); --draft-model speculative decoding and --kv-bits quantized-KV flags.
• Qwen3-TTS, the whole family — every checkpoint ships with every generation path:
  • CustomVoice (0.6B / 1.7B): 9 preset voices, emotion/style instructions (1.7B), buffered and streaming.
  • Base (1.7B): zero-shot voice cloning from a short reference clip (ref_audio + ref_text), buffered and streaming — token-exact streaming ICL.
  • VoiceDesign (1.7B): voices from a text description, buffered and streaming.
  • /v1/audio/speech (stream: true → chunked PCM, X-Audio-* headers), /v1/audio/voices, and the one-shot vllm-omni-mlx tts CLI.
• A batch-1 latency stack with receipts: compiled per-frame decode (fused predictor + shapeless talker step), a per-voice prefix cache that skips re-prefilling the static prompt (bitwise-reproducible streams per voice), and a single-codec-frame first chunk. Every optimization carries an interleaved A/B in its PR, and the numbers below were reproduced on two machines via the public protocol in #84.

Performance (4-bit checkpoints, serving defaults, no HTTP layer)

	M1 Max · 32 GB (cold)	M4 base · 16 GB (cold)	M4 base (sustained, warm)
time to first audio (warm voice)	88 ms	68–136 ms	~100 ms (stable)
sustained RTF, 1.7B speech	0.286	0.37–0.45	~1.0

• First audio ≈ 0.1 s on every checkpoint — a burst metric, stable across thermal states; the same (voice, text, seed) reproduces bit-for-bit across requests.
• RTF scales with the chip's memory bandwidth and thermal envelope (the M4 base throttles ~3× under sustained load; Max-class machines hold it). Full per-checkpoint tables and the reproduction protocol: #84.
• Peak memory 3.7–4.6 GiB for speech serving; the 0.6B checkpoint is the efficiency pick (RTF 0.34, TTFA 68 ms on M4).

Quick start

git clone https://github.com/ThinkFlowLab/vllm-omni-mlx && cd vllm-omni-mlx
pip install -e '.[tts]'

# speech-only server
vllm-omni-mlx serve mlx-community/Qwen3-TTS-12Hz-1.7B-CustomVoice-4bit --omni

curl http://localhost:8000/v1/audio/speech \
  -H 'content-type: application/json' \
  -d '{"input": "The quick brown fox jumps over the lazy dog.", "voice": "vivian", "stream": false}' \
  --output speech.wav

Chat + speech on one server: vllm-omni-mlx serve <omni-or-text-model> --tts-model mlx-community/Qwen3-TTS-12Hz-1.7B-CustomVoice-4bit (needs .[omni] too). More examples in examples/ and the README.

Known limitations

• speed != 1.0 on speech is not supported yet.
• Voice-cloning streams pay per-request setup (reference decode + ICL prefill; the reference re-encode is cached per clip): TTFA ~0.28 s on 1.7B / ~0.15 s on 0.6B (quiet M4, re-measured 2026-10-08) vs ~0.1 s for preset voices. The ICL prefill is compute-bound at the 4-bit GEMM floor — measured, no kernel headroom on 1.7B (#112); the 0.6B pair is the low-latency, small-memory clone pick (peak 3.7 vs 5.9 GiB).
• instructions are rejected on the 0.6B model (not trained for them); the VoiceDesign checkpoint runs bf16 weights.
• ASR (/v1/audio/transcriptions) is planned, not shipped (#68).
• More model families (e.g. VoxCPM2) are in flight and will land in a 0.1.x/0.2 line.

With thanks

Model execution rides on mlx-lm, mlx-vlm and mlx-audio (MIT), and the design follows vLLM-Omni — several TTS latency techniques here are MLX-native answers to its optimization playbook. Cross-machine numbers in #84 courtesy of @EJqy.

Full changelog: v0.1.0...main · roadmap: #2