Comparing MLX and vllm-mlx Performance with Qwen3-8B-4bit on MacBook Pro M5 Max 128GB

[louisnguyenxt]

vllm-mlx continuous batching — real-workload benchmark (Qwen3-8B, M5 Max)

Benchmark of vllm-mlx serve --continuous-batching --use-paged-cache against a plain
in-process mlx-lm baseline, on a production-style LLM-judge workload (IELTS essay
scoring: each request = 2 sequential chat completions, ~7.4k total tokens per
request, ~3.6k-token prompts, JSON output, temperature 0.05/0.1).

Setup

Hardware	Apple M5 Max, 18-core CPU / 40-core GPU, 128 GB unified memory
OS	macOS (Darwin 25.5.0)
vllm-mlx	0.4.0 (commit 0dd1157), Python 3.12
Model	mlx-community/Qwen3-8B-4bit
Baseline	mlx-lm in-process, single stream, speculative decoding (Qwen3-0.6B-4bit draft, num_draft_tokens=2), requests serialized by a lock
Requests	15 unique essays per concurrency level, thread pool of N concurrent clients

Both sides ran the exact same prompts, sampling params, and max_tokens; one warmup
request before timing in each mode. The two systems ran in separate phases (never
concurrently). Swap stayed at 0 throughout; server RSS ≈ 5.4 GB peak.

End-to-end throughput (full 2-call requests)

The concurrency tests were performed at 1, 3, 5, 7, 10, and 15. The results are as follows:

Concurrency	Throughput (req/min)	vs baseline	Latency mean	Latency p90
baseline (mlx-lm + speculative)	2.16	1.00×	27.7 s	30.1 s
vllm-mlx c=1	1.84	0.85×	32.6 s	35.0 s
vllm-mlx c=3	3.54	1.64×	49.2 s	50.6 s
vllm-mlx c=5	4.30	1.99×	68.1 s	72.0 s
vllm-mlx c=7	4.00	1.85×	93.0 s	106.3 s
vllm-mlx c=10	4.33	2.00×	118.1 s	149.7 s
vllm-mlx c=15	3.82	1.77×	216.0 s	228.5 s

The baseline is flat at ~2.1 req/min regardless of offered concurrency (serialized).
vllm-mlx loses ~15% single-stream (no draft-model speculative decoding), breaks even
at ~2 concurrent, and saturates around 2× baseline throughput at c=5–10. Past
c=10 throughput degrades slightly and per-request latency keeps growing.

Token-level metrics (single judge call, streaming)

Same 15 prompts (~3.6k prompt tokens, ~2k completion tokens each):

Mode	TTFT p50	Prefill tok/s (p50)	Per-req decode tok/s (p50)	Aggregate completion tok/s
baseline (speculative)	1.55 s	2368	107.6	~99 (serial)
vllm-mlx c=1	1.37 s	2605	87.0	82
vllm-mlx c=5	0.50 s*	7151*	42.0	193
vllm-mlx c=10	1.16 s*	3113*	20.3	197
vllm-mlx c=15	9.62 s	378	16.3	215

* TTFT/prefill at c≥5 benefit from the trie prefix cache: the same prompt set was
reused across levels, so prefill was partially cached. Decode and aggregate numbers
are unaffected by this. At c=15 prefill queueing dominates TTFT (p50 ≈ 9.6 s).

Aggregate decode scales from 82 → 215 tok/s (2.6×) between c=1 and c=15, while
per-request decode drops from 87 → 16 tok/s — the classic continuous-batching trade.

Output quality

JSON-mode reliability was perfect in both systems: 90/90 vllm-mlx requests and 45/45
baseline requests parsed and completed with zero errors. The greedy-ish baseline
(temp 0.05) was bit-stable across runs (15/15 identical scores). vllm-mlx produced
identical final scores on 13–14 of 15 essays per level (mean |Δ| ≤ 0.13 on a 0–9
half-band scale), consistent with sampling/batched-numerics differences.

Takeaways

• Continuous batching delivers a real ~2× end-to-end throughput win on this
  workload at c=5–10, even though the baseline had speculative decoding and
  vllm-mlx currently doesn't (draft-model speculative + batching would compound).
• Single-stream users are better served by plain mlx-lm + speculative (~15% faster).
• Sweet spot on this 8B/4-bit + M5 Max combo: max_num_seqs ≈ 5–10; beyond that,
  latency grows with no throughput gain.