Lightning MTP concurrency sweep on M5 Ultra (Qwen3.8-27B oQ8e), on vs off, 1–8 streams

[ReclaimedNode]

Lightning MTP on Apple Silicon: concurrency sweep (1/2/4/8 streams), MTP on vs off, plus built-in benchmark

Environment

• Mac Studio, M5 Ultra, 256 GB unified memory
• macOS 27.0.1 (26A434)
• omlx 0.7.3rc1 (build 3517), installed from the DMG
• Model: windowsxp811203/Qwen3.8-27B-Abliterated-MLX-oQ8e-mtp (8-bit oQ, includes MTP weights; Qwen3.8-27B is a hybrid Gated DeltaNet / full-attention model)
• Server: port 8001, max_concurrent_requests = 8
• MTP settings: mtp_enabled: true, mtp_adaptive_max_depth: 3, no mtp_num_draft_tokens set (default). vlm_mtp_enabled: false
• Sampling defaults (temp 1.0, top_p 0.95, top_k 20), enable_thinking: true. All other model options (A8, TurboQuant KV, ANE prefill, DFlash, SpecPrefill) off
• Log confirms: Speculative backend selected ... Lightning MTP (model_type=qwen3_5, active)

Method

• Stdlib Python client, N simultaneous non-streaming chat requests, unique prompt per request (nonce, to defeat prefix cache)
• max_tokens 300, temperature 0.7, one untimed warm-up, 2 runs per level, 30 s cooldown, median reported
• Combined tok/s = total completion tokens / wall-clock of the whole batch
• Same server, same session, same model; MTP toggled in model settings and the server restarted between runs. The MTP-off run has no MTP log lines
• Thinking left on, so counts include reasoning tokens

Results (combined tok/s, median of 2 runs)

Streams	MTP on	MTP off	Change
1	55.6	31.6	+76%
2	76.4	55.8	+37%
4	104.7	97.4	+7%
8	131.8	139.1	-5%

Per-stream tok/s, on / off: 55.6 / 31.6, 39.9 / 28.0, 26.8 / 24.6, 17.2 / 17.7.

Notes:

• No concurrency collapse. Combined throughput keeps rising to 8 streams in both modes.
• The -5% at 8 streams is within run-to-run spread for 2 runs (individual MTP-on runs at 8 streams were 135.4 and about 128). I would not call it a regression without more runs.
• The MTP-off baseline matches my earlier omlx 0.7.2 no-MTP run (32 / 56 / 102 / 145) within about 4%.
• For comparison, mlx-vlm with MTP (draft-kind mtp, block size 3, --max-num-seqs 2) on the same hardware gave about 37 combined tok/s at 8 streams. That run was capped at 2 concurrent sequences, so it isn't a like-for-like comparison.

omlx built-in benchmark, MTP on

Context: Code (Python), single request:

Test	TTFT (ms)	TPOT (ms)	pp tok/s	tg tok/s	E2E (s)	Peak mem
pp 1024 / tg 128	1009.3	15.8	1014.6	63.9	3.0	33.8 GB
pp 4096 / tg 128	2472.6	15.3	1656.6	65.8	4.4	34.9 GB
pp 8192 / tg 128	4916.8	12.2	1666.1	82.7	6.5	35.2 GB
pp 16384 / tg 128	10026.6	10.0	1634.1	100.5	11.3	35.8 GB
pp 32768 / tg 128	21293.2	6.6	1538.9	151.7	22.2	36.9 GB

Batch (pp 1024 / tg 128): 1x 63.9, 2x 99.4 (1.56x), 4x 123.9 (1.94x), 8x 181.9 tok/s (2.85x). Average TTFT at 8x: 5.07 s.

Context: Novel (English), single request:

Test	TTFT (ms)	TPOT (ms)	pp tok/s	tg tok/s	E2E (s)	Peak mem
pp 1024 / tg 128	786.8	23.1	1301.5	43.6	3.7	33.9 GB
pp 4096 / tg 128	2489.6	20.9	1645.3	48.1	5.2	34.9 GB
pp 8192 / tg 128	4904.0	19.0	1670.5	53.0	7.3	35.2 GB
pp 16384 / tg 128	10028.2	21.0	1633.8	48.1	12.7	35.8 GB
pp 32768 / tg 128	21297.6	24.3	1538.6	41.5	24.4	36.9 GB

Batch (pp 1024 / tg 128): 1x 43.6, 2x 75.1 (1.72x), 4x 101.9 (2.34x), 8x 144.6 tok/s (3.32x). Average TTFT at 8x: 4.71 s.

MTP log observations (~/.omlx/logs/server.log)

Code, single requests (pp 1024 → 32768): tok/cycle rises 2.51, 2.56, 3.46, 4.74, 7.53, tracking tg tok/s. Starting at pp 8192 a copy= counter appears and grows (copy=0/7, copy=56/65, copy=98/98 at 32K). At 32K, 98 of 128 tokens came through the copy path and only 12 drafts went through the MTP head (accept 11/12). So the speed-up with context on this prompt looks driven mostly by prompt-copy drafting.

Novel, 8 streams: several requests ended with finish=batch-parked at tok/cycle 1.4–1.9 and accept 24–58%. Others ended batch-standard-calibration or late-join-handoff. It looks like MTP is parked in favor of plain batching at high concurrency on low-predictability text, which would explain why on/off converge at 4–8 streams.

Code, concurrent phase: accept 65–85%, tok/cycle 2.3–3.0 across 8 streams (about 26 cycles per request).

Questions for the maintainers

1. Is the copy= path a prompt-lookup or n-gram drafter that runs alongside MTP? If so, it would be useful to report copy and MTP-head contributions separately in the benchmark output, so the Code results aren't read as pure MTP gains.
2. Is batch-parked / batch-standard-calibration the intended policy for choosing between MTP and plain batching under load, and is there a way to force MTP on at 4–8 streams for comparison?
3. mtp_adaptive_max_depth is 3 in my settings, but the logs show d4 counters. Is d4 reachable when max depth is 3, or is that a separate default?
4. Does build 3517 include the batched-MTP fixes from PR fix(mtp): keep the batch park backoff across cohorts #4810?

Caveats

• 2 runs per level; the 8-stream difference is within noise.
• Thinking was on for the sweep, so completion counts include reasoning tokens.
• Built-in benchmark was run with MTP on only; no paired MTP-off run.
• One model, one machine.