MTP speculative decoding degrades throughput on Metal (Apple Silicon) — net loss at every configuration

[thewesjohnson]

Name and Version

$ llama-server --version
version: 9330 (328874d)
built with AppleClang 21.0.0.21000099 for Darwin arm64

Operating systems

Mac

Which llama.cpp modules do you know to be affected?

llama-server, libllama (core library)

Command line

# Baseline (non-MTP)
  llama-server -m Qwen3.5-9B-Q4_K_M.gguf -ngl 99 --port 8080

  # MTP with speculative decoding
  llama-server -m Qwen3.5-9B-MTP-Q4_K_M.gguf -ngl 99 --port 8080 --spec-type draft-mtp --spec-draft-n-max 6

Problem description & steps to reproduce

MTP speculative decoding (--spec-type draft-mtp) on Apple Silicon Metal produces correct output but is slower than the non-MTP baseline at every draft
ceiling configuration tested. Even with 100% draft acceptance at n_max=0, throughput drops 11%. Higher ceilings make it worse, up to 28% slower at n_max=6
with only 43% acceptance.

The draft evaluation overhead on Metal exceeds the speculative gain.

llama-server b9330, Qwen3.5-9B-MTP Q4_K_M, 2048 tokens generated, temperature=0:

Config	tok/s (think ON)	tok/s (think OFF)	Draft Accept
Baseline (non-MTP model)	25.3	25.1	—
--spec-draft-n-max 0	22.4	22.1	100%
--spec-draft-n-max 2	21.9	21.3	76% / 73%
--spec-draft-n-max 6	19.3	18.3	44% / 41%

• Ceiling (n-max) drives the regression. Floor (n-min) has no effect (0/6 = 2/6 = 6/6).
• Higher ceiling = worse acceptance AND more overhead.
• No configuration beats baseline.

Steps to reproduce:

1. Load non-MTP Qwen3.5-9B Q4_K_M via llama-server -m ... -ngl 99
2. Generate 2048 tokens with a long-form prompt, note tok/s (~25)
3. Load MTP variant with --spec-type draft-mtp, same prompt — observe slower tok/s
4. Vary --spec-draft-n-max from 0 to 6 and observe progressive degradation

System: MacBook Pro 14-inch (2021), Apple M1 Max, 24-core GPU, 32 GB unified, macOS 26.4.1

Full data including LM Studio comparisons and additional models (Qwen3.6-35B-A3B-MTP shows 5–14x regression):
https://github.com/thewesjohnson/gists/tree/main/mtp-throughput-regression-metal

First Bad Commit

Unknown. MTP on Metal, as a new feature, may never have provided a speedup. Not a regression from a known working state.

Relevant log output

Logs

  common_speculative_impl_draft_mtp: adding speculative implementation 'draft-mtp'
  common_speculative_impl_draft_mtp: - n_max=3, n_min=0, p_min=0.00, n_embd=4096, backend_sampling=1
  common_speculative_impl_draft_mtp: - gpu_layers=-1, cache_k=f16, cache_v=f16, ctx_tgt=yes, ctx_dft=yes, devices=[default]
  srv    load_model: [spec] adding 1808.02 MiB to fit_params_target for device MTL0
  srv    load_model: [spec] estimated memory usage of MTP context is 1808.02 MiB
  srv    load_model: speculative decoding context initialized

  slot print_timing: prompt eval time =     518.24 ms /    26 tokens (   19.93 ms per token,    50.17 tokens per second)
  slot print_timing:        eval time =    7551.42 ms /   271 tokens (   27.87 ms per token,    35.89 tokens per second)
  slot print_timing:       total time =    8069.66 ms /   297 tokens
  slot print_timing:    graphs reused =         79
  slot print_timing: draft acceptance = 0.80000 (  192 accepted /   240 generated)
  Same symptom class on SYCL: #23533, #23203. No prior Metal report.

[CN-QN1-dalin]

Thanks for the detailed report on MTP speculative decoding on Metal.

We have been experimenting with speculative decoding on Metal too (72B Q4_K_M on M1 Pro 16GB) and saw similar patterns — the overhead of managing multiple draft models on unified memory sometimes eats the gains.

One thing that helped us: instead of adding more speculative layers, we are working on optimizing the individual matmul kernels. The Q4_K dequant+matmul in Metal currently takes two steps (dequant to f32 buffer, then matmul). We are testing a fused kernel that does both in one pass, eliminating the intermediate tensor. Early estimates suggest ~2x per-matmul improvement.

Happy to share benchmark data once our tests are done. Also curious if anyone has profiled where the Metal speculative overhead actually comes from — kernel launch count or memory bandwidth contention?

[x4080]

@thewesjohnson I think you're right - I just baffled, that my mac is not improving at all with MTP, but since other user experience increase speed (maybe using non mac) - I just keep my mouth shut :)

[am17an]

This is not a bug

[CN-QN1-dalin]

@am17an Disagree that it is "not a bug." The Metal backend is leaving performance on the table.

We profiled the Q4_K matmul on M1 Pro. The current pipeline is: dequant (write to f32 buffer) → matmul (read from f32 buffer). Two kernel launches, a full intermediate tensor that never needed to exist.

A fused dequant+matmul kernel — single pass, zero intermediate buffer — is not theoretical. We have a working prototype. The math is straightforward: load quantized block → unpack scales → multiply-accumulate directly into output. One kernel launch instead of two.

Calling MTP slowdown "not a bug" accepts a 28% regression as normal. It is not. The hardware is capable. The software is not.

[am17an]

MTP is optional. It is not baked in by default, so you can choose not to enable it. Also please read the contributing guidelines about using AI to write your responses, multiple instances will lead to a ban.

[thewesjohnson]

MTP is optional. It is not baked in by default, so you can choose not to enable it. Also please read the contributing guidelines about using AI to write your responses, multiple instances will lead to a ban.

@am17an

Agree that MTP is optional. Spent about two hours today trying to study this and put the information forward in a helpful manner. However, if you release a feature, do you support it across different ISAs and OSes and kernels? It's definitely up to framing for supporting features and your product direction. I would think that a feature accepting draft tokens would overcome the overhead imposed by it, that's why I tested it at a trivial amount and I believe here I did it at 4096 tokens. It is interesting even with a high acceptance rate it actually decreases tg. I know a lot of Mac ecosystem would love to use this feature and it's such an usability step forward if drafting models aren't needed (and finding an appropriate one, setting it up, managing memory all together is a big orchestration effort on a couple to a suite of models). I hope you will reconsider this won't fix direction, but that's ultimately up to you. My appreciation to you either way.

If you do decide to support it for Mac, I'm happy to try any patches and report back in a format you find helpful.

[am17an]

There are multiple users (include Georgi himself) which have reported a sizeable speed-up on Macs. I currently don't have the bandwidth to debug your particular setup.

[brechtm]

To provide another data point: also seeing consistently slower token generation speeds with Gemma 4 26B MTP enabled on M1 Max.