Pushing MTP to the limit with --spec-draft-p-min (15% faster)

[sanmai]

The guides for draft model MTP often call for a single knob: n-max or number of tokens to draft for speculative decoding.

--spec-type draft-mtp --spec-draft-n-max 2

If we increase just the n-max, eventually the generation speed starts to fade.

But there's also an important knob to increase the minimum speculative decoding probability. It stop the drafting head earlier, once it starts getting unsure:

--spec-draft-p-min 0.8

In other words, with this knob the drafting head does less work when it isn't sure; say, on novel work when a draft head is essentially guessing, this limit stops the head from generating too many tokens.

How can we use it?

But there's a flip side: if an output is fully encoded in the context, the drafting head can output a lot of tokens with a very high probability.

So if we increase the p-min from the default 0.0 to say 0.8, and bump the n-max to a higher number, say 16, we could get the best of both worlds: a sensible speedup on novel workloads without waste, and a major speedup for common code generation and refactoring tasks that usually include significant unchanged chunks of the existing code.

I tried the exact combo on a pair of dense models on a synthetic yet realistic agentic coding benchmark: refactor a function and emit the result.

--spec-type draft-mtp --spec-draft-n-max 16 --spec-draft-p-min 0.8

And it did not disappoint:

config	gemma-4-31B	Qwen3.6-27B
no speculation	47	50
n_max=1	78	77
n_max=4	112	112
n_max=16, p_min=0.8	163	178

I also tried this combo with a code recital benchmark (cite a function from a large file):

config	gemma-4-31B	Qwen3.6-27B
no speculation	47	50
n_max=1	80	83
n_max=4	118	140
n_max=16, p_min=0.8	218	225

Feel free to try this for yourself.

Hardware used: 5090 power limited to 400W.

[giveen]

My results

Gemma4-31B-MTP

Setting	Prompt t/s	Token-gen t/s	Draft attempted	Draft accepted	Acceptance	Mean accepted draft len
n-max=5, no p-min	117.19	128.42	275	199	0.72	4.62
n-max=16, p-min=0.8	110.26	136.92	356	197	0.55	5.10
n-max=12, p-min=0.8	117.95	132.18	362	195	0.54	4.90

[sanmai]

So I spend a couple of days running Gemma 4 31B in these two configuration while logging the output with tee:

Configuration	Settings	Log file
Legacy	LLAMA_ARG_SPEC_DRAFT_N_MAX=4 LLAMA_ARG_SPEC_DRAFT_P_MIN=0.0	draft-acceptance-4.log
Benchmark	LLAMA_ARG_SPEC_DRAFT_N_MAX=16 LLAMA_ARG_SPEC_DRAFT_P_MIN=0.8	draft-acceptance.log

And here are the results from the short program I used to analyze the logs.

Metric	Benchmark (16, 0.8)	Legacy (4, 0.0)	Delta
Number of Sessions	76	74
Global TPS	117.30	101.21	+15.9%
Avg Acceptance	0.580	0.701	-17.2%
Avg Draft Len	5.80	3.80	+52.6%

You'll notice that Avg Acceptance is lower for the Benchmark (58% vs 72%). Usually, lower acceptance means the draft model is failing more often, which should slow things down. But the Benchmark is actually ~15% faster: because the Benchmark is allowed to guess up to 16 tokens, when it is right, it gets a huge head start. This is reflected in the Avg Draft Len (5.8 vs 3.8).

That said, the Benchmark avoids the biggest pitfall of speculative decoding: guessing 16 tokens of complete nonsense. If the draft model is confused, it stops early and lets the main model take over.

So it sounds like it works not just in synthetic benchmarks but in real-life workloads.

[sanmai]

After more sessions the delta remains.

Metric	Benchmark (16, 0.8)	Legacy (4, 0.0)	Delta
Number of Sessions	130	98
Global TPS	118.72	99.29	+19.6%
Avg Acceptance	0.566	0.678	-16.4%
Avg Draft Len	5.39	3.71	+45.3%

Metric	Benchmark (16, 0.8)	Legacy (4, 0.0)	Delta
Number of Sessions	184	98
Global TPS	119.53	99.29	+20.4%
Avg Acceptance	0.557	0.678	-17.8%
Avg Draft Len	5.20	3.71	+40.2%

[SleepinDevil]

Interesting findings. I've always run --spec-draft-n-max 3 --spec-draft-p-min 0.80 without testing the n-max at a very high number 🤔. Everything you are saying and have found makes sense for code related work. Will need to give this a shot during my next vibe code session. Thank you for the cool idea, never thought to try it haha.

[sswtodo]

Confirmed