0.7.0rc1 on M5 Max 128 GB: A/B against 8a6bfb84, and two upgrade notes

[pbohannon]

On an M5 Max, rc1 is faster than the build it replaces on almost every measure. The largest gain is tail-block reuse (#3835), which cut the next-turn wait by more than the release notes claim. Flash-Next prefill is faster, but by less than the notes say. The single-stream decode gain from #3797 did not reproduce here. Two upgrade details are missing from the notes. The depth-key rename silently changes MTP depth. Flash-Next's paged-cache block size also grows from 2,048 to 8,192 tokens.

Setup. M5 Max 128 GB, macOS 26.6.2. A is main 8a6bfb84 and B is v0.7.0rc1 35be079d. Both use mlx 0.32.2 with the same mlx-lm and mlx-vlm pins. Everything ran in one sitting, with a fresh omlx serve and an empty paged SSD cache directory per arm. MTP was on, with a single stream, temperature 0, thinking off for speed runs, and the display asleep. Models were Qwen3.8-Flash-Next-oQ4e-mtp (adaptive max depth 2 on both builds) and qwen3.8-27b-mtplx-8bit.

Measure	FN A	FN B	27B A (ceiling 3)	27B B (forced 4)	27B B (mtp_fixed_depth: 3)
Next turn of a ~10K conversation: cached tokens, TTFT	8,192, 1.57 s	10,160, 0.36 s	8,192, 3.00 s	10,160, 0.50 s	10,160, 0.50 s
Short reply (29 tokens, 2.8K prompt, warm): request to done	0.99 s	0.45 s	1.86 s	0.95 s	0.90 s
Prefill 16K / 64K / 100K tok/s	1,570 / 1,512 / 1,486	1,794 / 1,812 / 1,820	769 / 655 / –	780 / 659 / –	795 / 670 / –
Decode code / prose tok/s	96.5 / 65.2	100.2 / 65.7	49.4 / 34.2	55.0 / 33.1	51.5 / 35.4
Tool-call probe (unclosed, truncated, multi-turn loop)	pass	pass	pass	pass	pass

Read the decode row as A against B. On 2026-09-21 the same 8a6bfb84 build measured 102.6 / 75.7 on Flash-Next with the same settings and prompts, so prose ran about 14% lower in this sitting on both builds. We haven't found why yet.

Against the release notes

• Tail-block reuse reproduced, with a larger effect than claimed. Re-prefill on the next turn fell from about 2,000 tokens to about 55. First token came 4x sooner on Flash-Next and 6x sooner on the 27B, against the notes' 2x (0.83 to 0.42 s).
• Flash-Next prefill improved, but by less than claimed. At 16K it rose 14%, against the claimed 31.9%. At 64K it rose 20%, against 29.4%. Our runs had the paged cache on, while the notes' figures had it off. Our "before" at 64K was also already faster than theirs (1,512 against 1,326).
• The single-stream decode gain from feat: faster Qwen decode with batched DFlash and improved Lightning MTP #3797 did not reproduce. Flash-Next code decode rose 3.9% and prose 0.8%, against the PR's +20.7% at batch size 1. Both of our arms capped adaptive depth at 2. If the PR number used a deeper ceiling, that may explain part of the gap.
• Short replies. The per-token rate inside the 29-token reply was lower on rc1 (Flash-Next 130.2 to 93.6 tok/s), which is consistent with the perf(engine): release the first generated chunk before continuing a decode burst #3854 effect reported in Flash-Next + Lightning MTP: per-cycle compute cost regression across dev3→main (5-version ladder, bisect-ready data) #3895. End to end the reply still finished twice as fast, because the prompt was almost fully cached (2,751 of 2,758 tokens, against 2,048). We did not isolate perf(engine): release the first generated chunk before continuing a decode burst #3854 from feat(cache): reuse prefix tail blocks in front of the generation prompt #3835.
• Truncated thinking. On a length stop, streaming Chat Completions now keeps reasoning out of content. Non-streaming /v1/messages still returns the reasoning as a text block; fix(thinking): keep prompt-opened truncated thinking out of content #3893 covers that case.

Two things the notes don't mention

1. mtp_num_draft_tokens is ignored without a warning (fix: migrate mtp_num_draft_tokens to mtp_adaptive_max_depth for backward compat #3896 adds a migration). For Flash-Next an old value of 2 falls back to the default ceiling of 3. The dense 27B gets a ceiling of at least 4 whatever the setting says. On rc1 that ceiling gave +7% code and -6.5% prose against mtp_fixed_depth: 3, which is the only way back below 4.
2. Flash-Next's paged-cache block size went from 2,048 to 8,192 tokens (perf(qwen4): speed up Flash-Next prefill #3903). An existing SSD cache is effectively cold after the upgrade.