v0.7.1 - Qwen3.8-Flash-Next decode +11%, plus image input and MTP

This release makes Qwen3.8-Flash-Next decode meaningfully faster, adds image input and MTP
speculative decoding, and rebases onto the current state of upstream PR #27742.

Support

If you want to support my work on making local inference better, you are welcome to do so here:

buymeacoffee.com/nathanw1014

It goes towards hardware, which means faster iteration on finding, testing and validating fixes,
and that feeds back into more and better releases.

Decode is 11% faster, and it was never a bandwidth problem

llama.cpp was rebuilding and rescheduling the entire 7271 node graph on every single token.
qwen4exp's two custom graph inputs never implemented can_reuse, so the base class returned false
and the graph was thrown away and rebuilt each step. graphs reused was 0.

How we found it: we cut 5.3% of GPU kernel time and got 0% more tokens per second. If the GPU had
been the bottleneck that would have converted into throughput. It did not, so the bottleneck was
the CPU rebuilding the graph.

22.96 to 25.05 t/s on the 64 GB box, byte identical output over 3 prompts x 128 tokens at
temp 0. graphs reused goes 0 to 322.

Worth knowing if you have been puzzled by the 22 to 24 t/s everyone reports regardless of quant:
only 27% of per token decode traffic is experts on this arch. Per token that is dense 2.34 GiB,
LM head 0.49 GiB (the vocab is 248320), experts 1.04 GiB. Of the 6.67B active params, 4.3B are non
expert. Changing expert bits moves a quarter of the bytes, which is why IQ1_S and IQ4_XS land in
the same place.

Fork vs upstream, same base, same flags

Measured against upstream PR #27742 at the identical commit, same model, counterbalanced, separate
launch per cell, on a 64 GB box:

	this fork	upstream
pp512*	101.66	95.26	+6.7%
pp4096*	98.73	93.74	+5.3%
tg128	25.38	21.82	+16.3%

* Prefill here is limited by the engram table, not by the GPU, and this is a 64 GB
constraint. Flash-Next carries a ~95 GiB PLE / engram table that cannot be offloaded, so it is
memory mapped and gathered 16 rows per token. pp512 pulls thousands of random rows out of that
mapping; on a 64 GB box the experts already hold ~56 GiB of GTT, leaving little page cache, so
most of those rows are cold reads from disk. The experts themselves are fully resident at
--n-cpu-moe 0, they are not streaming.

On a 128 GB box expect prompt processing in the 400 t/s range, roughly 4x what you see above.
Same weights, same code: with ~70 GiB free for page cache after the model, most of the engram
table stays cached and the gathers stop hitting disk. That is consistent with the 390 t/s reported
by u/4e57ljni on 128 GB. We do not have a 128 GB Strix box here, so treat that figure as reported
rather than measured by us.

Decode is not affected by any of this, which is why the tg128 gap is the largest of the three.

Image input works

Qwen3.8-Flash-Next is multimodal and it now runs:

llama-mtmd-cli \
  -m Qwen3.8-Flash-Next-Q3-*.gguf --mmproj mmproj-F16.gguf \
  --image photo.png -p "Describe this image." \
  -ngl 99 --n-cpu-moe 8 -fa on \
  --load-mode mmap --no-host --no-repack --fit off < /dev/null

Use the mmproj-F16.gguf from
unsloth/Qwen3.8-Flash-Next-GGUF.

Two gotchas:

• Give it GTT headroom. --n-cpu-moe 0 gets OOM killed on a 64 GB box once the vision tower
  is loaded beside the target. 8 works.
• --image and -p together force single shot. Without an image the CLI drops into a chat
  REPL, and on non interactive stdin it spins on its own prompt and writes gigabytes of log.
  Redirect stdin from /dev/null.

Running it

The engram / PLE table is ~95 GiB at f16. It cannot be offloaded and is not meant to be: it is
memory mapped and read a few KB per token.

llama-server -m Qwen3.8-Flash-Next-Q3-*.gguf \
  -ngl 99 --n-cpu-moe 0 -fa on \
  --load-mode mmap --no-host --no-repack --fit off

--load-mode mmap is not optional. The default auto disables mmap when a Vulkan device is
present and then tries to allocate the whole table, which is a real second copy rather than a
mapping. If you see ~95 GiB of anonymous RSS, that is what happened. Check with:

grep -E "RssFile|RssAnon" /proc/$(pgrep -f llama-server)/status

A mapping shows high RssFile and low RssAnon. The table should appear in the load log as
CPU_Mapped model buffer size, never under Vulkan0.

--n-cpu-moe 0 keeps every expert on the GPU and is what makes decode fast. Raise it only if the
model does not fit; each step spills more experts to the host and costs decode. Raising it beats
dropping -ngl, which moves whole layers rather than just experts.

MTP / speculative decoding: works, break even on 64 GB

NextN / MTP draft heads run on this release. Be aware of what that is worth on a 64 GB box before
you go looking for a speedup: it is roughly break even, and the reason is memory, not
speculation.

The draft model needs GTT alongside the target. At --n-cpu-moe 0 the target alone fills GTT and
the first queue submit dies with vk::DeviceLostError, so you have to spill experts to the host
to make room. That spill costs decode before speculation earns anything back:

config	decode
--n-cpu-moe 0, no MTP	25.38 t/s
--n-cpu-moe 4, no MTP	21.72
--n-cpu-moe 4, MTP with a matched draft	25.09

So a good draft earns back about what the spill cost, and you end up where you started. On a
128 GB box, where the draft fits without spilling, it should be additive, but we have not measured
that and will not claim it.

(The first row is llama-bench at depth 0; the MTP rows are llama-server at ~410 tokens of context.
Not strictly like for like, so read the direction rather than the exact gap.)

The part that does matter: match the draft quant to the target

If you do run MTP, this is the difference between break even and losing 15%, and it is
counter-intuitive. A draft whose quantisation error correlates with the target's agrees with it
more often, so a matched quant wins on acceptance and throughput while using less memory. A high
precision Q8_0 draft against a Q4 or Q3 target is the intuitive choice and it is the worst one.

Decode t/s against a Q3 target, --n-cpu-moe 4, three prompts (code / prose / math, ~410 token
context):

draft	acceptance	code	prose	math
none		16.92	21.91	21.72
Q8_0	0.70 - 0.84	16.57	18.66	21.36
Q4_K_M	0.66 - 0.79	17.86	19.95	22.79
Q3_K_M (matched)	0.75 - 0.79	19.66	24.70	25.09

Q8_0 is slower than running no MTP at all. Credit to Marshall /
agentionai for finding this first; the numbers above are our
own reproduction.

One more: do not requantise from someone else's quant. Going Q8_0 down to Q4 compounds the
error. Start from a BF16 extraction of the head.

llama-server -m Qwen3.8-Flash-Next-Q3-*.gguf \
  -md Qwen3.8-Flash-Next-MTP-Q3_K_M.gguf \
  --spec-type draft-mtp --spec-draft-n-max 3 \
  -ngl 99 --n-cpu-moe 4 -fa on \
  --load-mode mmap --no-host --no-repack --fit off

Base

Rebased onto the current upstream PR #27742,
picking up their latest fixes. Credit for the arch, converter and graph remains with
danielhanchen (unsloth) and the reviewers on that PR.