v0.7.2 - Qwen3.8-Flash-Next prompt processing 3.5x on 64 GB
This release makes Qwen3.8-Flash-Next prompt processing 3.5x faster on a 64 GB box, which brings
it into the same range as the 128 GB machines we have measurements from.
Support
If you want to support my work on making local inference better, you are welcome to do so here:
It goes towards hardware, which means faster iteration on finding, testing and validating fixes,
and that feeds back into more and better releases.
Prompt processing is 3.5x faster, and the drive was never the problem
v0.7.1 said prefill on 64 GB was limited by the engram table, and pointed at a 128 GB box as the
way to reach the 400 t/s range. The first part holds up. The second is worth revisiting, because
a good part of that limit turns out to be how the mapping is advised rather than how much memory
is present.
Flash-Next carries a ~95 GiB PLE / engram table that cannot be offloaded, so it is memory mapped
and gathered 16 rows per token. Those gathers are close to worst case for a file mapping: over
4.75M of them, no two consecutive gathers land on the same 4 KiB page. Meanwhile the loader
advises the whole mapping sequential and the kernel reads ahead 128 KiB per fault. You ask for
about 130 bytes and the kernel fetches a thousand times that.
Measured on the 64 GB box, one 512 token prompt: the default reads 249.7 GiB off the drive for
a 152 GiB model, 1.4x the entire file, at 1.51 GiB/s sustained. The run is disk bound end to
end. A faster NVMe would not fix that; the access pattern would.
This release advises just that table random and, critically, replaces the readahead it suppresses
with a batched prefetch of the rows the next ubatch will actually touch.
| v0.7.1 default | this release | ||
|---|---|---|---|
| pp512 | 99.01 | 352.38 | 3.6x |
| pp2048 | 96.36 | 339.80 | 3.5x |
| tg128 | 26.09 | 33.41 | +28% |
| disk read per cell | 249.7 GiB | 113.1 GiB | 2.2x less |
Cold page cache before every cell, counterbalanced, separate launch per cell, -ngl 99 --n-cpu-moe 0 -ub 256 -r 3, Q3KEXP-PLEf16.
Output is unchanged. Gated at temp 0 over 5 prompts x 192 tokens: byte identical to the previous
default, with a passing determinism control.
The win shrinks with depth
The gather is a fixed cost per token, so as attention work grows it becomes a smaller share:
| depth | v0.7.1 default | this release | |
|---|---|---|---|
| 0 | 99.44 | 349.75 | 3.52x |
| 4096 | 92.22 | 303.66 | 3.29x |
| 16384 | 85.14 | 233.89 | 2.75x |
| 32768 | 75.09 | 189.76 | 2.53x |
Decode follows the same shape, +27% at depth 0 down to +14% at 32k. Quote the depth you care
about, not the headline.
On 128 GB boxes, expect about 1.4x
Three community runs on 128 GB machines, model comfortably inside RAM:
| box | quant | model vs RAM | pp512 | tg128 |
|---|---|---|---|---|
| 1 | UD-Q4_K_XL | 0.85x | 1.35x | 0.90x |
| 2 | UD-Q4_K_XL | 0.83x | 1.53x | 1.04x |
| 3 | UD-Q3_K_XL | 0.67x | 1.43x | 1.00x |
Prefill gains consistently, around 1.4x. Decode is a wash: the three land at 0.90, 1.00 and 1.04,
mean 0.98. If your table already fits in page cache there is much less to win, which is exactly
what the mechanism predicts.
The 0.90 outlier is the only box that dropped caches before each cell, and a cold cache penalises
the arm that deliberately does not pre-populate. That is a plausible cold start artifact rather
than a steady state regression, but it is one run and it is not settled. If you serve
interactively on 128 GB and want to be careful, --tensor-read-lazy off restores the old
behaviour exactly.
Thanks to the three people who ran the harness on hardware I do not have.
Correction to previously published figures
Every Flash-Next number in the v0.7.1 notes and in benchmarks/ was measured with this feature
off, because it did not exist yet as a default. pp512 ~101 and tg128 ~25 understate the model on
a 64 GB box by roughly 3.5x and 28% respectively. The fork vs upstream deltas in those notes stand,
since both arms were measured the same way, but the absolute figures do not.
The "expect 400 t/s on 128 GB" line needs qualifying too. With this release a 64 GB box measures
352, which sits inside the 314 to 414 range the two usable 128 GB reports show with the same
feature on, where previously its 99 sat well below their 233 to 289. Three machines is not enough
to say what memory does or does not buy you here, and we still only have one 64 GB box, so read
that as the gap being smaller than it looked rather than as a claim about what you need.
--n-cpu-moe floor, corrected
At -ub 2048 on a 64 GB box the minimum that fits is 2, not the 4 previously suggested.
-ncmoe 0 and 1 both die with vk::DeviceLostError (GTT exhaustion); 2, 3 and 4 all run.
Each step above the minimum costs prefill: pp512 falls about 7.4% per step, 333.9 at 2, 310.8 at
3, 286.4 at 4. Decode moves much less, about 3% across the same range. If you are on -ncmoe 4 at
-ub 2048, dropping to 2 is worth roughly 17% prefill for nothing.
At -ub 256, -ncmoe 0 fits and is faster than any -ub 2048 configuration this box can run.
Running it
llama-server -m Qwen3.8-Flash-Next-Q3-*.gguf \
-ngl 99 --n-cpu-moe 0 -fa on -ub 256 \
--load-mode mmap --no-host --no-repack --fit off
--load-mode mmap is still not optional, for the same reason as before: the default auto
disables mmap when a Vulkan device is present and then tries to allocate the whole table.
New in this release, and on by default:
--tensor-read-lazy on|auto|off (default: auto)
auto applies it to arch marked gather tables over 4 GiB, which on Flash-Next means the PLE
table. off restores v0.7.1 behaviour exactly. on forces it regardless of size.
What changed under the hood
Ports upstream #27794's TENSOR_READ_LAZY plumbing, which is a better design than the
environment variable this fork carried, and keeps the half upstream left out. Their own commit
measured MADV_RANDOM without a replacement readahead at 94.4 s against 36.7 s for an untouched
mapping, and the pair together at 34.1 s. Suppressing the kernel's readahead only pays if you
replace it, so the batched gather prefetch stays.
Because of that, an arch that marks a tensor without wiring the prefetch would get precisely the
losing half. gemma4 is marked upstream and has no prefetch path, so ranges whose tensor is not
claimed by the model are dropped before the advice is applied. Verified: gemma-4-E2B is unchanged
either way, pp512 3901.77 off vs 3928.75 on.
Also adds --tensor-read-lazy to llama-bench, which upstream wired only into llama-cli and
llama-server. The flag was unreachable from the tool people benchmark with.