v0.7.3
This release makes Qwen3.8-Flash-Next generation at 32k context 20% faster, prompt processing
at 16k context 30% faster, and fixes a set of correctness issues,
including the greedy repeatability defect reported by lhl
(shisa.ai). The gains grow with context depth, and larger rigs than the 64 GB test box will
see more (see the note under Summary).
Support
If you want to support my work on making local inference better, you are welcome to do so here:
Summary
- Generation at depth: +10.6% at 4k, +15.6% at 16k, +20.1% at 32k, flat at zero depth. The
gains grow with context because the removed per-token scans grew with context. - Prompt processing at depth: +29.9% at 16k; +6% at 32k on our memory-constrained test box
(within run spread there, attention math dominates by then). Upstream measured +23% at 32k
on full-GPU rigs of the same silicon (#28032), which is what 128 GB boxes should expect. - Correctness: greedy repeatability restored on our test prompts, a silent last-layer graph
defect fixed, sequence-copy and block-keying fixes for multi-sequence serving, and the GDN
normalization brought in line with the reference implementations.
These numbers understate what bigger rigs will see: the test box has 64 GB, so 8 expert
layers run on the CPU (-ncmoe 8) and cap the ceiling. With the full model on GPU the fixed
costs these fixes remove are a larger share of each token.
Qwen3.8-Flash-Next UD-Q3_K_XL, gfx1151, -ncmoe 8 -ub 256, two counterbalanced passes:
| depth | v0.7.2 tg128 | v0.7.3 tg128 | change |
|---|---|---|---|
| 0 | 10.58 | 10.59 | flat |
| 4096 | 19.99 | 22.11 | +10.6% |
| 16384 | 17.30 | 20.00 | +15.6% |
| 32768 | 14.43 | 17.32 | +20.1% |
| depth | v0.7.2 pp512 | v0.7.3 pp512 | change |
|---|---|---|---|
| 0 | 162.3 | 162.0 | flat |
| 16384 | 149.3 | 194.0 | +29.9% |
| 32768 | 133.1 | 140.8 | +5.8% (within spread on this rig) |
The pp d0 row uses warm cells only; the first cell of a cold session pays the model load
inside the measurement and is discarded per our standard practice.
The Vulkan flash-attention op itself is now depth-flat for the QSA layers: 1212us at 128k
context down to 34us, 35.6x on the op.
Changes (every commit, oldest first)
Our work:
- qwen4exp: trim the last layer to output rows (upstream 6c84c7d5d parity) + follow-up
double-gather fix. Skips the whole last-layer tail for ubatches with no output rows. - vulkan: extend the top-k FA gather-compact path to GQA caches. The QSA selection now reaches
flash attention instead of only the mask, so the 12 full-attention layers stop reading the
entire KV cache every decoded token. 9.9x/18.4x/35.6x on the FA op at 32k/64k/128k. - qwen4exp: sequence-copy, block-keying and validation fixes (upstream PR #27941 squashed,
by danielhanchen/unsloth).
Cherry-picks from upstream, credited to their authors:
- #27812 vulkan graph_optimize view-alias fix (Eric A Stalee). The repeatability fix.
- #28032 vulkan radix top-k for k >= 1024 + QSA indexer fusion (Ruben Ortlam / 0cc4m).
- #27909 Strix Halo mat-vec row tuning for batch 5-8 (Simon Teixidor).
- #28068 GDN l2norm rsqrt form (danielhanchen). Affects qwen35/qwen35moe/qwen3next/qwen4exp/
kimi/bailingmoe3 outputs slightly; PPL measured neutral to better on Qwen3.8-27B
(5.0772 vs 5.0807). - #28023 qwen4exp indexer head sum by slices (Pascal / ServeurpersoCom). Prefill, grows with depth.
- #27977 extract, 3 of 6 commits (Pascal / ServeurpersoCom): n-gram predecessor scan early-exit,
windowed get_prev_tokens, bitmap used-cell set. Kills a CPU scan that reached ~20M iterations
per token at 88k context.
Validation
13784/13784 backend-ops tests passed on the release binary (FLASH_ATTN_EXT + TOP_K suites). State blob
round-trip test. Four-length repeat gate (below). PPL gate on the GDN change. Captured 900
token greedy generations on Qwen3.8-27B and Flash-Next, old and new arms, all clean (no
replacement characters, no control bytes, no repetition loops). MTP speculative serving smoke
at the documented -ncmoe 4 shape: clean 900 token output, acceptance 0.546.
Known issues
- MTP spec decode with the 152 GiB PLE-f16 build needs -ncmoe 4 or higher on 64 GB boxes
(-ncmoe 0 dies with DeviceLostError; present in v0.7.2 too, not a regression). - Deep-context (>= 64k) Flash-Next end to end is not measurable on 64 GB with everything on
GPU at any published quant; the smallest (IQ1_S, 67.6 GiB) already exceeds the GTT ceiling. - A pre-existing ~4.3% prefill same-top divergence vs upstream is under separate investigation.
Repeatability details
lhl's Flash-Next survey measured this fork 0/12 on greedy AR repeatability, which triggered
the investigation. Root cause was a missing upstream backend fix (#27812, view-alias
dependencies in Vulkan graph reordering) that landed one day after our previous backend sync.
On the four-length repeat gate (same prompt six times into one server process,
tools/repeat_gate.py): v0.7.2 failed with drift from character 0; v0.7.3 passes three of four
lengths exactly and shows only the first-request warm-up signature on the fourth, which
upstream master also shows. Upstream master currently fails this gate outright on the same
driver. The residual variation described under Known issues sits at the driver level, below
llama.cpp entirely. Thanks lhl.