v0.7.5

v0.7.5

v0.7.5 makes Qwen3.8-Flash-Next generation 35% faster at 32k context and prompt processing 59% faster at that depth, turns on a measured dense prefill gain, brings in the DSpark prefill fix from upstream, and adds Unsloth's shared MTP heads. Every change is validated against v0.7.4.1 on the 64 GB test box; the gains grow with context depth, so larger rigs will see more.

Support

If you want to support my work on making local inference better, you are welcome to do so here:

buymeacoffee.com/nathanw1014

Summary

• Qwen3.8-Flash-Next at depth: generation +8% at 8k and +35% at 32k, prompt processing +59% at 32k, flat at zero depth. The QSA indexer no longer rebuilds its pooled block keys for the whole context on every step (PR #9 by firelzrd, with a fix from me for several requests in flight on a unified KV cache). Reporter's own numbers on the same silicon: +66% generation at 131k.
• Dense prompt processing: the f16-B path is now on by default behind a predicate fitted on 276 measured shapes. Qwen3.8-27B gains 5.0% at ub2048 and 1.9% at ub256 with generation and perplexity unchanged; models the predicate does not engage measure parity.
• DSpark and other stateful drafters: the DFlash encoder is fused into the KV injection (upstream #27310), prompt processing with the drafter attached +17%, output byte-identical.
• Unsloth "shared" MTP heads for Qwen3.8-Flash-Next load in place of the self-contained ones (2.79 GB instead of 4.14 GB at Q8_0), including under the memory fitter.
• Hygiene: the scale-cache toggle behind v0.7.4's MoE regression is removed for good (shader source byte-identical to v0.7.4.1), and the MTP hidden-state input is zeroed on token-only batches.
• Quality: Flash-Next output changes relative to v0.7.4.1 at the rounding level and stays within what this model shows from a ubatch change on the same binary (KL divergence 0.015 against a floor of 0.026 at 4k context, 0.097 against 0.084 at 8k); perplexity is not degraded at either depth.

Qwen3.8-Flash-Next Q3KEXP with f16 PLE, gfx1151, same driver, llama-bench at -ub 512, two launches per arm with three repetitions each, arms counterbalanced:

depth	pp512 v0.7.4.1	pp512 v0.7.5	change	tg32 v0.7.4.1	tg32 v0.7.5	change
0	393.4	388.4	-1.3% (within spread)	32.3	32.5	+0.5%
8192	379.3	378.7	flat	28.3	30.5	+7.8%
32768	172.8	274.2	+59%	21.5	28.9	+35%

The v0.7.4.1 pp512 cell at 32k scattered between its two launches (133 and 212); v0.7.5's did not (270 and 279).

Qwen3.8-27B UD-Q4_K_XL (dense), dense f16-B path auto versus off on the same binary:

ubatch	pp2048 off	pp2048 auto	change	tg32 off	tg32 auto
256	373.3	380.2	+1.9%	12.29	12.29
2048	360.6	378.6	+5.0%	12.29	12.28

Parity rows, v0.7.4.1 versus v0.7.5 back to back: Qwen3.6-35B-A3B UD-Q5_K_XL pp512 1388 / 1387, pp2048 1632 / 1628, tg32 58.5 / 58.4; DeepSeek V4-Flash truncated to 10 layers, pp512 and tg32 at 0 / 8k / 32k within 1.2%.

Changes

1. Qwen3.8-Flash-Next long-context decode (PR #9, firelzrd)

Five patches on the QSA indexer, none of them touching a shader:

• One cell-table scan per compress ratio instead of one per QSA layer. This is upstream's own code from the commit that added the model; this fork's port had dropped it, and it is picked back with Daniel Han's authorship.
• The block-mean slices are summed without a copy; ggml_add reads strided operands on Vulkan and CUDA alike.
• The pooled indexer keys live in a per-block cache and only the tail the current ubatch can reach is recomputed. A full block never changes again, so its key is computed once. The cache is dropped by anything that moves cells (shift, removal, copy, state load).
• The two V allocations next to the indexer that nothing reads are narrowed to one element: 1.6 GB back at 262k context with an f16 cache.

One correction on top of the series. The cache is addressed by block index and blocks are numbered across the sequences of a stream, so with two requests in flight on a unified cache (llama-server's default with -np on auto) a second request growing blocks would shift the cache's rows under the first. Measured with two slots and a 12k-token first conversation continued after a second one grew: the series alone diverges, the shipped tree continues byte for byte as if alone. The fix trusts the cache only while one sequence is present in the stream, which keeps the whole gain for a single slot and falls back to the previous behaviour otherwise. LLAMA_QSA_POOL_CACHE=0 turns the cache off for A/B or support.

Not included: the sixth patch of the PR, which drops a scale before the RMS norm. It changes output and measured within noise here (to 64k depth on a small stand-in), so it stays out.

On exactness. The series is not byte-identical to v0.7.4.1 on Vulkan, and the change is already there after the first patch, which changes no arithmetic: sharing one input set across twelve layers changes the order the backend fuses over, and this backend's rounding is not invariant to that. The cache itself is bit-exact: with it switched off the KL-divergence statistics are identical to the last digit, and a greedy continuation after an 8k prompt is byte-identical with it on and off. The size of the rounding shift is measured against the same binary at -ub 256, which is what a harmless rounding change does to this model; the numbers are in the Summary. Under speculative decoding the verify batches take a different rounding path too, of the same size as the ubatch change (154 to 133 accepted drafts on the same prompt either way).

2. Dense prefill f16-B path on by default

GGML_VK_DENSE_F16B converts the f32 activation operand of a quantized dense matmul to f16 before the mul_mm dispatch, halving the bytes moved for that operand. It has been in the fork since v0.6, default off, because forcing it on loses on some models. It now engages when two measured conditions hold: the row width is an odd multiple of 1024, which is the case where halving the row stride moves it off a memory-channel camp on this memory system, and the weight rows carry at least 6144 bytes, the break-even for the conversion's cost. Fitted on an op-level grid of 23 real shapes from a tensor census of the models on this box, four quant types and three ubatch widths: 42 of 276 cells engage, none regress (worst -0.04%, mean +4.1%). Numerically free: mul_mm stages the operand into shared f16 either way, and Qwen3.8-27B perplexity is identical chunk for chunk. GGML_VK_DENSE_F16B=0 turns it off, =1 forces it on for a model you have measured yourself.

3. DSpark prefill (upstream #27310, #26756, #27711)

Every stateful drafter costs prompt processing by design (about 190 ms per launch plus 55 to 70 us per prompt token on this box), because the drafter is fed during the prompt. Upstream folds the DFlash encoder into the KV injection decode: one llama_decode instead of llama_encode plus llama_decode, and no device-host-device round trip of the encoder output. Ported with the DeepSeek V4 multi-sequence rollback fix (#26756) and the synthetic acceptance options for benchmarking (#27711). Output on the truncated V4-Flash is identical to v0.7.4.1 on four launches.

4. Unsloth shared MTP heads (issue #17)

mtp-Qwen3.8-Flash-Next-shared-*.gguf drops the token embedding and LM head and carries the draft's own mixer under its own names. The draft now borrows those two tensors from the running target, and the memory fitter measures the shared head against a metadata-only target so --fit budgets it. Usage is unchanged (-md the shared file). One thing to know: the borrowed LM head is whatever your target quant carries (Q6_K in UD-Q3_K_XL, where the self-contained head has Q8_0), so the two heads' drafts are close but not identical; acceptance is within a few percent of each other here.

5. Hygiene

The MMID_QK_SCACHE toggle is removed and the expert-GEMM scale cache is keyed off MUL_MAT_ID directly. It was read at three sites across two files, and keeping those in sync failed twice, the second time as v0.7.4's repeated-token output on MoE q4_K and q5_K models. The preprocessed shader source is byte-identical to v0.7.4.1 for every configuration, so nothing changes at runtime; the cache itself is a measured win (+5 to +8% pp512 at ub512 on three MoE models, neutral at ub2048), which corrects the July note that called it a loss. Also: the MTP hidden-state input is zeroed on token-only batches (four lines; it was uninitialised memory read by the draft graph on the first probe).

Known and not in this release

• MTP speculative decoding on Qwen3.8-Flash-Next still runs through context checkpoints on this fork (each is the 48-layer recurrent state, 112 MB), so every rejected round pays a restore. On a short code prompt with --spec-draft-n-max 3 that makes MTP slower than plain decode here (10.5 versus 18.5 t/s at 76% acceptance); on prose it is a wash (19 versus 18.5). Upstream #28123 (recurrent-state rollback for qwen4exp) was tried on top of this release and changes nothing until the server requests rollback slots for the MTP draft; that is the next item.
• Slot-save files (--slot-save-path) written by v0.7.4 or earlier for Qwen3.8-Flash-Next do not restore into v0.7.5: the indexer cache's V is now one element wide, so the saved state has a different shape. The restore fails with a logged mismatch and the request re-prefills; the in-process prompt cache is unaffected.
• A finding from the gates, present in every build: requesting n_probs makes the token stream itself vary between server launches. Within one server it is stable. Compare content across launches with plain requests.

Validation on this build

Gates on the CI artifact dev-20260909-dff6004 (portable tarball and the container image, run on the shipped bytes, gfx1151, 2026-09-09):

• ornith35-q4k: coherence PASS (59 distinct words, 468 chars)
• qwen36-q4kxl: coherence PASS (47 distinct words, 415 chars)
• ci-q7-sweep: identical requests, identical logprob streams
• ci-q27-1024x16: identical requests, identical logprob streams
• ci-fn-sweep: identical requests, identical logprob streams
• ci-fn-1024x16: identical requests, identical logprob streams
• image: pulled ghcr.io/nathanw1014/strix-halo-llamacpp:vulkan-dev-20260909-dff6004 (cdb88888ed55)
• clean-room image, Flash-Next 1024x16: 1 unique / 16 requests identical requests, identical logprob streams

On the release tree before the CI cut: backend ops on Vulkan0 (MUL_MAT_ID 4833/4833, MUL_MAT 4962/4962 auto and forced, ADD, TOP_K, SET_ROWS, GET_ROWS, RMS_NORM, ROPE all green); MoE q4_K coherence on Ornith-1.5-35B and Qwen3.6-35B; Flash-Next repeat gates (sweep x6, 1024x16 at 129 tokens, 32k x4) and Qwen3.8-27B 1024x16, all identical with identical logprob streams; two-slot control; DSpark equivalence 4 of 4; KL divergence and perplexity at 4k and 8k context; CPU rollback and architecture tests 5 of 5.

Where this work goes next

The Vulkan work is moving to the halo-box community fork, halo-box/strix-llama.cpp, where it is maintained with the other Strix Halo contributors. The v0.7.4 correctness set is in their master (halo-box/strix-llama.cpp#20), the Vulkan performance stack is up as halo-box/strix-llama.cpp#17 with its original authorship, and the runtime changes behind this fork's Flash-Next numbers, this release's included, follow once #17 lands. Until that port is complete, this repository and the fork remain the place for issues and pull requests, and releases here continue; the toolbox and its portable bundle will track whichever tree carries the work once it has moved.

Credit

firelzrd for PR #9 and the profiling behind it; Daniel Han for the shared QSA input set (upstream #27742); Bushido76 for asking for the shared MTP heads (#17); upstream authors of #27310, #27711 and #26756.