qwen4_exp --device cuda is fluent but NOT token-exact against the CPU control, and the divergence is already there at prefill

[localai-org-maint-bot]

Row: MODEL-MM-QWEN4-EXP

What changed

#2496 is fixed: the forward
read the host token_ids the asynchronous runner deliberately leaves stale for
decode rows, so every CUDA decode step was fed token id 0. With that repaired the
released unsloth/Qwen3.8-Flash-Next-GGUF UD-IQ1_S artifact goes from degenerate
to fluent on --device cuda. It does NOT go to token-exact.

Measured on thor:gpu0 (sm_110, driver 595.78, CUDA 13.0.88), shard-1 sha256
88a14208…, examples/vllm-server, greedy, max_tokens=8, prompt
The capital of France is, NO CUDA_LAUNCH_BLOCKING, one tree
(1ef7885ec28b4edde88c665a9c1ff00115b63e4a), one server binary
(sha256 082451c96aa20ec8ee34f1ed429972882cf0f8768294914ae2ca6d0691686663):

arm	token ids	text
--device cpu	11751 13 15767 411 2029 11 1092 369	Paris. Given this fact, what is
--device cuda	11751 13 15767 411 1928 11 628 567	Paris. Given this information, can we
--device cuda, before #2496's fix	11751 271 271 271 271 271 0 0	Paris\n\n\n\n\n\n\n\n\n\n!!

Five of eight agree, both continuations are grammatical English, and the first
disagreement is at token 4.

The divergence is NOT carried state, and that is measured

VT_Q4EXP_STATE_FP=1 prints a per-step summary of every persistent state and of
the layer loop's output. Post-fix, the PLE n-gram history rolls IDENTICALLY on
both arms — [9338,369] -> [369,11751] -> [11751,13] -> [13,15767] -> [15767,411]
— so the feedback #2496 broke is repaired.

What still differs is the prefill hidden state, at step 0, before any decode
state is read at all:

step=0 hidden  cpu   n=12800 nonfinite=0 maxabs=192 sumabs=28054.1 v=0.353516,-0.171875,3.42188,0.486328
step=0 hidden  cuda  n=12800 nonfinite=0 maxabs=192 sumabs=27964.7 v=0.326172,-0.132812,3.42188,0.470703

About 0.3% on the aggregate, on the very first forward. It compounds across
steps (step=4 maxabs 57.75 against 44.75) until a near-tie flips at token 4.
Token 0 agreed all along only because Paris after The capital of France is
is not a near-tie — which is why the whole of #2496 was diagnosed under the
belief that "prefill is right", and why that belief needed the fingerprint rather
than an argmax to check.

Candidate, not conviction

The tree already records that vt::Qwen4ExpGatedResidual's CUDA arm is NOT
bit-identical to its CPU arm: it routes its three projections through the shared
vt::MatmulBT rather than a hand-written device GEMV, and
src/vt/cuda/cuda_qwen4_exp.cu states the consequence in its own words — "a
device GEMM re-associates the K reduction, so this arm is NOT bit-identical to
its CPU sibling and its gate is a tolerance the suite MEASURES". That site runs
three times per layer over 48 layers plus the terminal mixer, which is the
largest non-bit-identical surface on the path. It is a candidate because it is
the biggest one, not because anything has attributed the 0.3% to it.

Everything else on the path IS gated arm-against-arm and passes:
test_qwen4_exp_cuda 351/351 and test_qwen4_exp_cuda_reductions 160/160 on
this same binary, with the norm and the compressor byte-identical.

What convicting it needs

A per-op tap, not another token run: dump hidden after each decoder layer on
both arms over the same artifact and the same prompt, and report the first layer
whose output leaves an arm-vs-arm band derived from that layer's own operands.
VT_Q4EXP_STATE_FP already prints the loop's output once per step; the layer
axis is what it does not have.

Until then --device cpu is the token-exact arm for this architecture and
--device cuda is fluent-but-not-exact, which is what docs/USAGE.md should
say if this model is documented as CUDA-capable.

[localai-org-maint-bot]

Measured. The first divergence is layer 0's Gated DeltaNet block, and it is NOT a defect

Wave PREFILLDIV put a per-layer, per-tap fingerprint (VT_Q4EXP_LAYER_FP) on
Qwen4ExpTextModel::Forward and ran three arms on ONE binary
(c3b355deb75efb0071fe6ec21d5067f5e2520722c672487fd67939c363e1ee58) over this
artifact on thor:gpu0 sm_110, 437 taps per step on every arm. Full result:
docs/bench-evidence/qwen4exp-cuda-prefill-divergence-20260902.md.

Where

Everything upstream of the first Gated DeltaNet block is bit-identical:

tap	CPU	CUDA	rel
emb	37.242316	37.242316	0
wide	148.969264	148.969264	0
L00 in	148.969264	148.969264	0
L00 ahc.mix	6517.5925	6517.5925	0
L00 ahc.inj	0.952026367	0.952026367	0
L00 blk	1073.65489	1074.03345	3.525e-04

The candidate this issue named is CLEARED

vt::Qwen4ExpGatedResidual reads 0.000e+00 at both hyper-connection sites of
layer 0 on the real weights at the real width. The device GEMM re-association
its own source comment warns about is absorbed by the bf16 store. The comment
was accurate about the arithmetic and wrong about the consequence, and reading
it was not a measurement.

The mechanism, by same-binary A/B

VT_GDN_CHUNKED=0 routes the CUDA arm to the same sequential recurrence the CPU
arm runs, and L00 blk falls from 3.525e-04 to 1.062e-06 — 332x. Layer 0
is the only layer that can carry that proof, because it is the only one whose
input is bit-identical on both CUDA arms; every later blk measures propagation.

And it is not a defect

vLLM's vendored Flash-Linear-Attention kernel (at 5559679229, a forward
reference) keeps h, u, w and v_new in bf16 with fp32 accumulation
(chunk_delta_h.py:352, :88-94; wy_fast.py:137-138), A in fp32 for the
triangular solve (chunk_scaled_dot_kkt.py:161 with output_dtype=torch.float32
from chunk.py:146), and final_state in fp32 (chunk_delta_h.py:353-355).
Our chunked port matches all five. The CUDA arm mirrors the oracle; the CPU
arm's exact sequential f32 recurrence is the arm that does not. It is more
accurate than vLLM, not more correct.

The tokens, and the negative result

arm	ids	agrees
--device cpu	11751 13 15767 411 2029 11 1092 369	control
--device cuda	11751 13 15767 411 1928 11 628 567	5 of 8
--device cuda, VT_GDN_CHUNKED=0	11751 13 15767 264 1103 314 5656 321	3 of 8

The sequential arm is 2.8x closer on the prefill hidden state (out rel
1.130e-03 against 3.189e-03) and agrees on fewer ids. Agreement between
our two arms is not monotone in the distance between them, because the decode is
an argmax over near-ties. A CPU-vs-CUDA token-exactness gate is not well posed
for this architecture at this precision.

One correction to this issue's own evidence

The block above records cpu … sumabs=28054.1 … v=0.353516,… and cuda … sumabs=27964.7 … v=0.326172,…. Both numbers reproduce exactly on a different
tree, the other way round: the arm whose out is 28054.1436 with
v0=0.353515625 is CUDA, and the arm whose out is 27964.6752 with
v0=0.326171875 is CPU — the one that emitted the control ids. The conclusion
this issue drew is unaffected; the labels are not.

What remains

The MoE block is a second and independent source: with the Gated DeltaNet
contribution removed, L00 moe still differs by 7.269e-05 from an input
differing by 2.1e-05, and the Gated DeltaNet lever moves it only 1.7x against
332x. It is the only remaining candidate for an actual defect on this path and
it lands named, not diagnosed.
#2552 owns the
selected-expert-id instrument it needs.

This issue's premise — that one defect stands between qwen4_exp and correct
CUDA output — is falsified at the first divergence. It should stay open only for
whatever is decided about the MoE source, which #2552 now carries; --device cpu remains the arm to use when the exact ids matter, and docs/USAGE.md now
says so.

[localai-org-maint-bot]

Two corrections, and a reframing of what this issue can ask for.

1. This issue's evidence block transposes its arms. Both numbers reproduce, the other way
round: the 28054.1 / 0.353516 arm is CUDA, not CPU. I wrote it; it is wrong as filed.

2. The premise — "one defect stands between qwen4_exp and correct CUDA output" — is
falsified at the first divergence.

Measured on thor:gpu0 sm_110, released UD-IQ1_S, one binary, 437 taps per step across three
arms: the first divergent tensor is decoder layer 0's Gated DeltaNet block output
(RunGdnBlockPaged), rel(sum|x|) = 3.525e-04, produced from an input that is
bit-identical on both arms (emb, wide, L00 in, L00 ahc.mix, L00 ahc.inj all
read exactly 0).

That op is not a defect. The CUDA arm mirrors vLLM tensor for tensor — bf16 chunk states
with fp32 registers, bf16 u/w/v_new, fp32 A for the triangular solve, fp32
final_state. It is the CPU arm's exact sequential f32 recurrence that diverges from
upstream. Our CPU path is more accurate than vLLM's, not more correct.

Proven by same-binary A/B: VT_GDN_CHUNKED=0 routes CUDA to the same sequential recurrence
and drops L00 blk by 332x.

And closing the numerical gap makes token agreement worse. The VT_GDN_CHUNKED=0 arm is
2.8x closer on the prefill hidden state and agrees on 3 of 8 ids
(11751 13 15767 264 1103 314 5656 321), against the default's 5 of 8
(11751 13 15767 411 1928 11 628 567). Agreement is not monotone in distance, because decode
is an argmax over near-ties.

So a CPU-vs-CUDA token-exactness gate is not well posed for this model, and none is
proposed. Demanding it would require the CUDA arm to reproduce a reference that is itself a
deliberate divergence from the oracle both arms are supposed to mirror.

A second, independent source is named and not diagnosed: the MoE block still turns a
2.1e-05 input difference into 7.269e-05 at layer 0 with the GDN contribution removed, and the
GDN lever moves it 1.7x against 332x. Filed as #2552, which owns the selected-expert-id
tap — a discrete selection has bimodal error, not a tolerance.