fix(MODEL-MM-QWEN4-EXP): decode on the DEVICE identifiers, so a CUDA step stops feeding itself token id 0 by localai-org-maint-bot · Pull Request #2550 · mudler/vllm.cpp

localai-org-maint-bot · 2026-09-02T04:30:44Z

On --device cuda the released unsloth/Qwen3.8-Flash-Next-GGUF UD-IQ1_S
artifact emitted 11751 271 271 271 271 271 0 0 where --device cpu on the same
tree and the same file emitted 11751 13 15767 411 2029 11 1092 369. Token 0
agreed and every token after it did not, bit-stably, across builds, trees and
CUDA_LAUNCH_BLOCKING.

ModelForwardInput::device_token_ids states its own contract: "the input ids for
this step are ALREADY on the device; the host vector is stale for decode rows".
The asynchronous runner's combine splices each decode row's sampled token into
that DEVICE buffer on the main queue and deliberately never writes it back,
because materialising it on the host is the synchronise that path exists to
remove. Twelve other translation units consume the field under #1305 -- which is this same
defect on Qwen3-MoE -- and this architecture was never wired to it, so it embedded
the stale vector. For a decode row the host never wrote, the stale value is ZERO.

The measurement that named it

A new VT_Q4EXP_STATE_FP=1 fingerprint prints one summary line per persistent
state per step. It is a summary and not a hash on purpose: the two arms are two
processes and two orders of arithmetic, so a checksum disagrees on a CORRECT pair
and says nothing. Every state agrees after the prefill. The first divergence is
the PLE n-gram history, at step 1, and it is int64 TOKEN IDS, so it needs no
tolerance to read:

--device cpu    [9338, 369] -> [369, 11751] -> [11751, 13] -> [13, 15767] ...
--device cuda   [9338, 369] -> [369,     0] -> [    0,  0] -> [ 0,     0] ...

The FIFO rolled correctly on both arms and what the device arm PUSHED was 0, so
the forward was HANDED token id 0. That explains every constraint the issue had
accumulated: token 0 is right because a prefill row is not a decode row; the
decode is wrong from the first step because every decode row is; it is bit-stable
because zero is a constant, not a race; and CUDA_LAUNCH_BLOCKING cannot move it
because nothing here is a launch order. It also explains why the REPORTED ids
stayed plausible -- they are the argmax of each step's logits, and what was broken
is the FEEDBACK.

The fix

The ids are materialised on the HOST rather than spliced over a device buffer
with detail::ApplyDeviceTokenIds, and that is forced rather than chosen. This
forward has a second reader besides the embed: RunQwen4ExpPleBlock hashes the
raw ids in host int64 arithmetic and advances the n-gram context the next step
reads. A device splice would repair the embed and leave the hash reading zeros.
The copy is enqueued on this step's queue, so it is ordered AFTER the combine
rather than racing it, and the drain is the read's other half. One drain per step
is the cost; a device-side n-gram hash is how to get it back, which is what
upstream does, and the spec's ## Owed carries it.

The gate runs on a CPU QUEUE

Nothing in CI runs a model forward on a GPU, which is exactly why this survived
every gate the tree has. The defect is not arithmetic and not a device kernel: it
is WHICH ARRAY the hook reads. So the case publishes a device identifier buffer
that DISAGREES with the host vector and asks which one the answer came from -- a
question a CPU queue answers exactly. It asserts the precondition first, that the
two identifiers decode to different rows, so the comparison cannot be a tautology.

Red-then-green, one local CPU binary, the fix made inert with
if (false && input.device_token_ids != nullptr) and restored byte-for-byte
(git diff empty):

leg	rc	what moved
green	0	64 assertions, 0 failed
RED (mutant)	1	`spliced != fresh`, `spliced == stale` byte for byte, and the n-gram history carries the stale id
green after restore	0	full suite 11 cases, 405 assertions, 0 failed

On device, and what it does NOT claim

thor:gpu0, sm_110, released artifact, examples/vllm-server, greedy,
max_tokens=8, no CUDA_LAUNCH_BLOCKING -- the production configuration:

arm	token ids
`--device cpu` control, re-taken on this tree	`11751 13 15767 411 2029 11 1092 369`
`--device cuda`, before	`11751 271 271 271 271 271 0 0`
`--device cuda`, after	`11751 13 15767 411 1928 11 628 567`

This is not a token-exact CUDA arm and this PR does not claim one. Five of
eight agree and both continuations are fluent English. The residual is present at
PREFILL -- the hidden state differs by about 0.3% before any decode state is read
-- so it is not carried state and not this defect. It is filed as #2547 with the
fingerprint that measures it and the tap that would attribute it.

Two instruments of this wave were also repaired by RUNNING them: an unqualified
kStream in the test helper resolved to the golden fixture's 16 instead of
qwen4_exp_fixture::kStream's 128 and overran a snapshot destination by 2016
bytes (rc 134 beside 365 | 365 passed | 0 failed), and a case asserted an
absolute band where its own header argued for a ratio.

Also files #2544: the device_token_ids contract is advisory, the runner sets it
for every model, and five more architectures reference it nowhere.

Closes #2496

FOLLOWING_AGENTS_PROTOCOL

Refs: #2496, #2544, #2547, #1305
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]

…2496 names a tensor #2496 is a whole-output symptom: the released artifact decodes `11751 271 271 271 271 271 0 0` on CUDA against `11751 13 15767 411 2029 11 1092 369` on the CPU, from one tree and one file. Token 0 agrees on both, and the CUDA sequence is byte-identical across builds and across `CUDA_LAUNCH_BLOCKING`, so prefill is right, decode is wrong, and no timing or ordering explanation survives. What is left is state the second step carries. A symptom that size cannot name a tensor. This case converts it into one. It runs one prefill and one decode through `ModelRegistry::Forward` on a CPU queue and on a CUDA queue, over the same fixture and the same pinned token ids, and compares the prefill logits, then every persistent buffer the prefill writes and the decode reads -- the GDN conv ring, the GDN temporal state, the PLE conv ring, the PLE n-gram history, the paged K/V and the indexer side cache -- then the decode logits. The first row that disagrees is the answer, and every difference is printed whether or not it trips an assertion. The second step's token is pinned rather than sampled, because sampling it per arm would feed the two arms different ids the moment the prefill logits differ at all, and the decode comparison would then be measuring two questions. FOLLOWING_AGENTS_PROTOCOL Refs: #2496 Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]

… has to emit The wave has an instrument committed and no hypothesis, which is the right order for a whole-output symptom, but the spec has to state the scope, the gate and the stop conditions before the fix goes in rather than after it. The section records what the three measurements on #2496 already exclude, the one shape that survives them, and -- the part a reader cannot re-derive -- WHY the list of candidate state is short. Within one step every buffer this forward writes is read back by that same step except three: the GDN conv ring, the GDN temporal state, the PLE conv ring and the PLE n-gram history are written at prefill and first read at decode, while the paged K/V and the indexer side cache round-trip inside the prefill and so are visible in token 0. That is what makes the wave finite. It also states, before the instrument runs, what the instrument cannot see: the fixture's layer-3 activations sit near 2^18 where one bf16 ULP is ~1024, and W5j measured 0 of 192 paged K/V words moving on it across two prompts. A green hermetic comparison is therefore not a claim that the device arm decodes correctly, and the acceptance gate says so: the GPU must emit `11751 13 15767 411 2029 11 1092 369` on the released artifact. FOLLOWING_AGENTS_PROTOCOL Refs: #2496 Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]

…be located on the REAL weights The hermetic CPU-vs-CUDA comparison committed alongside this asks the right question on a fixture, and the fixture may not be able to hold the answer: its layer-3 activations sit near 2^18, where one bf16 ULP is ~1024 and the K/V store saturates -- W5j measured 0 of 192 paged K/V words moving across two different prompts on it. This is the second instrument, and it runs where that objection does not apply. `VT_Q4EXP_STATE_FP=1` makes the production hook print, once per step, a summary of every persistent state the forward holds: each linear layer's GDN conv ring and temporal state and its two extra recurrent slots, the PLE conv ring and n-gram history, the QSA indexer side cache, and the layer loop's output. Default off, one `getenv` read once. It prints a SUMMARY, not a hash, and that is the point. The two arms are two processes and two orders of arithmetic, so a checksum disagrees on a CORRECT pair and says nothing at all. A count of non-finite words, a count of non-zero words, a maximum magnitude, a sum of magnitudes and the first four values separate a state that is WRONG -- zero, saturated, NaN, or a different tensor entirely -- from one that differs in its last bits, which is the only distinction this defect needs. The reads go through `Backend::Copy` and `Backend::Synchronize` rather than a host walk over a device pointer: `Copy` infers its direction from the pointer values and `Synchronize` is a no-op on the host backend, so the host arm pays nothing and the device arm cannot forget the drain. FOLLOWING_AGENTS_PROTOCOL Refs: #2496 Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]

…case under -Werror `thor:gpu0` build rc=1 at 1078 s, 586 objects in: test_qwen4_exp_layer_loop.cpp:2683:21: error: unused variable 'kPleIdx' [-Werror=unused-variable] The arm helper snapshots EVERY linear layer's four recurrent states rather than only the PLE layer's, so the constant naming the PLE layer's rank never gets read. A local `-fsyntax-only` check does not carry this tree's `-Wall -Wextra -Werror`, which is why it passed here and failed there; the check is re-run with those flags and is clean. FOLLOWING_AGENTS_PROTOCOL Refs: #2496 Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]

…el-internal `scripts/check-env-doc.py` reds on `VT_Q4EXP_STATE_FP`, which the state fingerprint reads and no document declares. It belongs on the allowlist rather than in `docs/ENVIRONMENT.md`: it changes no numerics, prints nothing on a default run, and its only reader is somebody bisecting this row's device arm -- the same classification `VT_GDN_DIAG_STEP_LOG` and `VT_DUMP_TRUST` carry. One line, in the file's own sort order. An earlier attempt rewrote the whole file from a sorted set and silently dropped 28 lines the sort did not carry; that is reverted and this is a single insertion, which the diffstat says. FOLLOWING_AGENTS_PROTOCOL Refs: #2496 Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]

…step stops feeding itself token id 0 #2496: on `--device cuda` the released `unsloth/Qwen3.8-Flash-Next-GGUF` UD-IQ1_S artifact emits `11751 271 271 271 271 271 0 0` where `--device cpu` on the same tree and the same file emits `11751 13 15767 411 2029 11 1092 369`. `ModelForwardInput::device_token_ids` states its own contract: "the input ids for this step are ALREADY on the device; the host vector is stale for decode rows". The asynchronous runner's combine splices each decode row's sampled token into that DEVICE buffer on the main queue and deliberately never writes it back to `token_ids`, because materialising it on the host is the synchronise that path exists to remove. Nine registries consume the field under #1305 -- which is this same defect on Qwen3-MoE -- and this architecture was never wired to it, so it embedded the stale vector. For a decode row the host never wrote, the stale value is ZERO. MEASURED rather than argued. `thor:gpu0`, the released artifact, the new `VT_Q4EXP_STATE_FP=1` fingerprint, the PLE n-gram history after each step: --device cpu [9338, 369] -> [369, 11751] -> [11751, 13] -> [13, 15767] ... --device cuda [9338, 369] -> [369, 0] -> [ 0, 0] -> [ 0, 0] ... The prefill agrees, and from the FIRST decode the model is fed token id 0 for ever. That is why token 0 was right and every token after it was not, why the sequence is bit-stable across builds and across `CUDA_LAUNCH_BLOCKING`, and why the reported ids stayed plausible: they are the argmax of each step's logits, and what was broken is the FEEDBACK. The ids are materialised on the HOST rather than spliced over a device buffer with `detail::ApplyDeviceTokenIds`, and that is forced rather than chosen. This forward has a second reader besides the embed: `RunQwen4ExpPleBlock` hashes the raw ids in host int64 arithmetic and advances the n-gram context the next step reads. A device splice would repair the embed and leave the hash reading zeros. The copy is enqueued on this step's queue, so it is ordered AFTER the combine rather than racing it, and the drain is the read's other half -- the stage-then-synchronise shape `qwen4_exp_qsa_block.cpp` already establishes. One drain per step is the cost; a device-side n-gram hash is how to get it back, and that is what upstream does. The gate runs on a CPU QUEUE and still convicts, because the defect is not arithmetic and not a device kernel: it is which array the hook reads. The case publishes a device identifier buffer that DISAGREES with the host vector and asks which one the answer came from, asserting first that the two identifiers decode to different rows so the comparison is not a tautology. FOLLOWING_AGENTS_PROTOCOL Refs: #2496 Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]

… this spec's own framing The instrument answered on its first run and named a tensor: the PLE n-gram history, at step 1, the first decode. It rolls `[9338,369] -> [369,11751] -> [11751,13]` on `--device cpu` and `[9338,369] -> [369,0] -> [0,0]` on `--device cuda`, over one artifact and one prompt. The history is int64 token ids, so the disagreement needs no tolerance to read. The section records what that named -- the forward was handed token id 0, because it read the host `token_ids` the asynchronous runner deliberately leaves stale -- and it corrects the scope's own reasoning rather than leaving it standing. The scope argued the wave was finite because only three states are written at prefill and first read at decode. The list was right; the conclusion was not. The divergence WAS in one of the three and the cause was upstream of all of them, in an input the forward is handed. An enumeration of state can locate a symptom; it cannot bound where the symptom comes from. `## Owed` gains two items. The `device_token_ids` contract is advisory and five more architectures ignore it (#2544), and the real debt there is making the contract enforceable rather than porting five forwards. And this architecture now pays a drain per step, because its n-gram hash is host int64 arithmetic while upstream computes the same ids in a device custom op. FOLLOWING_AGENTS_PROTOCOL Refs: #2496, #2544 Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]

…d main's moves `main` moved 30-odd commits while this wave measured on `thor:gpu0`. The merge is clean and the branch's own claims are re-read rather than assumed: the device measurement this branch reports was taken at `1ef7885ec`, and that SHA is named in the evidence rather than "the branch", so a merge cannot silently re-point it. FOLLOWING_AGENTS_PROTOCOL Refs: #2496 Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]

…rt the ratio the case argues for Two repairs to this wave's own instruments, both found by RUNNING them rather than by reading them. THE HEAP OVERFLOW. `RunQwen4ExpDecodeDivArm` sized the PLE conv ring's snapshot from an unqualified `kStream`. This translation unit has TWO: the golden fixture's private one at the top of the file, 16, and `qwen4_exp_fixture::kStream`, 128. The unqualified use took the first while the `DBuf` beside it was built from the second, so the snapshot downloaded 2304 bytes into a 288-byte destination -- 2016 bytes of heap corruption. It surfaced as `malloc(): unsorted double linked list corrupted` inside an unrelated `operator new` three statements later, which is why the crash line named the test case and not the write. On `thor:gpu0` it took `test_qwen4_exp_layer_loop` to rc 134 with `365 | 365 passed | 0 failed` beside it: every assertion that RAN passed and the binary still aborted, which is the shape a reader has to look past the assertion count to see. The pre-existing two-step case qualifies the same constant correctly, ten lines away. THE ASSERTION THAT WAS A DIFFERENT CLAIM. The CPU-vs-CUDA case asserted `rel1 <= 1e-3 && rel2 <= 1e-3`, i.e. that the two arms agree numerically. They do not, and not for a reason this case owns: #2547 measures the released artifact's PREFILL hidden state differing by about 0.3% between the arms, before any decode state is read at all. The case's own header argues for a different comparison -- that the DECODE is not an order of magnitude further apart than the PREFILL already is -- and that is what it now asserts, with both distances printed. The stale-identifier defect made the decode distance O(1) against a prefill distance near 1e-3, so the ratio convicts it where a widened absolute band would not. Local CPU build, one binary: 11 test cases, 405 assertions, 0 failed, rc 0. FOLLOWING_AGENTS_PROTOCOL Refs: #2496, #2547 Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]

One artifact, one prompt, one device, three runs, with the CPU control re-taken on each tree rather than inherited. The file carries the tokens before and after, the per-step n-gram history that named the cause, and the residual the fix does NOT close. It says plainly what is still open: the CUDA arm is fluent and agrees on five of eight tokens, and the remaining divergence is measurable in the PREFILL hidden state at about 0.3% before any decode state is read. That is #2547, not #2496, and the distinction is the reason the file records the fingerprint rather than only the token ids. FOLLOWING_AGENTS_PROTOCOL Refs: #2496, #2547 Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]

…them, and give DECODEDIV its row Two prose repairs, both to this wave's own writing. THE COUNT WAS WRONG. Three places said "nine registries already consume `device_token_ids`". `grep -rl 'input\.device_token_ids\|in\.device_token_ids' src/vllm/model_executor/models/` returns thirteen translation units including this one, so it is TWELVE others. The number came from eyeballing a truncated grep and was then repeated, which is exactly how a quoted number becomes treated as measured. The comment now names the command that produces it. THE `## Now` TABLE OWES A ROW. That section's own rule is that every reviewed wave which has landed has a row saying whether anything in production reaches it. DECODEDIV's says yes and says what it does NOT claim: the arm is fluent and five of eight tokens agree, and the residual is #2547. FOLLOWING_AGENTS_PROTOCOL Refs: #2496, #2547 Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]

…U and a CUDA build The gate for #2496 runs on a CPU queue by design, so the transcript records it in both places: a local CPU-only build and `thor:gpu0` inside the lease, at the same head, with the fix made inert and restored byte-for-byte each time. On device the mutant run reads 64 assertions with 61 passed and 3 failed, which is the three assertions that carry the claim and nothing else. It also records the two ways this wave's own instruments read as passes while measuring nothing: a mutation whose rebuild fails, and a case whose name carries a comma, which `doctest`'s `-tc` filter splits so that both legs returned `assertions: 0 | 0 passed | 0 failed` at rc 0. And it records that the layer-loop suite was rc 134 with `365 | 365 passed | 0 failed` beside it, because an assertion count is not an exit status. FOLLOWING_AGENTS_PROTOCOL Refs: #2496 Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]

… back the brace git hoisted `main` moved again while `thor:gpu0` confirmed this wave, and both sides had appended a test case to the tail of `test_qwen4_exp_layer_loop.cpp`: this branch's two #2496 cases and main's #2488 QSA query-width case. Git resolved that as one content conflict and HOISTED THE CLOSING BRACE out of both sides as shared context, so a take-both that only concatenates the two bodies leaves the file one `}` short and every static check still agrees with it. The brace is put back between the two bodies, and the resolution is checked by BUILDING rather than by reading: brace balance 0, 12 `TEST_CASE`s, one binary, 12 cases / 416 assertions / 0 failed. The branch's own claims are re-read rather than assumed, because a clean merge has falsified this tree's prose before. `main` touched `qwen4_exp_registry.cpp` and `qwen4_exp_qsa_block.cpp` in this range; the fix's three `step_token_ids` sites survive intact, the `device_token_ids` consumer count is still thirteen translation units so "twelve others" holds, and all five architectures #2544 names still reference the field nowhere. FOLLOWING_AGENTS_PROTOCOL Refs: #2496, #2488, #2544 Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]

…oundaries are exact ties The tap ran on `thor:gpu0` over the released UD-IQ1_S artifact, three arms on one binary. At `E = 512` and `top_k = 10` the router's top-k boundary is an EXACT bf16 tie at 79 of 240 prefill token-slots on the CPU control, 32.9%, and inside one representable bf16 step at 55.8%. At a tie the selection carries no information: the lowest-index tie-break decides it, and any perturbation upstream reshuffles it. The flip rate is the tie rate, 75 of 240 slots on the `VT_GDN_CHUNKED=0` arm against 79 tied. #2552's own headline number is NOT a flip, which is the half the issue got wrong. At layer 0 with the Gated DeltaNet source removed, the arm its table was taken on, both arms select the same experts for all five tokens. The `7.269e-05` residue decomposes onto the expert GEMM instead: `x` 2.139e-05 in, `logit` 2.378e-05 (the router GEMM does not amplify), `exp` 1.421e-04 at 6.6x, `shr` 4.310e-05. It is not a defect. vLLM routes this model on bf16 logits at 512 experts too: `Qwen4ExpSparseMoeBlock` inherits Qwen3Next's plain `ReplicatedLinear` gate with no `params_dtype`, the runner applies it as a plain `F.linear`, and the f32 widening happens inside `topk_softmax` — our polarity, our widening point, our lowest-index tie-break. The expert GEMM's arm-to-arm difference is the re-association `cuda_quant_dot.cu` already documents, and llama.cpp splits its own CPU and CUDA arms the same way. Read at a forward reference 1566 commits past the pin, which carries no `qwen4_exp` at all, and recorded as a source read rather than a measurement: no vLLM process was run on this checkpoint. Two independent instruments agree bit for bit on the MoE block's input. This tap's `x` axis reads the tensor `VT_Q4EXP_LAYER_FP` calls `L00 mhc.mix`, from a different tree, and all three arms reproduce PREFILLDIV's values exactly. The decode half is VOID and is published only to say so. On `origin/main` `a99b9c69a` the CUDA arm answers `11751 271 271 271 271 271 0 0`, not the fluent sequence PREFILLDIV recorded, because that wave measured a tree carrying #2550's decode fix and #2550 has not landed. From step 1 the arms run different token sequences, so the 336-of-336 decode flip count is a different-input artifact. `docs/USAGE.md` is deliberately untouched: its checkpoint row still claims no `qwen4_exp` step reaches a CUDA queue, which is false, and #2554 already carries that correction. Refs #2547. Refs #2550. Closes #2552. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code]

…recting what the merge falsified `main` moved from `a99b9c69a` to `bb78d1ee8` while this wave measured, and one of the commits it brought is #2550 — the decode fix this wave's evidence file named as NOT LANDED. That sentence was true of the tree three arms were measured on and false of the merge result, which is the shape where a merge falsifies a branch's own prose. Both the spec Outcome and the evidence file are corrected in this commit rather than left to read as a claim about `main` today: the decode numbers stay VOID because the arms ran different token sequences on the measured tree, the prefill result is untouched because step 0 consumes the prompt and not a sampled token, and a post-#2550 decode comparison is named as the cheap next measurement instead of being claimed. The spec conflict is a take-BOTH, not an automatic three-way merge: `main`'s DECODEDIV section and this branch's MOEDIV section were added at the same anchor above `## Now`, so the landed one is placed first and the new one after it, and both are whole. `src/vllm/model_executor/models/qwen3_5.cpp`, `tests/CMakeLists.txt` and `scripts/env-doc-allowlist.txt` auto-merged; the tap's definition, its one production call site, its ctest entry and its allowlist row were each re-verified present afterwards, because a clean auto-merge is not evidence that a scoped edit survived. Rebuilt and re-run on the merge result, read literally: build rc=0; `test_qwen4_exp_moe` rc=0 with `assertions: 121 | 121 passed | 0 failed` switch unset and rc=0 with `172 | 172 passed | 0 failed` switch set. Refs #2547. Refs #2550. Refs #2552. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code]

…nd and repairing a red main brought with it TWO CONFLICTS, RESOLVED ON THE RULE AND NOT BY A THREE-WAY MERGE. AGENTS.md "Records" requires taking the complete target-branch version and re-applying the scoped edit, then verifying unrelated keys are byte-for-byte equal. Both were done that way and both were VERIFIED that way rather than eyeballed: `scripts/env-doc-allowlist.txt` -- took main's 244-line file and inserted the one key `VT_Q4EXP_LAYER_FP` in sorted position. Diff against the target is exactly one added line and zero removed, and the file with that one line stripped is byte-identical to the target. #2550's `VT_Q4EXP_STATE_FP` survives; a take-both would have dropped one of the two. `.agents/specs/qwen4-exp-flash-next.md` -- this row's spec is written by several waves at once, so a take-both can apply two contradictory edits silently. This wave's change is PURE INSERTION, zero removals: 24 lines under `## Owed` and 191 lines before `## Now`. Took main's 9064-line file and re-applied both blocks at their anchors. Result is target + exactly 215 lines, 0 removed, and removing precisely those two blocks reproduces the target byte for byte. #2550's `## DECODEDIV (#2496)` section and its two `## Owed` bullets are intact. AND THE MERGE SURFACED A RED THAT IS MAIN'S, NOT THIS BRANCH'S (#2559). `tests/test_qwen4_exp_layer_loop` throws in THREE cases on the merged tree, and only one of them is this wave's. #2541 added `kMaxRank` to `dense_attn::MakeTensor` -- absent at `28b3786ed`, present at `35116605b` -- and #2550 and #2538 then landed rank-5 paged-KV buffers on top of it, conflicting with nothing and red on arrival. Every `ci` run on main since is `cancelled`, so no verdict was ever rendered. The header under test and both non-PREFILLDIV case bodies are byte-identical to main's, which is what makes the observation a measurement of main rather than of this merge. Repaired in flow, in main's own convention and as a pure spelling change: the vestigial unit axis `{2, 1, T, kKvHeads, kHeadDim}` becomes `{2, T, kKvHeads, kHeadDim}` beside its `num_blocks = 1; block_size = T`, and the paged form `{2, kBlocks0, kPage, kKvHeads, kHeadDim}` becomes `{2 * kBlocks0, kPage, kKvHeads, kHeadDim}`, whose element count is already spelled on the line above it. Same elements, same contiguous layout. `assertions: 352 | 352 passed | 0 failed` beside `3 failed` cases is the signature worth keeping: these cases throw BEFORE asserting, so an assertion count alone reads as a clean run. Gate on the MERGED result, every rc read literally: build rc=0, `test_qwen4_exp_layer_loop` rc=0 with 13/13 cases and 426/426 assertions, `test_qwen4_exp_forward` rc=0 with 429/429, and the fingerprint case still counts 41 == 41 taps. Refs #2547. Refs #2559. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code]

…ee is the composition main now carries The evidence file and the wave section both described #2550 as a pull request this wave had to merge locally to get one binary carrying the decode fix and the layer fingerprint together. It landed at `bb78d1ee8` while this branch was in review. That changes what the measured tree IS, and only that. `0283fae36` stops being a composition that exists nowhere and becomes the pairing `main` now has on its own, so a reader reproducing the numbers no longer has to reconstruct it. No number moves and none is restated on the new basis: everything reported was taken on `0283fae36` and is still reported against it. Refs #2547. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code]

…vergence is upstream's own algorithm, and main was red A per-layer fingerprint over 437 taps per step puts the first divergent tensor at decoder layer 0's Gated DeltaNet block output, rel 3.525e-04, from an input that is bit-identical on both arms. VT_GDN_CHUNKED=0 collapses that tap 332x, naming the chunked prefill decomposition -- which matches vLLM's vendored FLA precision map at all five anchors. The CUDA arm mirrors the oracle; our CPU arm's exact sequential f32 recurrence is the outlier, more accurate than vLLM rather than more correct. #2547's only named candidate is cleared by measurement. Token agreement is not monotone in numerical distance: the sequential arm is 2.8x closer on the hidden state and agrees on three of eight ids against five, because decode is an argmax over near-ties. No CPU-vs-CUDA token-exactness gate is proposed. This merge also repairs a break main was already carrying. #2541 added the kMaxRank bound to dense_attn::MakeTensor, and #2538 and #2550 then landed rank-5 paged-KV buffers on top of it. Neither conflicts textually with the bound, so both merged clean and both were red, and every ci run on main since has been cancelled, so no verdict was rendered. The repair is a pure spelling change with the same element count and the same contiguous layout. Worth keeping from that diagnosis: the failing binary printed 'assertions: 352 | 352 passed | 0 failed' beside three failed cases, because they throw before asserting. An assertion count alone reads as a clean run. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: claude-code:claude-opus-5 [Claude Code]

…cause a citation is not evidence The evidence file names `0283fae36` as the tree three arms were measured on. That commit is a throwaway merge of this branch with mudler#2550's, made so one binary could carry both the decode fix and the layer fingerprint; it is not a landing shape and belongs on no row branch. Deleting its worktree left the sha reachable only from a reflog on one machine. It is pushed as `measure/q4exp-prefilldiv-20260902` instead, and the file now says why. A reader who cannot check out the tree a number was taken on has a citation rather than evidence. Refs mudler#2547. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code]

… this wave's spec section onto the target `build-test-cpu` was red on #2556 and the failure was not this wave's: `test_qwen4_exp_layer_loop`, the rank-5 fixture break `main` was already carrying after #2541 bounded `dense_attn::MakeTensor` at `kMaxRank = 4` and #2538 and #2550 landed rank-5 paged-KV fixtures on top of it. Neither conflicted textually with the bound, so both merged clean and both were red, and every `ci` run on `main` is cancelled by the next push, so no verdict was ever rendered. #2554 repaired it as #2559. This branch pre-dated that repair; merging `main` is what clears the red, and it does: `test_qwen4_exp_layer_loop` rc=0 at `assertions: 426 | 426 passed | 0 failed`. The spec conflicted exactly where this wave predicted it would. #2554 added `## Wave PREFILLDIV` at the same anchor above `## Now` that this branch adds `## Wave MOEDIV` to. It is resolved on the keyed-record rule rather than by accepting a three-way merge: the complete target-branch file was taken and this wave's one contiguous block re-applied to it, and the result is proven numerically rather than eyeballed. Against the target, 0 lines removed and exactly 167 added, which is the block's own line count; deleting precisely those 167 lines reproduces the target BYTE FOR BYTE (`dd1d7ef61cedc14e61fa01f92e01a96797c4599a3e0c53d0ad9ad6396be68d3c` either side); and the re-applied block is byte-identical to the one this branch already carried. All three sibling sections survive whole and in landing order: `## DECODEDIV` (#2496), `## Wave PREFILLDIV` (#2547), `## Wave MOEDIV` (#2552). The conflict-free hunks were re-checked too, because a clean merge has falsified this tree's prose repeatedly. `scripts/env-doc-allowlist.txt` carries BOTH keys rather than one clobbering the other (`VT_MOE_SEL_FP` and #2554's `VT_Q4EXP_LAYER_FP`); the tap's definition and its one production call site (2), its switch read once (1) and its ctest entry (2) all survive; and #2554's `docs/USAGE.md` correction — that `--device cuda` NOW SERVES this artifact — is still on the merged tree rather than reverted by a branch that deliberately never touched that file. This wave's result is unchanged by the merge and is not re-derived here: it was measured on `a99b9c69a` and nothing merged since touches the sparse-MoE seam, the router or the keep-quant grouped GEMM. Read literally on the merge result: CMAKE rc=0, BUILD rc=0, `test_qwen4_exp_layer_loop` rc=0 (426/426), `test_qwen4_exp_moe` rc=0 at `121 | 121 passed` switch unset and rc=0 at `172 | 172 passed` switch set. Refs #2547. Refs #2559. Refs #2552. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code]

mudler added 13 commits September 1, 2026 23:36

localai-org-maint-bot mentioned this pull request Sep 2, 2026

diag(MODEL-MM-QWEN4-EXP): the first CUDA/CPU prefill divergence is the GDN block, and it mirrors vLLM rather than breaking it #2554

Merged

mudler merged commit bb78d1e into main Sep 2, 2026
24 of 28 checks passed

localai-org-maint-bot mentioned this pull request Sep 2, 2026

main is RED: kMaxRank bounds dense_attn::MakeTensor at 4, and two qwen4_exp CPU cases construct rank-5 KV buffers #2559

Closed

localai-org-maint-bot mentioned this pull request Sep 2, 2026

diag(MODEL-MM-QWEN4-EXP): the MoE expert selections DIFFER, and a third of the boundaries are exact bf16 ties (#2552) #2556

Merged

localai-org-maint-bot mentioned this pull request Sep 2, 2026

test_qwen4_exp_layer_loop reintroduces two rank-5 paged-KV fixtures past the #2541 rank bound, and build-test-cpu is red on main #2560

Closed

Navigation Menu

fix(MODEL-MM-QWEN4-EXP): decode on the DEVICE identifiers, so a CUDA step stops feeding itself token id 0 - #2550

fix(MODEL-MM-QWEN4-EXP): decode on the DEVICE identifiers, so a CUDA step stops feeding itself token id 0#2550
mudler merged 13 commits into
mainfrom
row/QWEN4EXP-DECODEDIV-2496

localai-org-maint-bot commented Sep 2, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

localai-org-maint-bot commented Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

The measurement that named it

The fix

The gate runs on a CPU QUEUE

On device, and what it does NOT claim

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

localai-org-maint-bot commented Sep 2, 2026 •

edited

Loading