feat(rocm): MoeRouterTopK bf16 logits + MoeSiluMul — the MoE-path ops the GDN slice surfaced (issue #41) - #348
Merged
Conversation
…spike (mudler#41) The Qwen3.5-0.8B M2 blocker on discrete ROCm (throws at op 77; the reference tier cannot install on a dGPU) needs ten GDN ops with no ROCm kernel. This commit is the spike spec and the claim; implementation PRs follow one family per PR in throw-order (state-IO, conv, norm-gate, core, fused). Pre-claim validation on 4x gfx1100 / ROCm 7.14 (scratch, recorded in the spec): all ten kernels hand-translated from the CUDA donors and 68/68 standalone checks green (state paths and dtype conversions bit-exact vs an independent host reference); the drop-in TU compiles with the exact production flags; five red-first cross-device cases run 16/16 against the real library with unregistered ops skipping correctly. The key design finding: registering the full indexed set flips IndexedGdnOpsNative(kROCM) true, so the model takes the CUDA lane's device-resident path with zero model or runner edits. M2 disposition follows the mudler#269/mudler#273-ratified distributional gate for the near-tie regime. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: pi:kimi-k3 [pi]
…1 of the gfx1100 GDN slice (mudler#41) The current Qwen3.5-0.8B M2 throw on discrete ROCm (op 77). Hand-translated from the cuda_gdn.cu:248-335 donors, gated by the new cross-device case: bit-exact vs the CPU oracle across f32/f16/bf16 cache arms, uniform and spec-widened layouts, and the has_initial_state i8/i32/absent forms (the bar is byte equality — indexed data movement plus the boundary RNE conversion both sides share). Evidence (4x gfx1100, ROCm 7.14, Release): - test_backend_cross_device GDN state case: 48/48 assertions (runs, not skips) - ctest -R 'rocm|cross_device': 3/3 - full ctest: same 7 failures as the unmodified base 5812b8b (pre-existing host/lane state — async-scheduling expectations on the ROCm lane and the GDN model tests that fail AT op 77 on base; on this branch the test_loaded_engine_dense throw advances 77 -> op 6) - M2 rerun (Qwen3.5-0.8B): op 77 now resolves vt-native; the throw advances to op 5 (kCausalConv1dFwd, family 2) FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: pi:kimi-k3 [pi]
…ly 2 of the gfx1100 GDN slice (mudler#41) Hand-translated from the cuda_gdn.cu scalar donors (:479/:884). The exact-chunks descriptor form IS supported — the first draft's loud VT_CHECK against it fired in the M2 rerun, proving Qwen3.5 prefill passes VT_CONV_EXACT_CHUNKS descriptors on the live path; the kernel takes the donor Reg kernel's chunk addressing over the scalar body (same tap order; write-back owned by the block whose chunk reaches the sequence end). The register-resident/tiled CUDA perf variants stay unported (docs/ROCM.md §6). Evidence (4x gfx1100, ROCm 7.14, Release): - conv cross-device case: 33/33 assertions (legacy + descriptor arms; outputs NMSE, state write-back/roll bit-exact) - ctest -R 'rocm|cross_device': 3/3 - full ctest: same 7 pre-existing failures as base (zero delta) - M2 rerun (Qwen3.5-0.8B): ops 5/6 resolve vt-native; the throw advances 5 -> op 65 (kGdnPostConv, the fused post-conv glue — family 3, reordered to throw-order) FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: pi:kimi-k3 [pi]
… gfx1100 GDN slice (mudler#41) The VT_GLUE_FUSE default path (conv-split + q/k l2norm + g/beta in one launch), hand-translated from cuda_gdn.cu:1155. Sequential per (token, head) threads matching the CPU oracle's reduction order; the donor's block-tree reduction and Fast/Split variants are M5 levers. Reordered to throw-order: this is the op the M2 rerun hit after the conv family landed. Evidence (4x gfx1100, ROCm 7.14, Release): - postconv cross-device case: 16/16 assertions, runs not skips (padded a/b row strides; q/k/v/g/beta all compared) - ctest -R 'rocm|cross_device': 3/3 - full ctest: same 7 pre-existing failures as base (zero delta) - M2 rerun (Qwen3.5-0.8B): op 65 resolves vt-native; the throw advances 65 -> op 10 (kGdnPrefill, the recurrence — family 4) FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: pi:kimi-k3 [pi]
…ily 4 of the gfx1100 GDN slice (mudler#41) The recurrence core over the portable scan (one kernel serves both ops: qsl != null => varlen prefill, qsl == null => single-step decode with optional state_idx + NULL-block zero-out), hand-translated from the cuda_gdn.cu:1856 donor. The WMMA chunked prefill and fused/packed/Triton-AOT decode machinery is NVIDIA-only and stays unported (docs/ROCM.md §6) — the scan is the correctness baseline; perf variants are M5 levers. Evidence (4x gfx1100, ROCm 7.14, Release): - recurrence cross-device case: 35/35 assertions, runs not skips (prefill two-sequence state carry; decode compact + indexed + NULL-slot zero-out; out AND in-place state compared) - ctest -R 'rocm|cross_device': 3/3 - full ctest: same 7 pre-existing failures as base (zero delta) - M2 rerun (Qwen3.5-0.8B): op 10 resolves vt-native; the throw advances 10 -> op 9 (kRmsNormGated — family 5, the final slice) FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: pi:kimi-k3 [pi]
…ities — family 5, completing the gfx1100 GDN slice (mudler#41) The final three ops Qwen3.5-0.8B calls (kRmsNormGated, kSigmoidGateBf16, kAttnQkNormRopeGate), plus the capability seam the live model path forced: the model allocates the conv state bf16 (vLLM mamba_cache_dtype default) and the validators gated that on a hard-coded kCUDA. Adds Backend::SupportsCompressedGdnState (default false), converts CheckGdnCommon to the capability query (the pattern CheckConvCommon already used), keeps CUDA on exactly its prior branch, and advertises both capabilities on ROCm — the registered conv-update and scan kernels carry the compressed-state arms. Shared-file touches are behavior-preserving for every existing backend. Evidence (4x gfx1100, ROCm 7.14, Release): - the five GDN cross-device cases: 162/162 assertions, all run (none skip) - ctest -R 'rocm|cross_device': 3/3; full ctest: same 7 pre-existing failures as base (zero delta) - M2 E2E COMPLETE: Qwen3.5-0.8B runs end to end on discrete ROCm, every op vt-native, deterministic 3/3. 'The sky is' holds 8/8 CPU parity (11/16 at longer horizon). Open prompts diverge at token 1-3 into list-style tokens; VT_GDN_STATE_BF16=0 shifts but does not close it, so it is not a state- rounding artifact. Strict CPU token parity NOT claimed: consistent with the flat-top-logit near-tie regime joral measured on mudler#269, but the GDN-path characterization (per-layer drift or a K-run oracle) is the named open follow-up, not a settled conclusion. - Named next blocker (MoE-bearing paths): the mudler#140 MoeRouterTopK ROCm kernel admits f32 logits only; test_bench/test_loaded_engine_dense now fail there instead of at op 77. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: pi:kimi-k3 [pi]
… the GDN slice surfaced (mudler#41) The mudler#140 router kernel admitted f32 logits only; the live GDN-MoE model path feeds bf16. Adds the bf16-logits arm (upcast at the boundary; softmax stays f32) and the companion MoeSiluMul elementwise op, fully generic over the three f32/bf16 dtype slots after the first same-dtype guard caught a real mixed call. Both gated vs the CPU oracle in new cross-device cases (router: weights NMSE + selected-experts EXACT; silu-mul bf16 arm bit-exact). Evidence (4x gfx1100, ROCm 7.14, Release): - router + silumul cross-device cases: 19/19 assertions, run not skipped - ctest -R 'rocm|cross_device': 3/3; full ctest: same 7 pre-existing failures as base (zero delta) - engine-test throw progression on this branch: op 77 (base) -> router dtype -> op 53 (MoeSiluMul) -> op 62 (kSharedExpertGate, next named MoE blocker; the grouped quant expert GEMM behind it is the real project) FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: pi:kimi-k3 [pi]
This was referenced Aug 11, 2026
Open
Collaborator
|
The whole stack (#334, #336, #341, #343, #345, #348) landed on main, each as its own merge commit so your per-family commits and authorship stay intact. It reviewed well and the reason is the shape of it. Five new ROCm-only TUs compiled solely under The one shared-seam change is the right shape. Conflicts on landing were all additive-vs-additive and resolved by union: the ROCm source list in Two follow-ups, neither blocking:
Reviewed and landed with Claude Code. |
4 tasks
localai-bot
pushed a commit
that referenced
this pull request
Aug 11, 2026
… I/O (#334) The spec for the gfx1100 GDN slice, plus family 1: kGdnStateGather / kGdnStateScatter, the indexed recurrent-state I/O the rest of the slice reads and writes through. First of five stacked PRs (#334, #336, #341, #343, #345), all landing in this session; #348 follows with the MoE-path ops the slice surfaced. New TU under src/vt/rocm/ compiled only under VLLM_CPP_HIP, plus its cross-device test coverage, which runs on every backend that registers the ops. Refs #334, #41. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude:claude-opus-5 [ClaudeCode]
localai-bot
pushed a commit
that referenced
this pull request
Aug 11, 2026
The GDN slice's NOW.md row pushed the file to 6019 characters, over the 6000-character budget check-now-current enforces (in agent-preflight and in CI). The row keeps what changed — #140, gfx1200 M0-M4 (#269), the GDN slice and MoE ops (#334-#348), Qwen3.5-0.8B running e2e all-native — and drops the restatement; the next-step cell names the two things actually open, strict CPU parity and the GDN drift characterization. python3 scripts/check-now-current.py OK (6019 > 6000 before) Refs #41. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude:claude-opus-5 [ClaudeCode]
mudler
added a commit
that referenced
this pull request
Aug 11, 2026
…Rs conflict (#364) (#368) * spec(record): shared record surfaces are a lock, and the lock is the conflicts (#364) 16 of 29 open PRs are CONFLICTING, and 13 of those 16 conflict in bookkeeping files ONLY, with no product code involved. Measured at origin/main d928e2c by running `git merge-tree --write-tree` against every open PR head: coordination.md in 8, NOW.md in 5, roadmap_v1.md in 4, check-public-doc-tables.py in 4, STATUS.md in 4 -- and any src/ or tests/ path in just 3, one of which is a stale mega-branch. Three surfaces are shaped so concurrent PRs must collide, rather than merely risk it. NOW.md is a fixed-size shared buffer sitting at EXACTLY 6000/6000 chars, so adding a row requires evicting another and every PR becomes a read-modify-write of one global -- where the conflict is the lucky outcome, since a clean three-way merge would apply both evictions and both additions and silently drop live rows. STATUS_RATCHET is a hardcoded byte count of a different file that may only fall, so a PR owing STATUS.md one lifecycle line must delete unrelated prose to pay for it; the checker's own comment already records the failure and answered it with slack rather than by removing the coupling. And coordination.md's claims table is insert-at-one-anchor, which is why six ROCm GDN PRs from one author's sequential stack conflict on nothing else. It also contradicts the protocol it serves. AGENTS.md holds that history is git and that there is no state log, yet both claims tables are state logs duplicating `gh pr list`, row/<ID> branch names and issue state. The argument that refuses a waiver registry applies unchanged to a claims registry, and this is the third instance of one failure mode: policy.csv was retired in 0f3e44e and the per-class line budgets on 2026-08-10, both because the gate fired on ordinary work. The exonerated surfaces share one property, one writer per file. Specs are one file per row and took ZERO conflicts across the whole sample, so W1-W5 rewrite the three offenders into that shape or derive them, keep every local per-cell and per-paragraph cap, and record the invariant: no surface that every PR must write. This commit is the spec and the record only. No checker semantic, no doc content and no product source changes; the four counter re-pins below are the existing bump convention for a real new row, each carrying its reason. Filing this one row tripped four separate hardcoded global counters (ENGINE_ROWS, two engine-matrix summary roll-ups, and RUNNABLE_BASELINE), which is the same defect the row exists to remove and is recorded here as evidence rather than worked around. Issue: #364 FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] * fix(record): remove the two global counters that made every PR a shared write (#364) W1, W3, W4 and W5 of the spec. The claims table (W2) follows separately. W1 removes the `chars` key of STATUS_RATCHET. It was a byte count of docs/STATUS.md stored in scripts/check-public-doc-tables.py and allowed to move only downward, so a PR owing the page one lifecycle line had to delete unrelated prose from another row to pay for it AND re-pin the constant -- two shared files per PR, both of which every other PR was also editing. RED-BEFORE measured: the live page was EXACTLY 243245 of 243245 permitted chars, and one ordinary capability line put it 27 over. The cut is narrower than the spec's W1 text, deliberately. The four keys are not alike: `chars` is a LENGTH and moves on every edit, while `h2_sections`, `long_paragraphs` and `oversized_cells` count QUALITY DEFECTS, so an ordinary line moves none of them and two concurrent PRs do not collide on them. Those three are kept and carry the whole anti-decay obligation -- BENCHMARKS.md's 11,127-line decay would have tripped two of them on its own. Deleting them would have dropped a real obligation for no conflict benefit, which is the case the spec's stop condition names. W3 removes MAX_CHARS from check-now-current.py. The tracked digest measured EXACTLY 6000 of 6000: tuned to the byte, no headroom, so a row could only be added by evicting one. MAX_LINES and MAX_ENTRY_CHARS stay and carry the obligation -- a row costs one line rather than a variable number of bytes, and each entry is still capped locally. W4 sorts the roadmap issue table by stable row ID. The 43 rows are reordered and otherwise byte-identical, verified as a set before and after. Appending at one anchor is why concurrent additions collided; sorting puts them in different hunks. W5 records the invariant in AGENTS.md: no surface that every PR must write, and the three shapes that are admissible instead. The new tests/scripts/test_record_merge_shape.py asks git the question directly rather than inspecting a checker, and it CORRECTED a claim I had made in the spec. Removing a budget does NOT on its own make two row additions merge -- two appends collide at the same anchor either way. What the budget added on top was a forced deletion of unrelated content, and that is the part with teeth: it made every PR edit lines it did not own, and it made a SUCCESSFUL merge unsafe, since git resolving two such branches applies both evictions and both victims vanish with no gate noticing. W3 removes the eviction, W4 removes the collision, and neither alone is sufficient. The spec is corrected in the same commit. Gates: preflight rc=0; check_public_doc_tables 52 OK, check_now_current 11 OK, record_merge_shape 8 OK; check-public-doc-tables, check-now-current and check-agent-record all green on the live tree. No product source is touched. Issue: #364 FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] * fix(record): a claim gets its own file, so concurrent claims stop colliding (#364) W2, and the largest single conflict source in the repository: the claims TABLE in coordination.md is insert-at-one-anchor, so every concurrent claim appends a row at the same line. It conflicted in 8 of the 16 conflicting open PRs measured at origin/main d928e2c, six of them (#334 #336 #341 #343 #345 #348) one author's sequential ROCm GDN stack whose ONLY conflict was this file. Their work did not overlap at all. New claims now go in .agents/claims/CLAIM-<ID>.md, one file per claim. parse_active_claims reads that directory AND the legacy table, so both are valid to the gate and the bidirectional cross-check is unchanged. A file with one writer cannot collide -- the shape .agents/specs/ already has, which is why specs took ZERO conflicts across the measured sample. The design differs from the spec's W2, which said to derive claims from open PRs and branch names. Reading the consumers showed that to be wrong twice over, and the spec is corrected rather than the change quietly reinterpreted. Deriving would put a NETWORK CALL inside an offline gate -- check-agent-record.py runs in preflight with no network -- and the table is not redundant with the PR list: check_row_contracts uses it for a bidirectional cross-check, where every SPIKE/ACTIVE matrix row must be claimed by a live claim and every claim must name rows in those states. A PR list cannot supply that, so deriving would have dropped a real guarantee. That is the case the spec's own stop condition names. Deliberately ADDITIVE: not one existing row is migrated. The 115 rows are spread across several interleaved tables, struck-through released entries and prose blocks, some below a later heading -- bulk-rewriting that is how a record gets lost, in a diff no reviewer could check. Rows leave as their claims close and the table empties on its own, while the conflict source is closed for every future claim from the day this lands. RED-BEFORE proven by mutation: with claim_sources reverted to coordination.md alone, tests/scripts/test_record_merge_shape.py goes 2 failures; restored byte-for-byte (md5 b9f325f1238a954001132a19309e7caa before and after) and green again at 11/11. Gates: preflight green apart from the commit-trailer repair in the same push; check-agent-record, claim-view --check-local, check-pr-size 34 all green. Issue: #364 FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] * test(record): mutation evidence in each changed checker's own suite (#364) check-pr-size.py was RIGHT and this is its finding, not a workaround. Three governance checkers changed in this row -- check-agent-record.py (per-claim file source), check-pr-size.py (the .agents/claims path class) and check-gate-commands.py (the RUNNABLE_BASELINE re-pin) -- and the evidence for all three lived in the NEW tests/scripts/test_record_merge_shape.py rather than in each checker's paired suite. The gate requires the paired suite, correctly: that is where a later reader of that checker looks, and a new file is easy to miss or delete wholesale. Added, each with a mutation that proves the assertion is load-bearing: test_agent_record.py -- a claim in its own file registers exactly as a table row does; patching claim_sources back to coordination.md alone HIDES it (the mutation); a claim declared twice across the two sources is still rejected, so reading a second source did not weaken the duplicate check. test_check_pr_size.py -- .agents/claims/CLAIM-*.md classifies as the same class as the per-row spec it mirrors; substituting a never-matching pattern for CLAIM makes classify_path raise (the mutation), proving the clause is not shadowed by a broader rule; and a non-markdown path in the directory still fails closed, so the pattern is not a blanket directory exemption -- which AGENTS.md forbids. test_check_gate_commands.py -- the row is credited runnable and its spec really names failable commands; removing the entry from RUNNABLE_BASELINE must break the exact pin (the mutation), so the baseline is not decorative. Also moved `if __name__ == "__main__"` to the END of test_agent_record.py and test_check_pr_size.py. It sat mid-file, so classes appended after it were defined AFTER unittest.main() had already collected and run -- the suites reported OK while never executing the new tests (34 and 25 before, 38 and 29 after). A green suite that silently skips what was just added is the same class of defect as the rest of this row. Suites: test_agent_record 29, test_check_pr_size 38, test_check_gate_commands 27, test_record_merge_shape 11, all OK. Issue: #364 FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Row
BACKEND-ROCM— small MoE-path addition surfacing from the GDN slice's engine tests (issue #41). Claim:CLAIM-ROCM-GDN-KERNELS. Stacked on #345 (GDN family 5).What changed
MoeRouterTopK(the feat(rocm): gfx1201 hipBLAS ops + Gemma-4-26B-A4B MoE (BF16/FP8) #140 kernel) gains a bf16-logits arm — upcast at the boundary, softmax math stays f32. The live GDN-MoE model path feeds bf16 logits; the f32-only guard threw on it.kMoeSiluMulROCm kernel (elementwisesilu(gate)*up), fully generic over the three f32/bf16 dtype slots — the first same-dtype-only draft caught a real mixed call via its own loud guard.Evidence (4× gfx1100, ROCm 7.14, Release)
ctest -R 'rocm|cross_device': 3/3; full ctest: same 7 pre-existing failures as base (zero delta)MoeSiluMul) → op 62 (kSharedExpertGate)Speed claims
Honest gaps
kSharedExpertGate(62, trivial elementwise),kMoeCombine/kMoeCombineGate, and the grouped quant expert GEMM (kMatmulBTQuantGrouped— the real project, consciously deferred in feat(rocm): gfx1201 hipBLAS ops + Gemma-4-26B-A4B MoE (BF16/FP8) #140's own notes). MoE-bearing models (test_bench,test_loaded_engine_dense, Qwen3.5-27B-class) still throw at op 62 on this branch.