feat(KERNEL-QUANT-CIQ-GEMM-ROCM): land the W1 keep-quant providers on kROCM - #2782
Open
ghazni101 wants to merge 1 commit into
Open
feat(KERNEL-QUANT-CIQ-GEMM-ROCM): land the W1 keep-quant providers on kROCM#2782ghazni101 wants to merge 1 commit into
ghazni101 wants to merge 1 commit into
Conversation
ghazni101
added a commit
to ghazni101/vllm.cpp
that referenced
this pull request
Sep 3, 2026
Adds the VT_GEMV_MMVQ=1 opt-in K-quant decode GEMV arm for MatmulBTQuant, bit-exact vs the CPU oracle. The arm folds activation quant into the MMVQ GEMV prologue (deleting the standalone QuantizeQ8KK launch) and widens the gate to engine dtypes (bf16/f16 activations, bf16/f32 outputs). Sub-levers: - lever-B1: VT_GEMV_MMVQ_FOLD_MAX makes the fold crossover tunable at runtime - lever-B2: VT_SKINNY_BF16=1 f32-out decode-skinny arm for GDN BA projections - repair: m-gates the whole dispatch and makes the GEMV bit-equal to baseline - repair-2: host-side dispatch-route counters + F1/F2 routing-witness gates - lever-B2 test: red-first f32-out decode-skinny gate, true-unset routing window Architecture: F1 moved the live MatmulBTQuantKernelRocm to rocm_quant_dot.hip (anonymous namespace, internal linkage). T4a's MMVQ arm lives in rocm_grouped_gemm.hip's version (external linkage, renamed to *Gdn). This PR adds delegation: rocm_quant_dot.hip forwards Q4_K/Q5_K/Q6_K calls to the Gdn version, preserving F1's IQ-type providers while activating T4a's MMVQ arm. The default path (VT_GEMV_MMVQ unset) is byte-unchanged from F1. The arm is opt-in and validated by test_rocm_quant_dot (6/6 cases, 719 assertions) and test_rocm_skinny_f32 (2/2 cases, 51 assertions). Token-identical to upstream baseline on Qwen3.5-4B Q4_K, 32-token greedy decode, seed 0. Depends on mudler#2782 (F1 keep-quant GEMM infra). FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:GLM-5-2 [OMP]
ghazni101
added a commit
to ghazni101/vllm.cpp
that referenced
this pull request
Sep 3, 2026
Adds the VT_GEMV_MMVQ=1 opt-in K-quant decode GEMV arm for MatmulBTQuant, bit-exact vs the CPU oracle. The arm folds activation quant into the MMVQ GEMV prologue (deleting the standalone QuantizeQ8KK launch) and widens the gate to engine dtypes (bf16/f16 activations, bf16/f32 outputs). Sub-levers: - lever-B1: VT_GEMV_MMVQ_FOLD_MAX makes the fold crossover tunable at runtime - lever-B2: VT_SKINNY_BF16=1 f32-out decode-skinny arm for GDN BA projections - repair: m-gates the whole dispatch and makes the GEMV bit-equal to baseline - repair-2: host-side dispatch-route counters + F1/F2 routing-witness gates - lever-B2 test: red-first f32-out decode-skinny gate, true-unset routing window Architecture: F1 moved the live MatmulBTQuantKernelRocm to rocm_quant_dot.hip (anonymous namespace, internal linkage). T4a's MMVQ arm lives in rocm_grouped_gemm.hip's version (external linkage, renamed to *Gdn). This PR adds delegation: rocm_quant_dot.hip forwards Q4_K/Q5_K/Q6_K calls to the Gdn version, preserving F1's IQ-type providers while activating T4a's MMVQ arm. The default path (VT_GEMV_MMVQ unset) is byte-unchanged from F1. The arm is opt-in and validated by test_rocm_quant_dot (6/6 cases, 719 assertions) and test_rocm_skinny_f32 (2/2 cases, 51 assertions). Token-identical to upstream baseline on Qwen3.5-4B Q4_K, 32-token greedy decode, seed 0. Depends on mudler#2782 (F1 keep-quant GEMM infra). FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:GLM-5-2 [OMP]
This was referenced Sep 3, 2026
ghazni101
force-pushed
the
row/GFX1100-TG200-F1
branch
from
September 3, 2026 19:24
dc036a4 to
29eebfc
Compare
ghazni101
added a commit
to ghazni101/vllm.cpp
that referenced
this pull request
Sep 3, 2026
Adds the VT_GEMV_MMVQ=1 opt-in K-quant decode GEMV arm for MatmulBTQuant, bit-exact vs the CPU oracle. The arm folds activation quant into the MMVQ GEMV prologue (deleting the standalone QuantizeQ8KK launch) and widens the gate to engine dtypes (bf16/f16 activations, bf16/f32 outputs). Sub-levers: - lever-B1: VT_GEMV_MMVQ_FOLD_MAX makes the fold crossover tunable at runtime - lever-B2: VT_SKINNY_BF16=1 f32-out decode-skinny arm for GDN BA projections - repair: m-gates the whole dispatch and makes the GEMV bit-equal to baseline - repair-2: host-side dispatch-route counters + F1/F2 routing-witness gates - lever-B2 test: red-first f32-out decode-skinny gate, true-unset routing window Architecture: F1 moved the live MatmulBTQuantKernelRocm to rocm_quant_dot.hip (anonymous namespace, internal linkage). T4a's MMVQ arm lives in rocm_grouped_gemm.hip's version (external linkage, renamed to *Gdn). This PR adds delegation: rocm_quant_dot.hip forwards Q4_K/Q5_K/Q6_K calls to the Gdn version, preserving F1's IQ-type providers while activating T4a's MMVQ arm. The default path (VT_GEMV_MMVQ unset) is byte-unchanged from F1. The arm is opt-in and validated by test_rocm_quant_dot (6/6 cases, 719 assertions) and test_rocm_skinny_f32 (2/2 cases, 51 assertions). Token-identical to upstream baseline on Qwen3.5-4B Q4_K, 32-token greedy decode, seed 0. Depends on mudler#2782 (F1 keep-quant GEMM infra). FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:GLM-5-2 [OMP]
ghazni101
added a commit
to ghazni101/vllm.cpp
that referenced
this pull request
Sep 3, 2026
…Norm epilogue Lever-C adds an opt-in fused norm-quant epilogue (VT_NORM_QUANT_FUSED=1): RmsNormRowKernel emits the row's Q8_K blocks alongside its normal output, and MatmulBTQuant's K-quant branch skips the standalone QuantizeQ8KK when the consuming activation matches the producer token. Byte-identical to the standalone path by construction (shared QuantQ8KSBlock body). New files: - src/vt/rocm/rocm_act_quant.h: shared Q8_K quant-block body - src/vt/rocm/rocm_norm_quant_bridge.h: producer-consumer token contract Also fixes T4a routing counter placement (moved outside anonymous namespace for external linkage) and restores VT_GEMV_MMVQ_FOLD_MAX env var reading that was lost during cherry-pick conflict resolution. The default path (VT_NORM_QUANT_FUSED unset) is byte-unchanged. Validated by test_rocm_quant_dot (12/12 cases, 797 assertions). Token-identical to upstream baseline on Qwen3.5-4B Q4_K, 32-token greedy decode, seed 0. Depends on mudler#2782 (F1) and mudler#2790 (T4a). FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:GLM-5-2 [OMP]
5 tasks
This was referenced Sep 4, 2026
… kROCM The GGUF loader routes a block-typed weight to MatmulBTQuant whenever the running device has the provider, so registering these two ops lights up keep-quant compute on every ROCm board with no model-path change: the dense and grouped MoE towers stage once through ResidentWeight and dispatch to the new device GEMM. Coverage mirrors the CUDA sibling exactly — the ten Q8_K-family encodings plus a native Q8_0 arm. The integer dots are the portable scalar forms of the CPU reference bodies in the CPU accumulation order, because gfx1100 exposes no signed byte dot (v_dot4_i32_iu8 is unsigned-only; sdot4 needs a feature this target does not offer), and the gate is bit-exactness against the CPU tier at NMSE 1e-6 with the f64 dequant band at 5e-4. Unsupported dtypes throw naming the dtype instead of silently falling back to a host kernel that cannot follow device pointers; VT_GGUF_KEEP_QUANT=0 restores load-time expansion. Gates on gfx1100 / ROCm 7.14.0: test_rocm_quant_dot 132,094 assertions green across all ten encodings (decode through prefill shapes, broadcast and per-row grouped arms over a poisoned output buffer), focused ctest 'rocm|cross_device|quant' 20/21 with only the pre-existing MoeSiluMul bf16 exactness failure (mudler#1588) remaining, and an end-to-end Qwen3.5-0.8B Q4_K_M decode that is deterministic on device. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:ox-alpha [omp]
ghazni101
force-pushed
the
row/GFX1100-TG200-F1
branch
from
September 4, 2026 07:34
29eebfc to
73478f6
Compare
ghazni101
added a commit
to ghazni101/vllm.cpp
that referenced
this pull request
Sep 4, 2026
Adds the VT_GEMV_MMVQ=1 opt-in K-quant decode GEMV arm for MatmulBTQuant, bit-exact vs the CPU oracle. The arm folds activation quant into the MMVQ GEMV prologue (deleting the standalone QuantizeQ8KK launch) and widens the gate to engine dtypes (bf16/f16 activations, bf16/f32 outputs). Sub-levers: - lever-B1: VT_GEMV_MMVQ_FOLD_MAX makes the fold crossover tunable at runtime - lever-B2: VT_SKINNY_BF16=1 f32-out decode-skinny arm for GDN BA projections - repair: m-gates the whole dispatch and makes the GEMV bit-equal to baseline - repair-2: host-side dispatch-route counters + F1/F2 routing-witness gates - lever-B2 test: red-first f32-out decode-skinny gate, true-unset routing window Architecture: F1 moved the live MatmulBTQuantKernelRocm to rocm_quant_dot.hip (anonymous namespace, internal linkage). T4a's MMVQ arm lives in rocm_grouped_gemm.hip's version (external linkage, renamed to *Gdn). This PR adds delegation: rocm_quant_dot.hip forwards Q4_K/Q5_K/Q6_K calls to the Gdn version, preserving F1's IQ-type providers while activating T4a's MMVQ arm. The default path (VT_GEMV_MMVQ unset) is byte-unchanged from F1. The arm is opt-in and validated by test_rocm_quant_dot (6/6 cases, 719 assertions) and test_rocm_skinny_f32 (2/2 cases, 51 assertions). Token-identical to upstream baseline on Qwen3.5-4B Q4_K, 32-token greedy decode, seed 0. Depends on mudler#2782 (F1 keep-quant GEMM infra). FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:GLM-5-2 [OMP]
ghazni101
added a commit
to ghazni101/vllm.cpp
that referenced
this pull request
Sep 4, 2026
…Norm epilogue Lever-C adds an opt-in fused norm-quant epilogue (VT_NORM_QUANT_FUSED=1): RmsNormRowKernel emits the row's Q8_K blocks alongside its normal output, and MatmulBTQuant's K-quant branch skips the standalone QuantizeQ8KK when the consuming activation matches the producer token. Byte-identical to the standalone path by construction (shared QuantQ8KSBlock body). New files: - src/vt/rocm/rocm_act_quant.h: shared Q8_K quant-block body - src/vt/rocm/rocm_norm_quant_bridge.h: producer-consumer token contract Also fixes T4a routing counter placement (moved outside anonymous namespace for external linkage) and restores VT_GEMV_MMVQ_FOLD_MAX env var reading that was lost during cherry-pick conflict resolution. The default path (VT_NORM_QUANT_FUSED unset) is byte-unchanged. Validated by test_rocm_quant_dot (12/12 cases, 797 assertions). Token-identical to upstream baseline on Qwen3.5-4B Q4_K, 32-token greedy decode, seed 0. Depends on mudler#2782 (F1) and mudler#2790 (T4a). FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:GLM-5-2 [OMP]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #2781.
Row:
GFX1100-TG200Lands the ROCm keep-quant GEMM infrastructure: the
kROCMprovider forOpId::kMatmulBTQuantandOpId::kMatmulBTQuantGrouped. The provider doesthe dequant-in-kernel dot product on-device, keeping the weight in its
block-quantized form through the GEMM rather than expanding to bf16 at load
time.
VT_GGUF_KEEP_QUANT=0restores load-time expansion.The new provider lives in
src/vt/rocm/rocm_quant_dot.hipand registersitself for
kROCM. The oldMatmulBTQuantKernelRocmregistration inrocm_ops.hip(which routed to the limitedrocm_grouped_gemm.hipQ8_0/Q4_K/Q5_K/Q6_K-only dispatch) is removed so the new provider takes
effect.
Gates on gfx1100 / ROCm 7.14.0:
test_rocm_quant_dot132,094 assertionsgreen across all ten encodings (decode through prefill shapes, broadcast
and per-row grouped arms over a poisoned output buffer). Token-identical
to upstream baseline on Qwen3.5-4B Q4_K, 32-token greedy decode, seed 0.
This is the foundation PR for the GFX1100-TG200 decode campaign. Most
lever PRs depend on the keep-quant GEMM infrastructure it introduces.
FOLLOWING_AGENTS_PROTOCOL
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:GLM-5-2 [OMP]