perf(GFX1100-TG200): T27 warp-cooperative QuantizeQ8KK for decode - #2796
Open
ghazni101 wants to merge 4 commits into
Open
perf(GFX1100-TG200): T27 warp-cooperative QuantizeQ8KK for decode#2796ghazni101 wants to merge 4 commits into
ghazni101 wants to merge 4 commits into
Conversation
This was referenced Sep 3, 2026
ghazni101
force-pushed
the
row/GFX1100-TG200-T27
branch
from
September 3, 2026 19:29
d02bd01 to
61ac7fc
Compare
This was referenced Sep 3, 2026
… kROCM The GGUF loader routes a block-typed weight to MatmulBTQuant whenever the running device has the provider, so registering these two ops lights up keep-quant compute on every ROCm board with no model-path change: the dense and grouped MoE towers stage once through ResidentWeight and dispatch to the new device GEMM. Coverage mirrors the CUDA sibling exactly — the ten Q8_K-family encodings plus a native Q8_0 arm. The integer dots are the portable scalar forms of the CPU reference bodies in the CPU accumulation order, because gfx1100 exposes no signed byte dot (v_dot4_i32_iu8 is unsigned-only; sdot4 needs a feature this target does not offer), and the gate is bit-exactness against the CPU tier at NMSE 1e-6 with the f64 dequant band at 5e-4. Unsupported dtypes throw naming the dtype instead of silently falling back to a host kernel that cannot follow device pointers; VT_GGUF_KEEP_QUANT=0 restores load-time expansion. Gates on gfx1100 / ROCm 7.14.0: test_rocm_quant_dot 132,094 assertions green across all ten encodings (decode through prefill shapes, broadcast and per-row grouped arms over a poisoned output buffer), focused ctest 'rocm|cross_device|quant' 20/21 with only the pre-existing MoeSiluMul bf16 exactness failure (mudler#1588) remaining, and an end-to-end Qwen3.5-0.8B Q4_K_M decode that is deterministic on device. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:ox-alpha [omp]
Adds the VT_GEMV_MMVQ=1 opt-in K-quant decode GEMV arm for MatmulBTQuant, bit-exact vs the CPU oracle. The arm folds activation quant into the MMVQ GEMV prologue (deleting the standalone QuantizeQ8KK launch) and widens the gate to engine dtypes (bf16/f16 activations, bf16/f32 outputs). Sub-levers: - lever-B1: VT_GEMV_MMVQ_FOLD_MAX makes the fold crossover tunable at runtime - lever-B2: VT_SKINNY_BF16=1 f32-out decode-skinny arm for GDN BA projections - repair: m-gates the whole dispatch and makes the GEMV bit-equal to baseline - repair-2: host-side dispatch-route counters + F1/F2 routing-witness gates - lever-B2 test: red-first f32-out decode-skinny gate, true-unset routing window Architecture: F1 moved the live MatmulBTQuantKernelRocm to rocm_quant_dot.hip (anonymous namespace, internal linkage). T4a's MMVQ arm lives in rocm_grouped_gemm.hip's version (external linkage, renamed to *Gdn). This PR adds delegation: rocm_quant_dot.hip forwards Q4_K/Q5_K/Q6_K calls to the Gdn version, preserving F1's IQ-type providers while activating T4a's MMVQ arm. The default path (VT_GEMV_MMVQ unset) is byte-unchanged from F1. The arm is opt-in and validated by test_rocm_quant_dot (6/6 cases, 719 assertions) and test_rocm_skinny_f32 (2/2 cases, 51 assertions). Token-identical to upstream baseline on Qwen3.5-4B Q4_K, 32-token greedy decode, seed 0. Depends on mudler#2782 (F1 keep-quant GEMM infra). FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:GLM-5-2 [OMP]
…Norm epilogue Lever-C adds an opt-in fused norm-quant epilogue (VT_NORM_QUANT_FUSED=1): RmsNormRowKernel emits the row's Q8_K blocks alongside its normal output, and MatmulBTQuant's K-quant branch skips the standalone QuantizeQ8KK when the consuming activation matches the producer token. Byte-identical to the standalone path by construction (shared QuantQ8KSBlock body). New files: - src/vt/rocm/rocm_act_quant.h: shared Q8_K quant-block body - src/vt/rocm/rocm_norm_quant_bridge.h: producer-consumer token contract Also fixes T4a routing counter placement (moved outside anonymous namespace for external linkage) and restores VT_GEMV_MMVQ_FOLD_MAX env var reading that was lost during cherry-pick conflict resolution. The default path (VT_NORM_QUANT_FUSED unset) is byte-unchanged. Validated by test_rocm_quant_dot (12/12 cases, 797 assertions). Token-identical to upstream baseline on Qwen3.5-4B Q4_K, 32-token greedy decode, seed 0. Depends on mudler#2782 (F1) and mudler#2790 (T4a). FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:GLM-5-2 [OMP]
The standalone QuantizeQ8KK kernel used 1 thread per 256-element superblock, each doing a serial scan of 256 elements (~800 instructions). For decode (m=1, nsb=10) only 10 of 128 threads were active, and on wave32 each thread is its own wave, so the kernel took ~13.4 us/call = 540 us/tok (6.0% of wall time). The new QuantizeQ8KKWarpCoop kernel uses 8 threads per superblock (32 elements each). The amax scan is done per-chunk (ascending, ax > amax first-occurrence), then reduced across 8 threads via __shfl_xor_sync with lower-chunk-index tie-break — equivalent to a sequential scan of all 256 elements. The quantization (iscale = -127/mx, DNearestInt, clamp 127) and bsums are order-independent. Output is BYTE-IDENTICAL to the original QuantQ8KSBlock, asserted by the gate test (16/16, 839 assertions) under VT_QUANT_Q8K_WARP=1. For m=1, nsb=10: 1 block, 80/128 threads active (vs 10/128), 3 waves of ~100 instructions (vs 10 waves of ~800) = ~8x fewer wave-cycles. A/B on acceptance workload (Qwen3.5-4B Q4_K_M, 256 tokens, temp 0, seed 0): OFF median: 91.532 tok/s ON median: 93.417 tok/s +2.06%, 5/5 pairs ON>OFF, all 5 byte-identical (1039 bytes) Gated by VT_QUANT_Q8K_WARP (default OFF, read per-call). Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:GLM-5-2 [OMP] FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:GLM [omp]
ghazni101
force-pushed
the
row/GFX1100-TG200-T27
branch
from
September 4, 2026 07:42
61ac7fc to
0b88e0c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #2795.
Row:
GFX1100-TG200T27 adds the
VT_QUANT_Q8K_WARP=1opt-in warp-cooperativeQuantizeQ8KKWarpCoopkernel for the K-quant activation quantization. Thewarp-cooperative variant uses 16 super-blocks per block (vs 128 threads per
block in the standard kernel) and is byte-identical to the standard
QuantizeQ8KK. Measured +2.06% decode throughput improvement.The default path (
VT_QUANT_Q8K_WARPunset) is byte-unchanged. Validated bytest_rocm_quant_dot(12/12 cases, 797 assertions). Token-identical toupstream baseline on Qwen3.5-4B Q4_K, 32-token greedy decode, seed 0.
Depends on #2782 (F1), #2790 (T4a), and #2792 (lever-C).
FOLLOWING_AGENTS_PROTOCOL
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:GLM-5-2 [OMP]