Skip to content

D3D12 backend v0.4.0 - faster small batches and MoE routing on the MTT S80

Choose a tag to compare

@koza-coder koza-coder released this 01 Nov 22:18
· 491 commits to main since this release
b8f48fd

ggml-d3d12 : v0.4.0 - faster small batches and MoE routing on the Moore Threads MTT S80
MTT S80: batches of 2-15 tokens (multi-token prediction drafts, speculative decoding, short prompts) of Q4_K / Q6_K and the other types without a multi-column kernel now run the one-column matvec once per column instead of the generic column loop: 2.7-2.9x faster. On by default only on the S80 (vendor 0x1ED5); GGML_D3D12_MV_PERCOL=N sets the column limit on any GPU, 0 turns it off.
All GPUs: the MoE router of models with sigmoid gating, expert bias and weight norm (LFM2 MoE and similar) runs as one fused topk_moe dispatch instead of 7-10 small ones (port of the upstream Vulkan TOPK_MOE_SIGMOID_NORM_BIAS fusion).
Still on upstream llama.cpp master caa8191 (2026-10-26), like v0.3.9.
Measured on the Moore Threads MTT S80 (driver 32.0.2609.1, test build with AVX1 CPU code, llama-bench, -ngl 99):
Qwopus3.5-9B-Coder-MTP Q4_K_M, time per batch: 2 tokens 319 -> 108 ms, 4 tokens 506 -> 191 ms, 8 tokens 986 -> 356 ms; generation (tg128) 15.4 t/s, unchanged.
LFM2-24B-A2B Q4_K_M: GPU dispatches per token 1081 -> 853; generation (tg64) +6-7 % (66.6 vs 71.5 ms per token, 8 alternating runs); 2 tokens 139.7 -> 113.1 ms, 4 tokens 239.8 -> 200.4 ms.
Note for LFM2-24B-A2B Q4_K_M on the 16 GB S80: it needs 13897 MiB of VRAM while Windows grants one process about 12970 MiB; keeping the experts of 4-6 layers on the CPU (llama.cpp --n-cpu-moe 4) costs nothing (tg64 16.05 t/s at 4, 15.6 t/s at 6) and brings it near or under the limit.
Op tests on the S80 (test-backend-ops, D3D120, 4 workers): MUL_MAT 2520/2520, MUL_MAT_ID 1107/1107, TOPK_MOE 576/576, FLASH_ATTN_EXT 5368/5368, ADD, MUL, RMS_NORM, GET_ROWS, SET_ROWS, CPY, SOFT_MAX, ROPE, SSM_CONV, CONCAT, SWIGLU, GEGLU, REGLU, SWIGLU_OAI, GEGLU_ERF, GEGLU_QUICK all pass.
Not run on the Radeon AI PRO R9700 and the Intel Iris Xe for this version (both offline); the small-batch change is off there by default, the router fusion is new for every GPU.