Skip to content

3D12 backend v0.4.4 - faster decode, MTP drafting and prompts on the MTT S80, newer llama.cpp

Choose a tag to compare

View all tags
@koza-coder koza-coder released this 07 Nov 10:12
· 357 commits to main since this release
v0.4.4
d6aad34
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

D3D12 backend v0.4.4 - faster decode and MTP drafting on the MTT S80

Windows x64, DirectX 12. On llama.cpp master 785322d (2026-11-01), as v0.4.3.
Changes since v0.4.1 (v0.4.2 and v0.4.3 were never published; their changes are part of this release).

What is new in v0.4.4

  • Moore Threads MTT S80: faster generation for Qwen models, much faster MTP / speculative drafting.
    • Qwopus3.5-9B-Coder-MTP Q4_K_M with --spec-type draft-mtp, 3 draft tokens: 18.5 -> 22.3 t/s.
    • Qwen3.5-9B Q4_K_M tg64: 15.8 -> 17.4 t/s; Qwen3.8-27B UD-IQ3_XXS tg128 4.72 -> 4.89 t/s.
    • TOP_K with k <= 32 (the MTP draft sampler, top_k 10 over a 248k vocabulary) runs in two passes instead of a full sort (was ~170 dispatches per draft token).
    • Q6_K matrix-vector product for 1-8 tokens loads every weight byte once.

Also in this release (from v0.4.2 / v0.4.3)

  • MTT S80: prompt processing about 40 % faster for IQ models. Qwen3.8-27B UD-IQ3_XXS: pp512 19.3 -> 28.0 t/s, pp4 8.6 -> 10.1 t/s (v0.4.1 -> v0.4.4).
    • The big prompt tile also loads IQ3_XXS, IQ3_S, IQ4_XS, IQ2_S, IQ2_XXS, IQ2_XS and Q2_K weights directly and starts at 16 tokens.
    • Shader buffers are bound through one descriptor table on Moore Threads, as on Intel. Very small models (0.5B) generate about 4 % slower with it; GGML_D3D12_UAV_TABLE=0 switches it off.
  • Batches of 5-8 tokens decode each weight once (v0.4.1: 2-4). Radeon AI PRO R9700, glm-4-9b IQ2_M: pp8 +85 %.
  • Q6_K keeps the faster matrix-vector path up to 8 tokens on Intel and AMD: Iris Xe Ministral-3-3B Q6_K pp5 5.2 -> 15.1 t/s.
  • llama.cpp updated by 124 upstream commits (FP8 E4M3 tensors are reported as unsupported by this backend).

Switches (for A/B tests): GGML_D3D12_TOPK_SMALL=0, GGML_D3D12_Q6K_V2=0, GGML_D3D12_UAV_TABLE=0|1, GGML_D3D12_BT_IQ=0, GGML_D3D12_BT_IQ2=0, GGML_D3D12_BT_MIN=<n>, GGML_D3D12_BT_TN=32|64|128, GGML_D3D12_MV_NC=0.

Tip for MTP models on the MTT S80: --spec-draft-n-max 3 was fastest (5 and 7 draft tokens were slower).

Tested

  • MTT S80: every shipped exe of this zip (llama-server, llama-cli, llama-mtmd-cli, llama-completion, llama-perplexity, llama-bench, test-backend-ops), PPL qwen2.5-0.5b 17.29, LFM2-24B-A2B and Qwen3.8-27B benches; test-backend-ops TOP_K 533/533, ARGSORT 98/98, MUL_MAT 2565/2565 of the same source.
  • Radeon AI PRO R9700: test-backend-ops MUL_MAT 2565/2565, TOP_K 533/533, ARGSORT 98/98 with the same changes; v0.4.3 source 13 op groups 11,643/11,643.

Not tested: Intel op sweep of this exact build, the other GPUs of the backend (Zhaoxin, Lisuan) - please report problems.

Assets 5
Loading