Skip to content

D3D12 backend v0.3.8 - tiled matmul for any K, faster Moore Threads prompt, int8-dot prompt matmul

Choose a tag to compare

View all tags
@koza-coder koza-coder released this 25 Oct 14:00
· 927 commits to main since this release
v0.3.8
b1b3175
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

ggml-d3d12 : v0.3.8 - tiled matmul accepts any K for F32/F16 weights

  • Matmuls with F32 or F16 weights whose K (the shared dimension) is not a multiple of 32 and with at least 16 columns used the one-thread-per-output scalar kernel. Small robot-policy models (vla.cpp: ACT, Octo, TurboVLA, Evo-1) have such matmuls everywhere (K = 21, 261, 535, 1025 ...). The tiled prompt kernel now handles them: the last K tile is loaded with guarded scalar loads and zero padding (-DK_TAIL in mul_mat_tiled.hlsl). Quantized weights and K multiples of 32 take the same path as in v0.3.7, so LLM speed is unchanged.
  • Default gate: K % 4 != 0. GGML_D3D12_TILED_TAIL=2 uses the tiled kernel for every K that is not a multiple of 32, GGML_D3D12_TILED_TAIL=0 turns it off.
  • The zip ships the warmed shader cache of v0.3.1 (d3d12-shader-cache); the new K-tail shader variants compile on first use.
  • Also in this pull request since the published v0.3.1 (versions v0.3.2 to v0.3.7): Moore Threads prompt speed-ups, the Q4_0/Q8_0 big tile with int8 dot products, the prompt flash attention opened for every GPU, Intel fixes (see the commit messages of the seven commits).

Measured with vla.cpp robot models (median warm predict, two runs each way, "was" = the v0.3.7 code plus a tiny vectorised ADD/MUL change that gains under 1 %):

GPU / model was now
Radeon AI PRO R9700, ACT 98.3 / 98.8 ms 12.6 / 12.6 ms (rows equal)
Radeon AI PRO R9700, Evo-1 840.6 / 846.9 ms 590.7 / 594.6 ms (-30 %)
Radeon AI PRO R9700, TurboVLA 30.5 / 29.6 ms 24.1 / 24.0 ms (rows equal)
Radeon AI PRO R9700, Octo-Small 25.0 / 24.8 ms 22.4 / 21.7 ms
Radeon AI PRO R9700, SmolVLA (K multiple of 4) 161.6 / 161.1 ms 161.4 / 167.9 ms (unchanged)
Intel Iris Xe, Evo-1 34.3 s 15.7 s (-54 %)
Intel Iris Xe, Octo-Small 173.1 / 172.9 ms 82.8 / 83.2 ms
Moore Threads MTT S80, ACT 219.2 ms 65.9 ms
Moore Threads MTT S80, Evo-1 7894 / 7889 ms 5138 / 5136 ms (-35 %)

Correctness: 704 new MUL_MAT test cases (F32 and F16 weights, K = 1..1025 with K % 32 != 0, 16 to 256 rows, 1 to 100 columns, batches, broadcast, permuted) pass 704/704 on the Radeon R9700, Intel Iris Xe and Moore Threads S80 (test-backend-ops, D3D120 against the CPU). Zip smoke: all 7 exes of this exact zip, plus two exit-shape bench runs and a Bonsai-8B-Q1_0 perplexity run, ran on the Radeon with 0 failed runs (qwen2.5-0.5b Q4_0 PPL 17.3638, Bonsai-8B-Q1_0 PPL 12.4121). Not run with this exact zip: Intel and S80 (boxes off after the test run); the code is identical to the tested builds.

Assets 5
Loading