Skip to content

D3D12 backend v0.3.1 - faster single-token generation

Choose a tag to compare

@koza-coder koza-coder released this 22 Oct 21:47
· 830 commits to main since this release
2cc59d2

ggml-d3d12 : v0.3.1 - queue small uploads without a CPU wait, tail readback of the logits

  • Faster single-token generation (tg) on small and mid-size models: small uploads (up to 256 KB: token ids, positions, masks) are queued without waiting for the GPU, and the logits are copied behind the graph so reading them back is a memcpy.
  • GGML_D3D12_NO_SMALL_UPLOAD=1 and GGML_D3D12_NO_TAIL_READBACK=1 turn the two paths off.
  • The zip ships a warmed shader cache (d3d12-shader-cache, 1303 files).

Measured with llama-bench on a Radeon AI PRO R9700 (tg64, mean of 2 runs, v0.3.0 -> v0.3.1; pp128 within +-2 %):

model v0.3.0 v0.3.1
qwen2.5-0.5b Q4_0 168.2 237.3 (+41 %)
gemma-4-E2B Q4_0 83.5 95.1 (+14 %)
Ministral-3-3B Q6_K 74.3 89.3 (+20 %)

Greedy text (32 tokens) identical to v0.3.0 for qwen2.5-0.5b and Ministral-3-3B.

Tested: all 7 exes of the zip ran once on the R9700 (perplexity qwen2.5-0.5b Q4_0 17.2885, Bonsai-8B Q1_0 12.4120). test-backend-ops, 0 failures: R9700 ADD RMS_NORM SOFT_MAX GET_ROWS UPSCALE CPY ROPE SET_ROWS 1820/1820 and 51 further op groups 4259/4259; Iris Xe MUL_MAT_ID 943/943, MUL_MAT 2519/2519, FLASH_ATTN_EXT 5308/5308. Not run: zip smoke on Intel and on the MTT S80.