Repository navigation
D3D12 backend v0.3.1 - faster single-token generation
ggml-d3d12 : v0.3.1 - queue small uploads without a CPU wait, tail readback of the logits
- Faster single-token generation (tg) on small and mid-size models: small uploads (up to 256 KB: token ids, positions, masks) are queued without waiting for the GPU, and the logits are copied behind the graph so reading them back is a memcpy.
GGML_D3D12_NO_SMALL_UPLOAD=1andGGML_D3D12_NO_TAIL_READBACK=1turn the two paths off.- The zip ships a warmed shader cache (
d3d12-shader-cache, 1303 files).
Measured with llama-bench on a Radeon AI PRO R9700 (tg64, mean of 2 runs, v0.3.0 -> v0.3.1; pp128 within +-2 %):
| model | v0.3.0 | v0.3.1 |
|---|---|---|
| qwen2.5-0.5b Q4_0 | 168.2 | 237.3 (+41 %) |
| gemma-4-E2B Q4_0 | 83.5 | 95.1 (+14 %) |
| Ministral-3-3B Q6_K | 74.3 | 89.3 (+20 %) |
Greedy text (32 tokens) identical to v0.3.0 for qwen2.5-0.5b and Ministral-3-3B.
Tested: all 7 exes of the zip ran once on the R9700 (perplexity qwen2.5-0.5b Q4_0 17.2885, Bonsai-8B Q1_0 12.4120). test-backend-ops, 0 failures: R9700 ADD RMS_NORM SOFT_MAX GET_ROWS UPSCALE CPY ROPE SET_ROWS 1820/1820 and 51 further op groups 4259/4259; Iris Xe MUL_MAT_ID 943/943, MUL_MAT 2519/2519, FLASH_ATTN_EXT 5308/5308. Not run: zip smoke on Intel and on the MTT S80.