Skip to content

b10669

Pre-release
Pre-release

Choose a tag to compare

@github-actions github-actions released this 28 Aug 12:44
be87620

sycl: bind the f16 KV cache in place for the oneDNN SDPA path (#27468)

Measured at a live KV length of 34816 (32768 depth plus one 2048 ubatch),
on Qwen3.8 27B Q4_K_S:

per tensor 4 * 34816 * 256 * 2 B = 71.3 MB
staged per call K and V, so 2x = 142.6 MB
traffic per call read once, write once = 285.2 MB
traffic per ubatch 285.2 MB * 16 calls = 4.56 GB

One ubatch is one ggml_cgraph submission (llama_context::process_ubatch ->
graph_compute), so that 4.56 GB is the cost of a single 2048-token prefill
chunk, and it scales with the live KV length: the first ubatch of the same run,
at seq = 2048, moves 0.27 GB.

Reproduce the two measured inputs with:

GGML_SCHED_DEBUG=2 llama-bench -m MODEL -p 8 -n 0 -r 1 -ngl 0
-fa on -ctk f16 -ctv f16 -v > nd.txt 2>&1
grep -E 'n_layer|n_head_kv|n_embd_head_k' nd.txt
awk '/node # 0 /{g++} g==1 && /(FLASH_ATTN)/{n++} END{print n+0}' nd.txt

Website:

Attestations:

macOS/iOS:

Linux:

Android:

Windows:

openEuler:

  • DISABLED
  • openEuler x86 (310p)
  • openEuler x86 (910b, ACL Graph)
  • openEuler aarch64 (310p)
  • openEuler aarch64 (910b, ACL Graph)

UI: