Repository navigation
Replies: 1 comment 2 replies
|
Is there something already in works for this? #31234 |
2 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
I run OpenCode for coding and regularly runs some agents in parallel against a single llama-server on a rack of cheap 16 GB cards, 3 RTX 2000 Ada in a PCIe 4.0 PLX88096 four-slot GPU expansion board using P2P (the 4th slot got filled recently). I prefer every job to get the full 262k context, not 262k÷4 so I use --kv-unified, and the shared pool pushes me to quantized KV (q8_0, then f8_e4m3 recently), the docs promise "same speed, less memory". It ran. I used it for quite some time.
Then I actually timed it
Cold-cache decode at long context sizes, on master, and my config turns out to be the very slow, slower than f16 KV on the same build on every model, and (except Nemotron, whose non-unified vec path is just that slow at head 128) slower than the same q8_0 KV without unified:
Up to ~35 % of my decode, burned for doing exactly what the docs recommend, on CUDA, np1, no-spec, with plenty of VRAM headroom (not #17033, not #30276). The why seems to be the unified decode ops carry a kv_rows index list, and the flash-attn vec kernel can't read it, so quantized-KV decode gets shoved onto the MMA path, which dequantizes K/V into f16 scratch on every single call, that copy eats you alive at depth. Proof it's exactly this: with f16 KV the routing is totally neutral (kvu ≡ nkvu, ±0.1 %), dropping --kv-unified brings the vec path back and with it most of the speed, and the penalty just grows with context.
The fix, and it's worth a lot:
diabluz/llama.cpp -
feat/fused-quant-kv-mmaPR at #31802
kv_rows(~40 lines, 2 files, the CUDA flash attention inefficiency in quantized KV cache handling, worsening with context length #29371 idea) for the fused-ineligible majority (head ≠ 256-ish or gqa ≤ 4): Mistral-Small-24B kvu decode +34.6 % on q8_0 and +49.1 % on f8_e4m3 at 48k.How the numbers are measured
Python driver boots a fresh llama-server per leg and fires greedy-128 requests at
/completionwithcache_prompt: false, and I take the timings from the server's owntimingsblock for different prompt/context sizes. Depth prompts are slices of an in-domain prose corpus cut to exact token counts withllama-tokenizeon each model's own tokenizer, and per-GPU VRAM is sampled at 0.5 s withnvidia-smiand windowed to each request. Output texts are sha256-hashed to prove the byte-identity claims (240/240 rows match master wherever the new code doesn't fire), kernel correctness is a standalone unit harness vs a CPU reference plustest-backend-ops, and the decode deltas were cross-checked against NVIDIA aiperf on the deployed presets and manually.I added dome plots to the PR discussion, Blackwell card results goes in soon, will rerun the benchmarks but happy if anybody else has the chance to try the branch out, to me looks like a solid case and worth to integrate to the upstream but happy to learn as well if I'm doing anything wrong/suboptimal.
For the tests I used:
CUDA_VISIBLE_DEVICES=0,1,2 (Mistral: 0,1) CUDA_SCALE_LAUNCH_QUEUES=2x
llama-server -m
-c 262144 (nemotron 131072, mistral 65536)
-ngl -1 -fa on [--kv-unified]
-ctk <q8_0|f8_e4m3|f16> -ctv <same|q4_0>
-np 1 -b 2048 -ub 512 (35b/gemma: 1024)
--split-mode tensor -ts 1,1,1 (nemotron: layer; mistral: -ts 1,1)
--fit off -lm none
--threads 16 --port 11437 --no-webui --log-verbosity 4
Requests: /completion, temperature 0.0, n_predict 128, cache_prompt false, ignore_eos true.
Models used:
All reactions