Turboquant is slower for me

[simusid]

I know turboquant is being very actively worked. This was my very first attempt and maybe I did something wrong. I was not able to find a good walkthrough or tutorial. I was hoping for a t/s improvement but instead I saw about a 15% decrease in performance.

I cloned thetom/llama-cpp-turboquant and checked out origin/feature/turboquant-kv-cache. I built it with the exact same cmake settings as my base llama.cpp. I fired up two instances of llama-server and loaded unsloth/qwen3.5-397B-A17B:UD-Q4_K_S. Each instance is using two H200 GPUs. the second instance using turboquant I started with --cache-type-k turbo3 --cache-type-v turbo3.

My regular llama-server got 78 t/s
The turboquant llama-server gets 67 t/s

I get very close to that using turbo2 and turbo4. what have I missed?

[Krysalide]

i also cloned thetom/llama-cpp-turboquant, i don't think using turboquant will increase speed? VRAM used is bigger. Turboquant allows to get bigger context and is not supposed to modify weights. For those who want to test, i found this huggingface space:
https://huggingface.co/spaces/ai-engineering-at/llama-cpp-turboquant-guide
The question of speed remains very interesting and turboquant does not seem to satisfy everybody

[pauldev-hub]

Hi @simusid ,

Your setup and observation make sense, and you haven't necessarily done anything wrong — this is a known gotcha with turboquant on high-bandwidth hardware like H200s.

Why you're seeing a slowdown instead of a speedup:

TurboQuant's KV cache quantization helps most when you're memory-bandwidth bottlenecked by KV cache size — typically on consumer GPUs (A100, 3090, etc.) with tighter VRAM. H200s have 141GB HBM3e with extremely high memory bandwidth (~4.8 TB/s). At that bandwidth level, your KV cache reads are already fast enough that the overhead of dequantizing turbo3 back to fp16 during attention actually costs more than the memory savings gain you. You're compute-bottlenecked on the dequant, not bandwidth-bottlenecked on the cache read.
Why turbo2 and turbo4 are closer to baseline:
turbo2 is more aggressively quantized — smaller cache, but even more dequant overhead
turbo4 is closer to fp16 in precision — less dequant overhead, less savings

turbo3 sits in an awkward middle ground that happens to maximize the cost/benefit ratio in the wrong direction on H200s

What to actually try:

Test with a much longer context (32k–128k tokens). TurboQuant's advantage only appears when KV cache size becomes the actual bottleneck — at short contexts on H200s there's nothing to gain.
Try --cache-type-k turbo4 --cache-type-v q8_0 — asymmetric quantization often works better than symmetric turbo on both sides.

Check if you're actually KV-cache bottlenecked at all — on two H200s with Qwen3.5-397B at Q4_K_S, your weights alone are ~200GB+. At 78 t/s you're likely already compute-bound on the matmuls, not memory-bound on the cache. TurboQuant won't help a compute-bound workload.

The honest answer: TurboQuant is most impactful on mid-range single GPU setups. On dual H200s at short-to-medium context, standard fp16 KV cache is hard to beat.

If this helped you, please mark it as the answer — it helps others in the community who run into the same issue find the solution faster!

[xRebellion]

Turboquant's main selling point (at least, as far as I'm aware) is giving the ability to fit in a bigger context window (kv cache) onto a smaller memory space, while preserving model's quality.

I briefly compared q4_0 and turbo3 within my 4gb vram for qwen3.5-4b. they both allowed me to run it with all layers in vram, but i did see model degradation in q4_0. While on turbo3 it preserved its quality (I don't exactly have a qualitative test for this, just by brief experience)

Basically, it's probably only a big deal if you use mid-lower end hardware (?)
Not sure how it fares on bigger models on higher end GPUs, apart from your experience