Problem
When --flash-attn on is used with quantized KV cache types (e.g. --cache-type-k q8_0 --cache-type-v q4_0), and llama.cpp was built without -DGGML_CUDA_FA_ALL_QUANTS=ON, the server silently falls back to CPU-based attention. This results in:
- Dramatically worse prefill speed (25-45x slower in our tests)
- GPU shows 0% utilization during inference
- No visible error or warning to the user
This was confirmed in #20866 and by our testing with Gemma 4 12B QAT + MTP (head_size=512, SWA, Blackwell RTX 5070 Ti).
Suggestion
Two options (either would be a big improvement):
Option A: Enable FA_ALL_QUANTS by default
Add -DGGML_CUDA_FA_ALL_QUANTS=ON to the default CMake configuration. The build time increase is a one-time cost that most users would accept for 25x performance with quantized KV cache.
Option B: Add a runtime warning
In llama-server, when --flash-attn on is set with a quantized KV cache type, check if the appropriate CUDA FA kernel was compiled. If not, emit a clear warning:
Warning: Flash attention with quantized KV cache requires building with -DGGML_CUDA_FA_ALL_QUANTS=ON.
Using CPU fallback for attention which will be extremely slow.
Consider using --cache-type-k f16 --cache-type-v f16 instead.
Our Test Environment
Problem
When
--flash-attn onis used with quantized KV cache types (e.g.--cache-type-k q8_0 --cache-type-v q4_0), andllama.cppwas built without-DGGML_CUDA_FA_ALL_QUANTS=ON, the server silently falls back to CPU-based attention. This results in:This was confirmed in #20866 and by our testing with Gemma 4 12B QAT + MTP (head_size=512, SWA, Blackwell RTX 5070 Ti).
Suggestion
Two options (either would be a big improvement):
Option A: Enable FA_ALL_QUANTS by default
Add
-DGGML_CUDA_FA_ALL_QUANTS=ONto the default CMake configuration. The build time increase is a one-time cost that most users would accept for 25x performance with quantized KV cache.Option B: Add a runtime warning
In
llama-server, when--flash-attn onis set with a quantized KV cache type, check if the appropriate CUDA FA kernel was compiled. If not, emit a clear warning:Our Test Environment