[Feature Request] Sageattention over Flashattention

[SammCheese]

Describe the Issue
As an end-user, i find myself occasionally struggling with low speeds. Flashattention helps, but there is a faster implementation available which suggests 2-5x lossless speedup compared to Flashattention.

Additional Information:
Sageattention is a direct upgrade over the existing implementation and is a worthwhile option to consider.

[LostRuins]

You should make the request upstream in llama.cpp instead - I don't know how to implement it myself

[henk717]

Going to close this as not planned, if its ever in llamacpp we can reconsider.

[SammCheese]

Going to close this as not planned, if its ever in llamacpp we can reconsider.

Yeah. As it seems lcpp doesn't really use FA either but GG's own interprementation of it.

[gustrd]

Seems like SageAttention is only useful for diffusion models, not having gains for LLMs. They have a different type of math, you can check with your favorite AI.

[SammCheese]

Seems like SageAttention is only useful for diffusion models, not having gains for LLMs. They have a different type of math, you can check with your favorite AI.

One quick look at the repo I linked in the original post reveals a clear speedup for LLMs. Doublecheck your sources.

[gustrd]

Quick Gemini Pro 3.1 research:

#########
"When we look beyond the academic benchmarks and at actual production ecosystems, gustrd is actually much closer to the truth. While SammCheese correctly read the authors' claims, gustrd accurately reflects why the open-source community treats SageAttention strictly as a tool for diffusion models right now.

Here is the reality of why serious LLM engines like vLLM have passed on SageAttention, and why it has instead become the darling of the ComfyUI and Stable Diffusion communities:

1. The Causal Masking & Accuracy Crisis

While diffusion models (like Sora, Flux, or Wan2.1) look at the whole image/video at once using bidirectional attention, LLMs predict text step-by-step using causal masking (where the AI is blocked from looking at future words).

In theory, SageAttention supports causal masking. In practice, developers integrating it into LLMs have reported catastrophic accuracy drops. Open issues on the SageAttention GitHub tracker reveal that when applied to LLMs, it regularly causes extreme perplexity degradation (the AI starts generating gibberish) and throws NaN (Not a Number) errors, particularly on newer hardware like Hopper H100 GPUs. Diffusion models simply don't trigger these same mathematical instabilities within the SageAttention kernel.

2. The Bottleneck: Compute vs. Memory Bandwidth

Even if the accuracy bugs were fixed, SageAttention doesn't solve an LLM engine's biggest problem.

• Diffusion Models are Compute-Bound: Generating a video frame requires pushing massive, dense matrices through the GPU cores simultaneously. SageAttention uses 8-bit (and recently 4-bit) quantization to accelerate this raw math, resulting in massive, immediate speedups (often 30%+ faster video generation).
• LLMs are Memory-Bound (During Generation): For an LLM serving multiple users, the bottleneck isn't the raw math of the attention mechanism; it's the memory bandwidth required to constantly move the massive KV Cache (the memory of the conversation) from the GPU's VRAM into the compute cores.

Engines like vLLM rely on highly specialized kernels like PagedAttention or FlashInfer. These are designed specifically to manage memory fragmentation and optimize the KV Cache during the autoregressive "decode" phase. Dropping SageAttention into vLLM would only speed up the initial prompt reading (prefill) but do virtually nothing to speed up the actual text generation, all while risking the accuracy issues mentioned above.

The Verdict

SammCheese read the README. gustrd understands the current software ecosystem. If you are generating a video with Wan2.1 in ComfyUI, SageAttention is a must-have. If you are hosting Llama 3 on a server, it is currently useless—and the vLLM maintainers know it.

Reference: thu-ml/SageAttention#238
#########

[henk717]

Either way guys please don't waste your tokens on this. Regardless if its nice to have or not we can't add it until llamacpp were to add it themselves. So if the conclusion is that its not neccesary then cool, if the conclusion is that it could be nice to have it wouldn't change the situation. Its them you'd have to convince.