SAGE ATTENTION: 2.1 TIMES FASTER THAN FLASH ATTENTION 2 AND 2.7 TIMES FASTER THAN XFORMERS

[joseph777111]

https://arxiv.org/abs/2410.02367

SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Acceleration

ABSTRACT:

The transformer architecture predominates across various models. As the heart of the transformer, attention has a computational complexity of O(N^2), compared to O(N) for linear transformations. When handling large sequence lengths, attention becomes the primary time-consuming component. Although quantization has proven to be an effective method for accelerating model inference, existing quantization methods primarily focus on optimizing the linear layer. In response, we first analyze the feasibility of quantization in attention detailedly. Following that, we propose SageAttention, a highly efficient and accurate quantization method for attention. The OPS (operations per second) of our approach outperforms FlashAttention2 and xformers by about 2.1 times and 2.7 times, respectively. SageAttention also achieves superior accuracy performance over FlashAttention3. Comprehensive experiments confirm that our approach incurs almost no end-to-end metrics loss across diverse models, including those for large language processing, image generation, and video generation.

[joseph777111]

@ggerganov @ikawrakow @kalomaze @slaren @JohannesGaessler @calvintwr @LostRuins @bartowski1182 Thoughts? 😋

[bartowski1182]

Love the shoutout but I am NOT on the technical side of things like this 😂😂 that said, 8 bit cache is still not really my cup of tea, but would be interested in some metrics nonetheless

[joseph777111]

Love the shoutout but I am NOT on the technical side of things like this 😂😂 that said, 8 bit cache is still not really my cup of tea, but would be interested in some metrics nonetheless

You're always invited to the party, bro! 😋

[phoebdroid]

Using sage attention with comfyUI and it's really, REALLY fast with almost no visible quality loss. So, simply can't wait the day it replaces fattn for my llama server, @ggerganov, please ?

[JohannesGaessler]

To my knowledge the "FlashAttention" and "SageAttention" software packages both implement the same fundamental "FlashAttention" algorithm. llama.cpp/ggml uses neither of these implementations but instead has its own implementation of the "FlashAttention" algorithm. So these numbers are not directly comparable and you can't just swap out the implementation like you would with a PyTorch-based project.

[sorasoras]

This should be interesting as it can perform attention int8 so it does support most hardware

[Thomas-MMJ]

Updated version Sage Attention 2 - SageAttention2: Efficient Attention with Thorough Outlier Smoothing and Per-thread INT4 Quantization

"The operations per second (OPS) of SageAttention2 surpass FlashAttention2 and xformers by about 3x and 5x on RTX4090, respectively. Comprehensive experiments confirm that our approach incurs negligible end-to-end metrics loss across diverse models, including those for large language processing, image generation, and video generation."

https://arxiv.org/abs/2411.10958
https://github.com/thu-ml/SageAttention

[subspecs]

Any news on this?

[0cc4m]

#9901 (reply in thread)