Flash Attention v2 achieved 44% faster than xformers/pytorch_sdp_attention on large image #11884
Replies: 6 comments 13 replies
-
|
Flash Attention 2 has landed in xformers but my initial tests aren't showing any significant speed difference between xformers 0.0.20 and 0.0.21-dev571. |
Beta Was this translation helpful? Give feedback.
-
|
I have tried it and it's faster even if not by much. (also sorry for the error message I forgot about that)
We can get an improvement in inference speed of about 5% max, I would say? |
Beta Was this translation helpful? Give feedback.
-
|
0.0.20 with Torch 2.01, DDIM @ 512x512 gives me a consistent 5.86 it/s +/- 0.05 it/s:
And 1.19 it/s at 1024x1024:
Then, installing xformers-0.0.21+85c0244.d20230720-cp310-cp310-linux_x86_64 and torch-2.1.0.dev20230719+cu121-cp310-cp310-linux_x86_64: 5.88 it/s at 512x512 (0.3% improvement, within margin of error, IMO)
And 1024x1024 gives me ~1.29 it/s (+8.4%):
Launch args are just So, the performance gains on the larger image are certainly welcome, but not exactly blowing the doors off. |
Beta Was this translation helpful? Give feedback.
-
|
Looks reasonable, only on large images has a big performance gains. Personally I usually turned on "Hires. fix" to get a image with more detail (it works well on anime style as far as I know). So it's a good help for me. Also, I think the performance gains may various from different GPUs, maybe the gains is higher on the high-end GPU, more testing is needed. |
Beta Was this translation helpful? Give feedback.
-
|
Not for colab I guess |
Beta Was this translation helpful? Give feedback.
-
|
Colab T4 or V100 won’t work, but A100 will |
Beta Was this translation helpful? Give feedback.
-
|
I made a draft PR for testing: #11902 |
Beta Was this translation helpful? Give feedback.
-
|
i think xformers is also making updates latest dev version is not working on free google colab gpu |
Beta Was this translation helpful? Give feedback.
-
|
Please be smart and read here |
Beta Was this translation helpful? Give feedback.
-
|
I compiled and installed the xformers (mainline) and run a benchmark on the same settings. The performances is close to |
Beta Was this translation helpful? Give feedback.
-
|
hello, can you add more test data? For example, different picture sizes and different sd configurations. We can make a comparison. |
Beta Was this translation helpful? Give feedback.
-
nice what we install via pip (dev version) still don't have this version right? because i didnt see dramatic increase |
Beta Was this translation helpful? Give feedback.
-
|
I'm not sure about the pip dev version wheels, it also looks updated within the same day as the git commit to support v2, while I can't use the pip pre-compiled version because I am using a different version of pytorch (2.0.0). |
Beta Was this translation helpful? Give feedback.
-
|
@FurkanGozukara I tried the dev version released an hour ago, can confirm I get some speedup. Went from 1.2-1.4it/s to 1.8-1.9it/s. Edit: xformers version is 0.0.21.dev575 |
Beta Was this translation helpful? Give feedback.
-
why did I have a opposite result 🤣 speed go down about 10% between xformers 0.0.20 and 0.0.21 dev 2080 Ti at Win11 |
Beta Was this translation helpful? Give feedback.
-
|
Looks the performance gains may various from different GPUs. And AFAIK the flash-attn's FlashAttention-2 implementation only works on Ampere, Ada, or Hopper GPUs (https://github.com/Dao-AILab/flash-attention#installation-and-features) |
Beta Was this translation helpful? Give feedback.
-
|
I have 1050TI and I have got 3x slowdown from upgrading to xformers==0.0.21 and also my memory usage has doubled (and thus this took a significant part in this enormous slowdown) |
Beta Was this translation helpful? Give feedback.
-
|
The benefits of xformers 0.0.21 are mainly for 30xx and up |
Beta Was this translation helpful? Give feedback.
-
|
In the end I have solved the problem and performance remained the same as before with same memory consumption because of some buggy SDWUI option. |
Beta Was this translation helpful? Give feedback.
-
FlashAttention-2 (repo: https://github.com/Dao-AILab/flash-attention, package:
flash-attn, paper) released this weak, which provide "Better Parallelism and Work Partitioning" than existing works, leading to a much higher performance (up to 2x said in their paper) than old FlashAttention (similar to xformers/pytorch_sdp_attention).I wrote a small patch for sd webui and did a quick benckmark. I use "Hires. fix" in the tests, which is popular and you could both see the performance on small image and large image:
It has a massive improvement on large image. And it's 44% faster total time than pytorch_sdp_attention (38s vs 55s), while xformers has a similar performance with pytorch_sdp_attention on my device. The output image may has very little difference from baseline, just like xformers or pytorch_sdp_attention.
I think it's worth to been added to stable diffusion webui, and I will soon make a draft PR about this. However, their are still some problems to discuss:
flash-attnversion 2.0 package will breaktransformers, which is a dependency of sd webui and prevent it to run. This is due toflash-attn2.0 renamed some functions. We may need to waiting fortransformersto fix it. (A workaround is set a alisa in flash-attn package files)flash-attnpackage, I think make a PR to supportflash-attnmay still been useful.flash-attnonly support fp16 and bfloat16, so it's need fallback to other attention implementations (like sdp_a) when a attention is performed in other precisions. This won't affect performance a lot.flash-attnnot support windows yet, andflash-attnwill takes a lot of memory when installing (it will compile something), so we may left it for advanced users to install it by them self.Beta Was this translation helpful? Give feedback.
All reactions