xFormers 0.0.16

[danthe3rd]

Todo list for 0.0.16 release:

• Support varying sequence length
• Pip wheels available (Making built wheels available for install #533)
• Make space for additional conda builds
• Non-cryptic error message when using the wrong version of pytorch
• Write a proper changelog since the last one (0.0.13)
• Test pip wheels (linux only)

Not included in 0.0.16:

• Attention with bias + dropout ([feat] cutlass FlashAttention bias+dropout support #587)

[lizelive]

can you add pytorch 2 wheels for this release?

[danthe3rd]

can you add pytorch 2 wheels for this release?

PyTorch 2 is not yet released, this means there is a new version every day, and we would need to release new wheels every day (because binaries are linked to a specific pt version).
This is not something we plan to do in the short term

[petalas]

@danthe3rd would it be possible to make pip wheels available built with cuda 11.8 please?

I've been trying out xformers 0.0.16rc395 0.0.16rc396 with torch 1.13.1+cu117 and torchvision 0.14.1+cu117

Found that I get a decent performance boost if I manually copy over and replace the .dll files
from: cudnn-windows-x86_64-8.6.0.163_cuda11-archive\bin
to: stable-diffusion-webui\venv\Lib\site-packages\torch\lib

[brucethemoose]

+1, I had to build from source for Arch Linux's CUDA 11.8 base.

Is it too early for a cuda 12 wheel?

[danthe3rd]

Oh that's interesting - which GPU are you using and what did you get speedup on? For CUDA 11.8 It means you had to build torchvision and PyTorch from source? I believe the binary wheels are only for CUDA 11.6/11.7

For CUDA 12, I don't think it's supported by PyTorch.

[petalas]

Oh that's interesting - which GPU are you using and what did you get speedup on? For CUDA 11.8 It means you had to build torchvision and PyTorch from source? I believe the binary wheels are only for CUDA 11.6/11.7

For CUDA 12, I don't think it's supported by PyTorch.

Hello, I am using a 4090 FE, got a ~50% speedup generating images (txt2img stable diffusion).

I did not build anything from source, just a regular pip install torch==1.13.1+cu117 torchvision==0.14.1+cu117 xformers==0.0.16rc396.

Then I manually replaced the installed .dll files in torch\lib with the ones from the 11.8 bin.
I am trying to avoid having to do this last manual bit (or build from source), I'm guessing/hoping if they're built with cuda 11.8 we'll get the same speedup.

[brucethemoose]

Oh that's interesting - which GPU are you using and what did you get speedup on? For CUDA 11.8 It means you had to build torchvision and PyTorch from source? I believe the binary wheels are only for CUDA 11.6/11.7

For CUDA 12, I don't think it's supported by PyTorch.

Nah no torch building, I am on Arch Linux (CachyOS specifically), which builds pytorch to target the rest of their packages (Python 3.10/CUDA 11.8 at the moment): https://archlinux.org/packages/community/x86_64/python-pytorch-opt-cuda/

Right now I am just testing on a laptop RTX 2060, and the speedup is measurable but single digit %. But I am considering setting stuff up for an Ampere or Ada instance.

[blefaudeux]

For CUDA 11.8 It means you had to build torchvision and PyTorch from source?

this use case is actually surprisingly common (and no need to build torch from source in that case), nvidia ships cuda 11.8 with their reference docker image (I know this firsthand :))

[danthe3rd]

We build at the moment for cuda 11.7. I believe this is compatible with cuda 11.8.
Might be a matter of making the cuda version requirement less strict then (if you are using conda).

[danthe3rd]

We plan to release it as soon as #641 is fixed

[danthe3rd]

We have released v0.0.16rc425 last Friday on conda (channel "main") and pypi. We plan to release the stable version on that tag if users are happy with the rc.
Changes added to main in between will be added to 0.0.17

[danthe3rd]

We just released v0.0.16