-
Notifications
You must be signed in to change notification settings - Fork 2.4k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add instructions for installing xFormers on linux (#2360)
I've written up the install procedure for xFormers on Linux systems. I need help with the Windows install; I don't know what the build dependencies (compiler, etc) are. This section of the docs is currently empty. Please see `docs/installation/070_INSTALL_XFORMERS.md`
- Loading branch information
Showing
2 changed files
with
215 additions
and
3 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,206 @@ | ||
--- | ||
title: Installing xFormers | ||
--- | ||
|
||
# :material-image-size-select-large: Installing xformers | ||
|
||
xFormers is toolbox that integrates with the pyTorch and CUDA | ||
libraries to provide accelerated performance and reduced memory | ||
consumption for applications using the transformers machine learning | ||
architecture. After installing xFormers, InvokeAI users who have | ||
CUDA GPUs will see a noticeable decrease in GPU memory consumption and | ||
an increase in speed. | ||
|
||
xFormers can be installed into a working InvokeAI installation without | ||
any code changes or other updates. This document explains how to | ||
install xFormers. | ||
|
||
## Pip Install | ||
|
||
For both Windows and Linux, you can install `xformers` in just a | ||
couple of steps from the command line. | ||
|
||
If you are used to launching `invoke.sh` or `invoke.bat` to start | ||
InvokeAI, then run the launcher and select the "developer's console" | ||
to get to the command line. If you run invoke.py directly from the | ||
command line, then just be sure to activate it's virtual environment. | ||
|
||
Then run the following three commands: | ||
|
||
```sh | ||
pip install xformers==0.0.16rc425 | ||
pip install triton | ||
python -m xformers.info output | ||
``` | ||
|
||
The first command installs `xformers`, the second installs the | ||
`triton` training accelerator, and the third prints out the `xformers` | ||
installation status. If all goes well, you'll see a report like the | ||
following: | ||
|
||
```sh | ||
xFormers 0.0.16rc425 | ||
memory_efficient_attention.cutlassF: available | ||
memory_efficient_attention.cutlassB: available | ||
memory_efficient_attention.flshattF: available | ||
memory_efficient_attention.flshattB: available | ||
memory_efficient_attention.smallkF: available | ||
memory_efficient_attention.smallkB: available | ||
memory_efficient_attention.tritonflashattF: available | ||
memory_efficient_attention.tritonflashattB: available | ||
swiglu.fused.p.cpp: available | ||
is_triton_available: True | ||
is_functorch_available: False | ||
pytorch.version: 1.13.1+cu117 | ||
pytorch.cuda: available | ||
gpu.compute_capability: 8.6 | ||
gpu.name: NVIDIA RTX A2000 12GB | ||
build.info: available | ||
build.cuda_version: 1107 | ||
build.python_version: 3.10.9 | ||
build.torch_version: 1.13.1+cu117 | ||
build.env.TORCH_CUDA_ARCH_LIST: 5.0+PTX 6.0 6.1 7.0 7.5 8.0 8.6 | ||
build.env.XFORMERS_BUILD_TYPE: Release | ||
build.env.XFORMERS_ENABLE_DEBUG_ASSERTIONS: None | ||
build.env.NVCC_FLAGS: None | ||
build.env.XFORMERS_PACKAGE_FROM: wheel-v0.0.16rc425 | ||
source.privacy: open source | ||
``` | ||
|
||
## Source Builds | ||
|
||
`xformers` is currently under active development and at some point you | ||
may wish to build it from sourcce to get the latest features and | ||
bugfixes. | ||
|
||
### Source Build on Linux | ||
|
||
Note that xFormers only works with true NVIDIA GPUs and will not work | ||
properly with the ROCm driver for AMD acceleration. | ||
|
||
xFormers is not currently available as a pip binary wheel and must be | ||
installed from source. These instructions were written for a system | ||
running Ubuntu 22.04, but other Linux distributions should be able to | ||
adapt this recipe. | ||
|
||
#### 1. Install CUDA Toolkit 11.7 | ||
|
||
You will need the CUDA developer's toolkit in order to compile and | ||
install xFormers. **Do not try to install Ubuntu's nvidia-cuda-toolkit | ||
package.** It is out of date and will cause conflicts among the NVIDIA | ||
driver and binaries. Instead install the CUDA Toolkit package provided | ||
by NVIDIA itself. Go to [CUDA Toolkit 11.7 | ||
Downloads](https://developer.nvidia.com/cuda-11-7-0-download-archive) | ||
and use the target selection wizard to choose your platform and Linux | ||
distribution. Select an installer type of "runfile (local)" at the | ||
last step. | ||
|
||
This will provide you with a recipe for downloading and running a | ||
install shell script that will install the toolkit and drivers. For | ||
example, the install script recipe for Ubuntu 22.04 running on a | ||
x86_64 system is: | ||
|
||
``` | ||
wget https://developer.download.nvidia.com/compute/cuda/11.7.0/local_installers/cuda_11.7.0_515.43.04_linux.run | ||
sudo sh cuda_11.7.0_515.43.04_linux.run | ||
``` | ||
|
||
Rather than cut-and-paste this example, We recommend that you walk | ||
through the toolkit wizard in order to get the most up to date | ||
installer for your system. | ||
|
||
#### 2. Confirm/Install pyTorch 1.13 with CUDA 11.7 support | ||
|
||
If you are using InvokeAI 2.3 or higher, these will already be | ||
installed. If not, you can check whether you have the needed libraries | ||
using a quick command. Activate the invokeai virtual environment, | ||
either by entering the "developer's console", or manually with a | ||
command similar to `source ~/invokeai/.venv/bin/activate` (depending | ||
on where your `invokeai` directory is. | ||
|
||
Then run the command: | ||
|
||
```sh | ||
python -c 'exec("import torch\nprint(torch.__version__)")' | ||
``` | ||
|
||
If it prints __1.13.1+cu117__ you're good. If not, you can install the | ||
most up to date libraries with this command: | ||
|
||
```sh | ||
pip install --upgrade --force-reinstall torch torchvision | ||
``` | ||
|
||
#### 3. Install the triton module | ||
|
||
This module isn't necessary for xFormers image inference optimization, | ||
but avoids a startup warning. | ||
|
||
```sh | ||
pip install triton | ||
``` | ||
|
||
#### 4. Install source code build prerequisites | ||
|
||
To build xFormers from source, you will need the `build-essentials` | ||
package. If you don't have it installed already, run: | ||
|
||
```sh | ||
sudo apt install build-essential | ||
``` | ||
|
||
#### 5. Build xFormers | ||
|
||
There is no pip wheel package for xFormers at this time (January | ||
2023). Although there is a conda package, InvokeAI no longer | ||
officially supports conda installations and you're on your own if you | ||
wish to try this route. | ||
|
||
Following the recipe provided at the [xFormers GitHub | ||
page](https://github.com/facebookresearch/xformers), and with the | ||
InvokeAI virtual environment active (see step 1) run the following | ||
commands: | ||
|
||
```sh | ||
pip install ninja | ||
export TORCH_CUDA_ARCH_LIST="6.0;6.1;6.2;7.0;7.2;7.5;8.0;8.6" | ||
pip install -v -U git+https://github.com/facebookresearch/xformers.git@main#egg=xformers | ||
``` | ||
|
||
The TORCH_CUDA_ARCH_LIST is a list of GPU architectures to compile | ||
xFormer support for. You can speed up compilation by selecting | ||
the architecture specific for your system. You'll find the list of | ||
GPUs and their architectures at NVIDIA's [GPU Compute | ||
Capability](https://developer.nvidia.com/cuda-gpus) table. | ||
|
||
If the compile and install completes successfully, you can check that | ||
xFormers is installed with this command: | ||
|
||
```sh | ||
python -m xformers.info | ||
``` | ||
|
||
If suiccessful, the top of the listing should indicate "available" for | ||
each of the `memory_efficient_attention` modules, as shown here: | ||
|
||
```sh | ||
memory_efficient_attention.cutlassF: available | ||
memory_efficient_attention.cutlassB: available | ||
memory_efficient_attention.flshattF: available | ||
memory_efficient_attention.flshattB: available | ||
memory_efficient_attention.smallkF: available | ||
memory_efficient_attention.smallkB: available | ||
memory_efficient_attention.tritonflashattF: available | ||
memory_efficient_attention.tritonflashattB: available | ||
[...] | ||
``` | ||
|
||
You can now launch InvokeAI and enjoy the benefits of xFormers. | ||
|
||
### Windows | ||
|
||
To come | ||
|
||
|
||
--- | ||
(c) Copyright 2023 Lincoln Stein and the InvokeAI Development Team |