Triton for Torch2 + 15-16 xformers on Windows

[playlogitech]

https://transfer.sh/me0xpC/triton-2.0.0-cp310-cp310-win_amd64.whl

steps:

1. copy to webui folder
2. open commandline from the webui folder and activate venv
3. then pip install triton-2.0.0-cp310-cp310-win_amd64.whl

all cudos for this build to a guy from russian 2ch https://2ch.hk/ai/res/41107.html#41194

[DustyCooper]

What is Triton? I've been getting the errors of it missing from the webui.

[playlogitech]

optimisator for xformers, but i think it needs additional support from webui side, im not sure i've got any speed up from it

[aliencaocao]

the error on windows is normal as triton is only supported on linux. It is a transformer acceleration library designed by Nvidia for Nvidia GPUs

[ice051128]

the error on windows is normal as triton is only supported on linux. It is a transformer acceleration library designed by Nvidia for Nvidia GPUs

Why is this post advertising trition for windows when it isn't even supported? Is this a Russian scam?

[PleezDeez]

Bc the russian built one that works in windows. Install process is wrong tho bc you need to drop the contents of the triton folder it creates into the triton folder in xformers folder before it'll work but it doesn't speed up anything. It optimizes the memory footprint while generating in tandem with xformers to reduce it allowing for bigger batches at higher resolutions.

[ke1ne]

Avoid russian stuff, especially from anonymous 2ch

[ghost]

Im gonna add +1 to russian encirclement here. Surrender. xD

[Nacurutu]

@playlogitech stop feeding the troll! dnt waste ur time bruh...

[ice051128]

Avoid russian stuff, especially from anonymous 2ch

I would have to agree. This is probably just a Russian scam since trition isn't even supported on windows

[mezotaken]

There just was no binary build for windows, but this guy managed to build wheel from source. (apparently)

[ke1ne]

so what? its russian military who is in charge, not a nation

russian military IS the russian nation who did this https://www.nytimes.com/2023/01/16/world/europe/ukraine-dnipro-death-toll.html

[healthyfat]

Looks like only 5-10% increase and only on large pictures:

On small 512x512 pictures, there is no increase, but on large ones with a highres fix, there is definitely, in the region of 5-10%

Аноучсы, которые пользуются xformers 0.0.15/0.0.16 и torch-2.0.0
Я сделал невозможное и собрал triton под ШИНДОШС
Колёсный файл:
https://transfer.sh/me0xpC/triton-2.0.0-cp310-cp310-win_amd64.whl
Собиралось под Python 3.10.8, CUDA 11.8, LLVM 14.0.6, MSVC 14.34.31933
На маленьких пикчах 512х512 прироста никакого нет, но на больших с highres фиксом точно есть, в районе 5-10%

[playlogitech]

ШИНДОШС

[ice051128]

Although he claims this:

On small 512x512 pictures, there is no increase, but on large ones with a highres fix, there is definitely, in the region of 5-10%

Аноучсы, которые пользуются xformers 0.0.15/0.0.16 и torch-2.0.0
Я сделал невозможное и собрал triton под ШИНДОШС
Колёсный файл:
https://transfer.sh/me0xpC/triton-2.0.0-cp310-cp310-win_amd64.whl
Собиралось под Python 3.10.8, CUDA 11.8, LLVM 14.0.6, MSVC 14.34.31933
На маленьких пикчах 512х512 прироста никакого нет, но на больших с highres фиксом точно есть, в районе 5-10%

Define large ones? What resolution?

[PleezDeez]

pip installing leads to no such file error

[PleezDeez]

never mind Im a nob and still new to this sort of stuff. after downloading the file from the the transfer.sh site I was able to get it installing... I wonder if this will now make deepspeed installable via pip... bout to find out when it's done building

[PleezDeez]

it does not

[ffdown]

https://transfer.sh/me0xpC/triton-2.0.0-cp310-cp310-win_amd64.whl

steps:

1. copy to webui folder
2. open commandline from the webui folder and activate venv
3. then pip install triton-2.0.0-cp310-cp310-win_amd64.whl

all cudos for this build to a guy from russian 2ch https://2ch.hk/ai/res/41107.html#41194

how activate VENV?))

[Bigfield77]

to activate venv,
open a command prompt to your stable diffusion folder
cd venv
cd script
activate
cd ../..

[PleezDeez]

./venv/Scripts/activate after opening a terminal from the main 1111 folder is so much faster and less confusing tho

[ffdown]

WHL installed, but all time

Launching Web UI with arguments: --xformers --opt-split-attention --opt-channelslast
[WinError 127] Не найдена указанная процедура
WARNING: [WinError 127] Не найдена указанная процедура
Need to compile C++ extensions to get sparse attention support. Please run python setup.py build develop
A matching Triton is not available, some optimizations will not be enabled.
Error caught was: No module named 'triton'

Click "Generate"
Error completing request Arguments: ('wolf', '', 'None', 'None', 20, 0, False, False, 1, 1, 7, -1.0, -1.0, 0, 0, 0, False, 512, 512, False, 0.7, 2, 'Latent', 0, 0, 0, 0, 0, False, False, False, False, '', 1, '', 0, '', True, False, False) {} Traceback (most recent call last): File "E:\AI\SD\stable-diffusion-webui\modules\call_queue.py", line 45, in f res = list(func(*args, **kwargs)) File "E:\AI\SD\stable-diffusion-webui\modules\call_queue.py", line 28, in f res = func(*args, **kwargs) File "E:\AI\SD\stable-diffusion-webui\modules\txt2img.py", line 52, in txt2img processed = process_images(p) File "E:\AI\SD\stable-diffusion-webui\modules\processing.py", line 479, in process_images res = process_images_inner(p) File "E:\AI\SD\stable-diffusion-webui\modules\processing.py", line 586, in process_images_inner samples_ddim = p.sample(conditioning=c, unconditional_conditioning=uc, seeds=seeds, subseeds=subseeds, subseed_strength=p.subseed_strength, prompts=prompts) File "E:\AI\SD\stable-diffusion-webui\modules\processing.py", line 754, in sample samples = self.sampler.sample(self, x, conditioning, unconditional_conditioning, image_conditioning=self.txt2img_image_conditioning(x)) File "E:\AI\SD\stable-diffusion-webui\modules\sd_samplers.py", line 537, in sample samples = self.launch_sampling(steps, lambda: self.func(self.model_wrap_cfg, x, extra_args={ File "E:\AI\SD\stable-diffusion-webui\modules\sd_samplers.py", line 440, in launch_sampling return func() File "E:\AI\SD\stable-diffusion-webui\modules\sd_samplers.py", line 537, in <lambda> samples = self.launch_sampling(steps, lambda: self.func(self.model_wrap_cfg, x, extra_args={ File "E:\AI\SD\stable-diffusion-webui\venv\lib\site-packages\torch\autograd\grad_mode.py", line 27, in decorate_context return func(*args, **kwargs) File "E:\AI\SD\stable-diffusion-webui\repositories\k-diffusion\k_diffusion\sampling.py", line 145, in sample_euler_ancestral denoised = model(x, sigmas[i] * s_in, **extra_args) File "E:\AI\SD\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl return forward_call(*input, **kwargs) File "E:\AI\SD\stable-diffusion-webui\modules\sd_samplers.py", line 338, in forward x_out = self.inner_model(x_in, sigma_in, cond={"c_crossattn": [cond_in], "c_concat": [image_cond_in]}) File "E:\AI\SD\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl return forward_call(*input, **kwargs) File "E:\AI\SD\stable-diffusion-webui\repositories\k-diffusion\k_diffusion\external.py", line 112, in forward eps = self.get_eps(input * c_in, self.sigma_to_t(sigma), **kwargs) File "E:\AI\SD\stable-diffusion-webui\repositories\k-diffusion\k_diffusion\external.py", line 138, in get_eps return self.inner_model.apply_model(*args, **kwargs) File "E:\AI\SD\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\models\diffusion\ddpm.py", line 858, in apply_model x_recon = self.model(x_noisy, t, **cond) File "E:\AI\SD\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl return forward_call(*input, **kwargs) File "E:\AI\SD\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\models\diffusion\ddpm.py", line 1329, in forward out = self.diffusion_model(x, t, context=cc) File "E:\AI\SD\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl return forward_call(*input, **kwargs) File "E:\AI\SD\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\diffusionmodules\openaimodel.py", line 776, in forward h = module(h, emb, context) File "E:\AI\SD\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl return forward_call(*input, **kwargs) File "E:\AI\SD\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\diffusionmodules\openaimodel.py", line 84, in forward x = layer(x, context) File "E:\AI\SD\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl return forward_call(*input, **kwargs) File "E:\AI\SD\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\attention.py", line 324, in forward x = block(x, context=context[i]) File "E:\AI\SD\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl return forward_call(*input, **kwargs) File "E:\AI\SD\stable-diffusion-webui\modules\sd_hijack_checkpoint.py", line 4, in BasicTransformerBlock_forward return checkpoint(self._forward, x, context) File "E:\AI\SD\stable-diffusion-webui\venv\lib\site-packages\torch\utils\checkpoint.py", line 235, in checkpoint return CheckpointFunction.apply(function, preserve, *args) File "E:\AI\SD\stable-diffusion-webui\venv\lib\site-packages\torch\utils\checkpoint.py", line 96, in forward outputs = run_function(*args) File "E:\AI\SD\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\attention.py", line 262, in _forward x = self.attn1(self.norm1(x), context=context if self.disable_self_attn else None) + x File "E:\AI\SD\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl return forward_call(*input, **kwargs) File "E:\AI\SD\stable-diffusion-webui\modules\sd_hijack_optimizations.py", line 229, in xformers_attention_forward out = xformers.ops.memory_efficient_attention(q, k, v, attn_bias=None) File "E:\AI\SD\stable-diffusion-webui\venv\lib\site-packages\xformers\ops\fmha\__init__.py", line 193, in memory_efficient_attention return _memory_efficient_attention( File "E:\AI\SD\stable-diffusion-webui\venv\lib\site-packages\xformers\ops\fmha\__init__.py", line 289, in _memory_efficient_attention return _memory_efficient_attention_forward( File "E:\AI\SD\stable-diffusion-webui\venv\lib\site-packages\xformers\ops\fmha\__init__.py", line 311, in _memory_efficient_attention_forward out, *_ = op.apply(inp, needs_gradient=False) File "E:\AI\SD\stable-diffusion-webui\venv\lib\site-packages\xformers\ops\fmha\cutlass.py", line 103, in apply out, lse = cls.OPERATOR( File "E:\AI\SD\stable-diffusion-webui\venv\lib\site-packages\xformers\ops\common.py", line 13, in no_such_operator raise RuntimeError( RuntimeError: No such operator xformers::efficient_attention_forward_cutlass - did you forget to build xformers with python setup.py develop?

[PleezDeez]

dude's definitely into some loli stuff lol... gross

[PleezDeez]

it works fine and did notice a little bit of a boost but still... gross

[ffdown]

it works fine and did notice a little bit of a boost but still... gross

random text no more, the question is completely irrelevant

[ice051128]

dude's definitely into some loli stuff lol... gross

Yeah, that link contains some creepy shit

[PleezDeez]

it works fine and did notice a little bit of a boost but still... gross

random text no more, the question is completely irrelevant

only warning people who wanted to know more about the guy who made it before installing some random dude's file that they should probably avoid his social media page if they don't want to see AI child porn... the rest is just letting others know it should work if installed properly and I did notice a small improvement.

[Seedmanc]

you say it like it's something bad

[Nacurutu]

May I ask, is this just for Linux? or also Windows?

Works with all of the cards or just the 4000 series?

Thanks

[PleezDeez]

this is for windows. Triton is traditionally a linux only feature that's required for stuff like Microsoft's own Deepspeed to function which is a method for further distributing work loads across multiple devices including both CPU and GPU training new models together instead of 1 or the other opening up Dreambooth to even lower VRAM cards.

[PleezDeez]

if youre on linux
pip install triton and you have it

[PleezDeez]

oh and not just for 40 series owners tho anything below the 20 series won't see much benefit from most of the optimizations beyond lowering the memory footprint bc they utilize nvidia's tensor cores which are only on RTX cards

[Nacurutu]

@PleezDeez Thanks a lot..!

I'm on Windows, so I need to do the whole process explained in the OP.

[PleezDeez]

if you want to actually benefit from it you need to be on Cuda 11.7+, xformers 0.0.016+ (right click from main folder and open in terminal then input ./venv/Scripts/activate then pip install xformers==0.0.16rc404), delete every folder in your folder/venv/Lib/site-packages prefaced torch, back in the terminal enter pip3 install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu117
it'll ask for torchmetrics, torchlightning, pyre extensions, etc when it's done. just pip install and copy/paste whatever it asks for. back in the main folder for 1111 change 176 of launch.py to read "pip3 install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu117" instead of cu113 or just scroll a bit til you find it in notepad plus make sure "--xfomers" is in line 178 plus whatever arguments like --medvram etc. Once all that's done you have to find the triton folder in your site-packages installed by following the instructions at the top of this page and copy everything inside over into the triton folder inside the xformers folder and replace any files it conflicts...

oh and be sure to right click venv in your main 1111 folder and go to properties, security, edit and give full control level permission to everybody in there...

simplest way I can break it down and should cover about 90ish percent of the errors i've seen popping up

[PleezDeez]

i've seen some people saying you need python 3.10.6 but it works fine on the 3.10.8 I use from miniconda3

[shikkoku]

'triton-2.0.0-cp310-cp310-win_amd64.whl' link was just display 'not found'.

[shikkoku]

that is not a link, you have to download the file and place it in your working dir then open cmd from that dir

I can't download with that link.

[aliencaocao]

Oh then the link is dead

[shikkoku]

grab it from my drive . click here

pip3 show triton Name: triton Version: 2.0.0 Summary: A language and compiler for custom Deep Learning operations Home-page: https://github.com/openai/triton/ Author: Philippe Tillet Author-email: phil@openai.com

python -c "import triton" ....... Python310\site-packages\triton\testing.py", line 9, in <module> import triton._C.libtriton.triton as _triton ImportError: DLL load failed while importing libtriton: The specified module could not be found.

[hwindo]

in case someone still looking for it, i found it in HuggingFace
https://huggingface.co/r4ziel/xformers_pre_built/tree/22505e3edeead471f3801ff2c3d478ffa51be755

[DarkAlchy]

too bad triton-2.0.0-cp310-cp310-win_amd64.whl is old now and screws stuff up (flat out will not let a lot of my AI stuff run). We need at least 2.2 version.