Simple and fast Colab version, fast-stable-diffusion, +25% speed increase + memory efficient

[TheLastBen]

Colab adaptation for AUTOMATIC1111 Webui version of stable diffusion implementing the optimization suggested by https://github.com/MatthieuTPHR : huggingface/diffusers#532, using the MemoryEfficientAttention implementation from xformers (cc. @fmassa, @danthe3rd, @blefaudeux) to both speedup the cross-attention speed and decrease its GPU memory requirements.

All you have to do is enter your huggingface token only once and you're all set, the colabs will install the repos and the models inside Gdrive, so the loading will be fast everytime you use it, enjoy !!

from : https://github.com/TheLastBen/fast-stable-diffusion

[kaneda2004]

Appears to be broken, upon image generation it errors out indicating CUDA Error: No kernel image is available for execution on the device

Std Out log below:

0% 0/20 [00:03<?, ?it/s]
Error completing request
Arguments: (' a profile portrait of an african american, by annie leibovitz, large format, 4x5, zeiss lens, masterpiece, dramatic lighting, hairlight', '', 'None', 'None', 20, 0, False, False, 1, 1, 7, -1.0, -1.0, 0, 0, 0, False, 512, 512, False, False, 0.7, 0, False, False, None, '', 1, '', 4, '', True) {}
Traceback (most recent call last):
File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/modules/ui.py", line 134, in f
res = list(func(*args, **kwargs))
File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/webui.py", line 58, in f
res = func(*args, **kwargs)
File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/modules/txt2img.py", line 43, in txt2img
processed = process_images(p)
File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/modules/processing.py", line 340, in process_images
samples_ddim = p.sample(conditioning=c, unconditional_conditioning=uc, seeds=seeds, subseeds=subseeds, subseed_strength=p.subseed_strength)
File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/modules/processing.py", line 444, in sample
samples = self.sampler.sample(self, x, conditioning, unconditional_conditioning)
File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/modules/sd_samplers.py", line 289, in sample
samples = self.func(self.model_wrap_cfg, x, sigmas, extra_args={'cond': conditioning, 'uncond': unconditional_conditioning, 'cond_scale': p.cfg_scale}, disable=False, callback=self.callback_state)
File "/usr/local/lib/python3.7/dist-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/k_diffusion/sampling.py", line 77, in sample_euler_ancestral
denoised = model(x, sigmas[i] * s_in, **extra_args)
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/modules/sd_samplers.py", line 176, in forward
uncond, cond = self.inner_model(x_in, sigma_in, cond=cond_in).chunk(2)
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/k_diffusion/external.py", line 100, in forward
eps = self.get_eps(input * c_in, self.sigma_to_t(sigma), **kwargs)
File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/k_diffusion/external.py", line 126, in get_eps
return self.inner_model.apply_model(*args, **kwargs)
File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/ldm/models/diffusion/ddpm.py", line 987, in apply_model
x_recon = self.model(x_noisy, t, **cond)
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/ldm/models/diffusion/ddpm.py", line 1410, in forward
out = self.diffusion_model(x, t, context=cc)
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/ldm/modules/diffusionmodules/openaimodel.py", line 732, in forward
h = module(h, emb, context)
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/ldm/modules/diffusionmodules/openaimodel.py", line 85, in forward
x = layer(x, context)
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/ldm/modules/attention.py", line 418, in forward
x = block(x, context=context)
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/ldm/modules/attention.py", line 252, in forward
hidden_states = self.attn1(self.norm1(hidden_states)) + hidden_states
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/ldm/modules/attention.py", line 330, in forward
out = xformers.ops.memory_efficient_attention(q, k, v, attn_bias=None, op=self.attention_op)
File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/xformers/ops.py", line 575, in memory_efficient_attention
query=query, key=key, value=value, attn_bias=attn_bias, p=p
File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/xformers/ops.py", line 196, in forward_no_grad
causal=isinstance(attn_bias, LowerTriangularMask),
File "/usr/local/lib/python3.7/dist-packages/torch/_ops.py", line 143, in call
return self._op(*args, **kwargs or {})
RuntimeError: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

[DenseFork]

That is a typo!

[TheLastBen]

so it's _C_flashattention.so

[TheLastBen]

how many iterations/s do you get at 512x512, before _C_flashattention.so and after adding the file

[danthe3rd]

Now that's weird because flash attention does not support P100. It should fallback on xformer's kernels, which are in _C.so

[TheLastBen]

Now that's weird because flash attention does not support P100. It should fallback on xformer's kernels, which are in _C.so

Could be that he got a Tesla T4 instead of a P100 from Colab at the moment of compiling.

[Andrew3216767]

Hi. Thank you for your optimisation implementation. I don't use Gdrive for my SD, I only use One Drive from Microsoft. Is there any simple way to install it in a specific folder where I have my AUTOMATIC's repo, while keeping all the models intact (which was already downloaded)? Thanks.

[TheLastBen]

Hi, as far as I know, colab works only with google drive or its own provided storage.

[Andrew3216767]

Hi, as far as I know, colab works only with google drive or its own provided storage.
Considering I already have Repo up and running, can I just change the attention.py?

[TheLastBen]

this optimization needs the compiled files of xformers, and I provide 2 versions for the Tesla T4 and the P100 for free colab users, you can't use it locally, changing only attention.py won't suffice .

[precompute]

https://github.com/t-e-r-m/Portable-Stable-Diffusion-Notebook
I think I got it, try this out.

[precompute]

Sans gdrive: https://github.com/t-e-r-m/Portable-Stable-Diffusion-Notebook

[Mr-Jay]

HI, i tried your colab, i ran all cells, i just imported the model.cpkt by moving the one i already got on my own automatic version on my drive.

But when i run the last cell i got this issue:

When i want to rerun, do it need to run all cell again every time?

[Mr-Jay]

i will run the cell to import the model again to be sure.

[Mr-Jay]

indeed it's better now, when i moved the model i put it in "sd/stable-diffusion-webui\models\Stable-diffusion" but i seen another folder in sd just named "stable-diffusion" with a subfolder model as well.

Where should i put my models if i place manually?

[TheLastBen]

sd/stable-diffusion-webui/models/Stable-diffusion

[Mr-Jay]

It's look like once the runtime is finished, if i run again i have to run all cell, and by doing that all my models in this folder are deleted

[TheLastBen]

Nope, nothing is deleted by the colab, check your gdrive