Stable Diffusion 2 #1392

vvvm23 · 2022-11-24T08:52:05Z

Model/Pipeline/Scheduler description

Perhaps you already have something in the works for this, but I couldn't see an existing issue for this.

Stable Diffusion 2 just got released on a separate repo. There are a few variations including the base, depth2img, and inpainting models. Though useable in the stable diffusion repo now, it would be really awesome to have support in diffusers to allow for ease of us in other applications!

Open source status

The model implementation is available
The model weights are available (Only relevant if addition is not a scheduler).

Provide useful links for the implementation

Model code and inference scripts can be found here

Model weights seem already available on 🤗 Hub at stabilityai/stable-diffusion-2-***

The text was updated successfully, but these errors were encountered:

0xdevalias · 2022-11-24T10:52:04Z

Semi-Related:

0xdevalias · 2022-11-24T11:00:57Z

rudimentary support for stable diffusion 2.0

MrCheeze/stable-diffusion-webui@069591b

Originally posted by @152334H in AUTOMATIC1111/stable-diffusion-webui#5011 (comment)

0xdevalias · 2022-11-24T11:01:21Z

https://github.com/hafriedlander/diffusers/blob/stable_diffusion_2/scripts/convert_original_stable_diffusion_to_diffusers.py

Notes:

Only tested on the two txt2img models, not inpaint / depth2img / upscaling

You will need to change your text embedding to use the penultimate layer too

It spits out a bunch of warnings about vision_model, but that's fine

I have no idea if this is right or not. It generates images, no guarantee beyond that. (Hence no PR - if you're patient, I'm sure the Diffusers team will do a better job than I have)

Originally posted by @hafriedlander in #1388 (comment)

Here's an example of accessing the penultimate text embedding layer https://github.com/hafriedlander/stable-diffusion-grpcserver/blob/b34bb27cf30940f6a6a41f4b77c5b77bea11fd76/sdgrpcserver/pipeline/text_embedding/basic_text_embedding.py#L33

Originally posted by @hafriedlander in #1388 (comment)

doesn't seem to work for me on the 768-v model using the v2 config for v

TypeError: EulerDiscreteScheduler.init() got an unexpected keyword argument 'prediction_type'

Originally posted by @devilismyfriend in #1388 (comment)

You need to use the absolute latest Diffusers and merge this PR (or use my branch which has it in it) #1386

Originally posted by @hafriedlander in #1388 (comment)

(My branch is at https://github.com/hafriedlander/diffusers/tree/stable_diffusion_2)

Originally posted by @hafriedlander in #1388 (comment)

averad · 2022-11-24T11:56:12Z

🤗 Diffusers with Stable Diffusion 2 is live!

anton-l commented (#1388 (comment))
diffusers==0.9.0 with Stable Diffusion 2 are live! https://github.com/huggingface/diffusers/releases/tag/v0.9.0

📰 News

Weights for the 768x768 model is up! - Averad
- https://huggingface.co/stabilityai/stable-diffusion-2
add v prediction #1386 - by patil-suraj and patrickvonplaten has been merged into --main by patil-suraj
Weights for the 512x512 models are up! - patrickvonplaten
- https://huggingface.co/stabilityai/stable-diffusion-2-base
- https://huggingface.co/stabilityai/stable-diffusion-2-inpainting

✏️ Notes & Information

Related huggingface/diffusers Pull Requests:

👇 Quick Links:

👁️ User Submitted Resources:

Test Version of convert_original_stable_diffusion_to_diffusers.py by hafriedlander
Example of accessing the penultimate text embedding layer by hafriedlander
NovelAI Improvements on Stable Diffusion

💭 User Story

Stable Diffusion 2.0 has recently been released. When you run convert_original_stable_diffusion_to_diffusers.py on the new Stability-AI/stablediffusion models the following errors occur.

convert_original_stable_diffusion_to_diffusers.py --checkpoint_path="./512-inpainting-ema.ckpt" --dump_path="./512-inpainting-ema_diffusers"

Output:

Traceback (most recent call last):
File "convert_original_stable_diffusion_to_diffusers.py", line 720, in <module> 
        unet.load_state_dict(converted_unet_checkpoint)
File "lib\site-packages\torch\nn\modules\module.py", line 1667, in load_state_dict
        raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for UNet2DConditionModel:
        size mismatch for down_blocks.0.attentions.0.proj_in.weight: copying a param with shape torch.Size([320, 320]) from checkpoint, the shape in current model is torch.Size([320, 320, 1, 1]).
        size mismatch for down_blocks.0.attentions.0.transformer_blocks.0.attn2.to_k.weight: copying a param with shape torch.Size([320, 1024]) from checkpoint, the shape in current model is torch.Size([320, 768]).
        size mismatch for down_blocks.0.attentions.0.transformer_blocks.0.attn2.to_v.weight: copying a param with shape torch.Size([320, 1024]) from checkpoint, the shape in current model is torch.Size([320, 768]).
        size mismatch for down_blocks.0.attentions.0.proj_out.weight: copying a param with shape torch.Size([320, 320]) from checkpoint, the shape in current model is torch.Size([320, 320, 1, 1]).
.... blocks.1.attentions blocks.2.attentions etc. etc.

0xdevalias · 2022-11-24T12:34:24Z

testing in progress on the horde https://github.com/Sygil-Dev/nataili/tree/v2
try it out Stable Diffusion 2.0 on our UI's

https://tinybots.net/artbot
https://aqualxx.github.io/stable-ui/
https://dbzer0.itch.io/lucid-creations

https://sigmoid.social/@stablehorde/109398715339480426

SD 2.0

Initial implementation ready for testing

img2img

inpainting

k_diffusers support

Originally posted by @AlRlC in https://github.com/Sygil-Dev/nataili/issues/67#issuecomment-1326385645

0xdevalias · 2022-11-24T13:21:02Z

TheLastBen/fast-stable-diffusion@11fd38b

Create pathsV2.py

TheLastBen/fast-stable-diffusion@fe445d9

Support for SD V.2

TheLastBen/fast-stable-diffusion@da9b380

fix

TheLastBen/fast-stable-diffusion@6c84728

fix

TheLastBen/fast-stable-diffusion@04ba92b

fix

TheLastBen/fast-stable-diffusion@ebea134

Create sd_hijackV2.py

TheLastBen/fast-stable-diffusion@88496f5

Create sd_samplersV2.py

TheLastBen/fast-stable-diffusion@f324b3d

fix V2

Originally posted by @0xdevalias in TheLastBen/fast-stable-diffusion#599 (comment)

0xdevalias · 2022-11-24T13:36:11Z

Should work now, make sure you check the box "redownload original model" when choosing V2

https://colab.research.google.com/github/TheLastBen/fast-stable-diffusion/blob/main/fast_stable_diffusion_AUTOMATIC1111.ipynb

Requires more than 12GB of RAM for now, so free colab probably won't suffice.

Originally posted by @TheLastBen in TheLastBen/fast-stable-diffusion#599 (comment)

vvvm23 · 2022-11-24T18:06:00Z

From @pcuenca on the HF discord:

We are busy preparing a new release of diffusers to fully support Stable Diffusion 2. We are still ironing things out, but the basics already work from the main branch in github. Here's how to do it:

Install diffusers from github alongside its dependencies:

pip install --upgrade git+https://github.com/huggingface/diffusers.git transformers accelerate scipy

Use the code in this script to run your predictions:

from diffusers import DiffusionPipeline, EulerDiscreteScheduler
import torch

repo_id = "stabilityai/stable-diffusion-2"
device = "cuda"

scheduler = EulerDiscreteScheduler.from_pretrained(repo_id, subfolder="scheduler", prediction_type="v_prediction")
pipe = DiffusionPipeline.from_pretrained(repo_id, torch_dtype=torch.float16, revision="fp16", scheduler=scheduler)
pipe = pipe.to(device)

prompt = "High quality photo of an astronaut riding a horse in space"
image = pipe(prompt, width=768, height=768, guidance_scale=9).images[0]
image.save("astronaut.png")

0xdevalias · 2022-11-25T06:19:48Z

More related:

0xdevalias · 2022-11-25T06:40:00Z

how sure are you that your conversion is correct? I'm trying to diagnose a difference I get between your 768 weights and my conversion script. There's a big difference, and in general I much prefer the results from my conversion. It seems specific to the unet - if I replace my unet with yours I get the same results.

Originally posted by @hafriedlander in #1388 (comment)

OK, differential diagnostic done, it's the Tokenizer. How did you create the Tokenizer at https://huggingface.co/stabilityai/stable-diffusion-2/tree/main/tokenizer? I just built a Tokenizer using AutoTokenizer.from_pretrained("laion/CLIP-ViT-H-14-laion2B-s32B-b79K") - it seems to give much better results.

Originally posted by @hafriedlander in #1388 (comment)

I've put "my" version of the Tokenizer at https://huggingface.co/halffried/sd2-laion-clipH14-tokenizer/tree/main. You can just replace the tokenizer in any pipeline to test it if you're interested.

Originally posted by @hafriedlander in #1388 (comment)

Marcophono2 · 2022-11-25T17:21:35Z

And available in 0.9.0! Damn fast job, guys!

0xdevalias · 2022-11-26T03:47:46Z

diffusers==0.9.0 with Stable Diffusion 2 is live!

https://github.com/huggingface/diffusers/releases/tag/v0.9.0

Originally posted by @anton-l in #1388 (comment)

0xdevalias · 2022-11-26T07:21:15Z

when will Dreambooth support sd2

While it's not dreambooth, this repo seems to have support for finetuning SDv2:

https://github.com/smirkingface/stable-diffusion

https://github.com/smirkingface/stable-diffusion#news

Added support for inference and finetuning with the SD 2.0 base model (inpainting is still unsupported).

https://github.com/smirkingface/stable-diffusion/blob/main/docs/sd2.0.md

Originally posted by @0xdevalias in JoePenna/Dreambooth-Stable-Diffusion#112 (comment)

And looking at the huggingface/diffusers repo, there are a few issues that seem to imply people may be getting dreambooth things working with that (or at least trying to), eg.:

Dreambooth example on SD2-768 model is producing weird results #1429

Originally posted by @0xdevalias in JoePenna/Dreambooth-Stable-Diffusion#112 (comment)

vvvm23 · 2022-11-30T10:19:48Z

And available in 0.9.0! Damn fast job, guys!

Closing :)

vvvm23 closed this as completed Nov 30, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Stable Diffusion 2 #1392

Stable Diffusion 2 #1392

vvvm23 commented Nov 24, 2022 •

edited

Loading

0xdevalias commented Nov 24, 2022

0xdevalias commented Nov 24, 2022

0xdevalias commented Nov 24, 2022

averad commented Nov 24, 2022 •

edited

Loading

0xdevalias commented Nov 24, 2022

SD 2.0

0xdevalias commented Nov 24, 2022 •

edited

Loading

0xdevalias commented Nov 24, 2022

vvvm23 commented Nov 24, 2022

0xdevalias commented Nov 25, 2022

0xdevalias commented Nov 25, 2022

Marcophono2 commented Nov 25, 2022

0xdevalias commented Nov 26, 2022

0xdevalias commented Nov 26, 2022 •

edited

Loading

vvvm23 commented Nov 30, 2022

Stable Diffusion 2 #1392

Stable Diffusion 2 #1392

Comments

vvvm23 commented Nov 24, 2022 • edited Loading

Model/Pipeline/Scheduler description

Open source status

Provide useful links for the implementation

0xdevalias commented Nov 24, 2022

0xdevalias commented Nov 24, 2022

0xdevalias commented Nov 24, 2022

averad commented Nov 24, 2022 • edited Loading

🤗 Diffusers with Stable Diffusion 2 is live!

📰 News

✏️ Notes & Information

💭 User Story

0xdevalias commented Nov 24, 2022

SD 2.0

0xdevalias commented Nov 24, 2022 • edited Loading

0xdevalias commented Nov 24, 2022

vvvm23 commented Nov 24, 2022

0xdevalias commented Nov 25, 2022

0xdevalias commented Nov 25, 2022

Marcophono2 commented Nov 25, 2022

0xdevalias commented Nov 26, 2022

0xdevalias commented Nov 26, 2022 • edited Loading

vvvm23 commented Nov 30, 2022

vvvm23 commented Nov 24, 2022 •

edited

Loading

averad commented Nov 24, 2022 •

edited

Loading

0xdevalias commented Nov 24, 2022 •

edited

Loading

0xdevalias commented Nov 26, 2022 •

edited

Loading