Converting from Stable diffusion v2.1 CKPT or safetensors to diffusers, then back to CKPT/safetensors breaks the model

[TheLastBen]

Describe the bug

I noticed when converting a v2.1 SD model to diffusers then back to CKPT again makes the model unusable.

Reproduction

Try converting https://huggingface.co/stabilityai/stable-diffusion-2-1/resolve/main/v2-1_768-ema-pruned.ckpt here https://colab.research.google.com/github/TheLastBen/fast-stable-diffusion/blob/main/fast-DreamBooth.ipynb, train it for only 20 steps to allow convertion to ckpt, then test the model, the output will be random noise.

Logs

No response

System Info

Colab

[patrickvonplaten]

Hey @TheLastBen can you share your full conversion commands here?

[TheLastBen]

It works fine for v1.5, but when using the v2 command, the model becomes corrupt, this command is for 768 for example :

!wget -q -O config.yaml https://github.com/Stability-AI/stablediffusion/raw/main/configs/stable-diffusion/v2-inference-v.yaml
!python /content/diffusers/scripts/convert_original_stable_diffusion_to_diffusers.py --checkpoint_path $MDLPTH --dump_path "$OUTPUT_DIR" --original_config_file config.yaml --upcast_attention --prediction_type v_prediction --image_size 768

[patrickvonplaten]

Hmmm, weird and that's for the official SD v2.1 model? Or a fine-tuned one from your notebooks?

[TheLastBen]

the official one, the script I'm using currently works, but the default one doesn't, I think the issue comes from the text encoder conversion

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.