VAE not work,when use StableDiffusionPipeline.from_single_file

[zhaoyun0071]

Describe the bug

pipe = StableDiffusionPipeline.from_single_file("xx.safetensors",
                                                torch_dtype=torch.float16, load_safety_checker=False, controlnet=False,vae=AutoencoderKL.from_pretrained("stabilityai/sd-vae-ft-mse",torch_dtype=torch.float16).to("cuda")
                                                ).to("cuda")

**above code ,vae not work!! pictures use same seed with or without vae have no difference

but bellow work ,hahhaaaaa**

pipe = StableDiffusionPipeline.from_single_file("xx.safetensors",
                                                torch_dtype=torch.float16, load_safety_checker=False, controlnet=False
                                                ).to("cuda")
pipe.vae=AutoencoderKL.from_pretrained("stabilityai/sd-vae-ft-mse",torch_dtype=torch.float16).to("cuda")

Reproduction

above code

Logs

No response

System Info

diffusers-0.19.0.dev0 6b1abba

Who can help?

@patrickvonplaten, @sayakpaul, and @williamberman

[patrickvonplaten]

I have no idea what this issue is supposed to mean

[zhaoyun0071]

I have no idea what this issue is supposed to mean

pipe = StableDiffusionPipeline.from_single_file("xx.safetensors",
torch_dtype=torch.float16, load_safety_checker=False, controlnet=False,vae=AutoencoderKL.from_pretrained("stabilityai/sd-vae-ft-mse",torch_dtype=torch.float16).to("cuda")
).to("cuda")
vae in this pipe not load。

[sayakpaul]

Could we have a fully reproducible and self-contained code snippet please?

[sayakpaul]

@zhaoyun0071 I can do the following:

from diffusers import StableDiffusionXLPipeline, AutoencoderKL
import torch

pipe = StableDiffusionXLPipeline.from_single_file(
    "https://huggingface.co/stabilityai/stable-diffusion-xl-base-0.9/blob/main/sd_xl_base_0.9.safetensors",
    torch_dtype=torch.float16,
    use_safetensors=True,
    variant="fp16"
)

vae_2 = AutoencoderKL.from_pretrained("madebyollin/sdxl-vae-fp16-fix", torch_dtype=torch.float16)
pipe.vae = vae_2

Isn't this what you're looking for?

[zhaoyun0071]

@zhaoyun0071 I can do the following:

from diffusers import StableDiffusionXLPipeline, AutoencoderKL
import torch

pipe = StableDiffusionXLPipeline.from_single_file(
    "https://huggingface.co/stabilityai/stable-diffusion-xl-base-0.9/blob/main/sd_xl_base_0.9.safetensors",
    torch_dtype=torch.float16,
    use_safetensors=True,
    variant="fp16"
)

vae_2 = AutoencoderKL.from_pretrained("madebyollin/sdxl-vae-fp16-fix", torch_dtype=torch.float16)
pipe.vae = vae_2

Isn't this what you're looking for?

yes this is right

but when vae used as a para of from_single can't work

[sayakpaul]

Could you please provide a reproducible snippet preferably in the format which I followed?

[dai-ichiro]

wget https://huggingface.co/gsdf/Counterfeit-V2.5/resolve/main/Counterfeit-V2.5_pruned.safetensors
wget https://huggingface.co/gsdf/Counterfeit-V2.5/resolve/main/Counterfeit-V2.5.vae.pt

Method 0 (default VAE)

from diffusers import StableDiffusionPipeline
import torch

pipe = StableDiffusionPipeline.from_single_file(
    "Counterfeit-V2.5_pruned.safetensors",
    load_safety_checker=False
    )
pipe.to("cuda")

prompt = "1girl"

seed = 20000

generator = torch.Generator(device="cuda").manual_seed(seed)
image = pipe(
    prompt=prompt,
    generator=generator,
    ).images[0]
image.save("method0_result.png")

Method 1

from diffusers import StableDiffusionPipeline, AutoencoderKL
import torch

pipe = StableDiffusionPipeline.from_single_file(
    "Counterfeit-V2.5_pruned.safetensors",
    vae=AutoencoderKL.from_single_file("Counterfeit-V2.5.vae.pt"),
    load_safety_checker=False
    )
pipe.to("cuda")

prompt = "1girl"

seed = 20000

generator = torch.Generator(device="cuda").manual_seed(seed)
image = pipe(
    prompt=prompt,
    generator=generator,
    ).images[0]
image.save("method1_result.png")

Method 2

from diffusers import StableDiffusionPipeline, AutoencoderKL
import torch

pipe = StableDiffusionPipeline.from_single_file(
    "Counterfeit-V2.5_pruned.safetensors",
    load_safety_checker=False
    )
pipe.vae = vae=AutoencoderKL.from_single_file("Counterfeit-V2.5.vae.pt")
pipe.to("cuda")

prompt = "1girl"

seed = 20000

generator = torch.Generator(device="cuda").manual_seed(seed)
image = pipe(
    prompt=prompt,
    generator=generator,
    ).images[0]
image.save("method2_result.png")

Method 0 and Method 1 have the same results because vae not works.
I think method 2 is the right way. But perhaps some people prefer method 1 to method 2.

[zhaoyun0071]

wget https://huggingface.co/gsdf/Counterfeit-V2.5/resolve/main/Counterfeit-V2.5_pruned.safetensors
wget https://huggingface.co/gsdf/Counterfeit-V2.5/resolve/main/Counterfeit-V2.5.vae.pt

Method 0 (default VAE)

from diffusers import StableDiffusionPipeline
import torch

pipe = StableDiffusionPipeline.from_single_file(
    "Counterfeit-V2.5_pruned.safetensors",
    load_safety_checker=False
    )
pipe.to("cuda")

prompt = "1girl"

seed = 20000

generator = torch.Generator(device="cuda").manual_seed(seed)
image = pipe(
    prompt=prompt,
    generator=generator,
    ).images[0]
image.save("method0_result.png")

Method 1

from diffusers import StableDiffusionPipeline, AutoencoderKL
import torch

pipe = StableDiffusionPipeline.from_single_file(
    "Counterfeit-V2.5_pruned.safetensors",
    vae=AutoencoderKL.from_single_file("Counterfeit-V2.5.vae.pt"),
    load_safety_checker=False
    )
pipe.to("cuda")

prompt = "1girl"

seed = 20000

generator = torch.Generator(device="cuda").manual_seed(seed)
image = pipe(
    prompt=prompt,
    generator=generator,
    ).images[0]
image.save("method1_result.png")

Method 2

from diffusers import StableDiffusionPipeline, AutoencoderKL
import torch

pipe = StableDiffusionPipeline.from_single_file(
    "Counterfeit-V2.5_pruned.safetensors",
    load_safety_checker=False
    )
pipe.vae = vae=AutoencoderKL.from_single_file("Counterfeit-V2.5.vae.pt")
pipe.to("cuda")

prompt = "1girl"

seed = 20000

generator = torch.Generator(device="cuda").manual_seed(seed)
image = pipe(
    prompt=prompt,
    generator=generator,
    ).images[0]
image.save("method2_result.png")

Method 0 and Method 1 have the same results because vae not works. I think method 2 is the right way. But perhaps some people prefer method 1 to method 2.

That's what I mean, why can't developers understand

[sayakpaul]

from diffusers import StableDiffusionPipeline, AutoencoderKL
import torch

pipe = StableDiffusionPipeline.from_single_file(
    "Counterfeit-V2.5_pruned.safetensors",
    load_safety_checker=False
    )
pipe.vae = vae=AutoencoderKL.from_single_file("Counterfeit-V2.5.vae.pt")
pipe.to("cuda")

prompt = "1girl"

seed = 20000

generator = torch.Generator(device="cuda").manual_seed(seed)
image = pipe(
    prompt=prompt,
    generator=generator,
    ).images[0]
image.save("method2_result.png")

Should be the recommended method as IIUC while using from_single_file() the provided keyword arguments (like vae) are overridden by what's constructed as a part of unpacking the single file checkpoint. Not sure if this needs fixing.

Ccing @patrickvonplaten for further details if I am missing any.

[patrickvonplaten]

@dai-ichiro @zhaoyun0071 thanks for clarifying - I now understand the problem I think. The problem was that passing the vae to from_single_file had no effect. Can you check if #4242 works?

[dai-ichiro]

pip install torch==2.0.1+cu118 --index-url https://download.pytorch.org/whl/cu118
pip install git+https://github.com/huggingface/diffusers.git@allow_vae_to_be_loaded_from_single_file
pip install accelerate transformers safetensors omegaconf pytorch-lightning

wget https://huggingface.co/gsdf/Counterfeit-V2.5/resolve/main/Counterfeit-V2.5_pruned.safetensors
wget https://huggingface.co/gsdf/Counterfeit-V2.5/resolve/main/Counterfeit-V2.5.vae.pt

from diffusers import StableDiffusionPipeline, AutoencoderKL
import torch

pipe = StableDiffusionPipeline.from_single_file(
    "Counterfeit-V2.5_pruned.safetensors",
    vae=AutoencoderKL.from_single_file("Counterfeit-V2.5.vae.pt"),
    load_safety_checker=False
    )
pipe.to("cuda")

prompt = "1girl"

seed = 20000

generator = torch.Generator(device="cuda").manual_seed(seed)
image = pipe(
    prompt=prompt,
    generator=generator,
    ).images[0]
image.save("method1_result.png")

It works fine. Thanks you.

[sayakpaul]

Closing the issue then.