support from_single_file for SDXL inpainting

[yiyixuxu]

This PR adds support to use from_single_file for StableDiffusionXLInpaintPipeline

fix
#4402
and
#4354

from diffusers import StableDiffusionXLInpaintPipeline, StableDiffusionXLPipeline
import torch

from diffusers.utils import load_image


img_url = "https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo.png"
mask_url = "https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo_mask.png"

init_image = load_image(img_url).convert("RGB")
mask_image = load_image(mask_url).convert("RGB")

prompt = "A majestic tiger sitting on a bench"

pipe = StableDiffusionXLInpaintPipeline.from_single_file(
    "https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/blob/main/sd_xl_base_1.0.safetensors",
    torch_dtype=torch.float16, 
    use_safetensors=True)

print(type(pipe))

pipe = pipe.to("cuda")

generator = torch.Generator(device="cuda").manual_seed(0)
image = pipe(prompt=prompt, image=init_image, mask_image=mask_image, num_inference_steps=50, strength=0.80, generator=generator).images[0]
image.save("inpaint_from_single_out.png")


pipe = StableDiffusionXLInpaintPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0", torch_dtype=torch.float16, use_safetensors=True
)
pipe = pipe.to("cuda")

generator = torch.Generator(device="cuda").manual_seed(0)
image = pipe(prompt=prompt, image=init_image, mask_image=mask_image, num_inference_steps=50, strength=0.80, generator=generator).images[0]
image.save("inpaint_from_pretrained_out.png")

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[patrickvonplaten]

Nice!

[jwaitzel]

Hi everyone, thanks for all the amazing work! 🙌
Is this commit in the last release or it will be in a new one?

Thanks!

[yiyixuxu]

@jwaitzel in next one

[skyheros001]

nice!nice!

[segatecm]

It is not work for me:
Diffusers V0.20.1

code:
pipe = StableDiffusionXLInpaintPipeline.from_single_file(
"XL/sd_xl_refiner_1.0.safetensors", torch_dtype=torch.float16, variant="fp16", use_safetensors=True
)
pipe.to("cuda")
print(type(pipe))
img_url = "https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo.png"
mask_url = "https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo_mask.png"

init_image = load_image(img_url).convert("RGB")
mask_image = load_image(mask_url).convert("RGB")

prompt = "A majestic tiger sitting on a bench"
image = pipe(prompt=prompt, image=init_image, mask_image=mask_image, num_inference_steps=50, strength=0.80).images[0]

result:
<class 'diffusers.pipelines.stable_diffusion_xl.pipeline_stable_diffusion_xl_img2img.StableDiffusionXLImg2ImgPipeline'>
TypeError: StableDiffusionXLImg2ImgPipeline.call() got an unexpected keyword argument 'mask_image'