fix a bug in inpaint pipeline when use regular text2image unet

[yiyixuxu]

This PR fixes the bug reported here #4854

The issue comes from this line of code here - when we use inpainting pipeline with a regular text-to-image unet with 4 channels, it will only use the first image when given more than one image inputs.

diffusers/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_inpaint.py

Line 1002 in 19edca8

init_latents_proper = image_latents[:1]

I'm not sure if there is a reason it was currently done this way though.

This code currently fails but will be fixed in this PR

import PIL
import requests
import torch
from io import BytesIO

from diffusers import StableDiffusionInpaintPipeline


def download_image(url):
    response = requests.get(url)
    return PIL.Image.open(BytesIO(response.content)).convert("RGB")


imgs = []
masks = []

for img_url, mask_url in [
    ("https://i.ibb.co/DM9qbW8/x0.png","https://i.ibb.co/4TTHp6w/xxx.png"),
    ("https://i.ibb.co/J7DXFmB/x1.png","https://i.ibb.co/2gkxyWB/x1.png"),
    ("https://i.ibb.co/8gRgDYG/x3.png","https://i.ibb.co/WvZ7Q5h/x2.png"),
    ("https://i.ibb.co/KXVWT5D/x4.png","https://i.ibb.co/P5rTZFQ/x3.png")]:

    imgs.append(download_image(img_url))
    masks.append(download_image(mask_url))

pipe = StableDiffusionInpaintPipeline.from_pretrained(
    "runwayml/stable-diffusion-v1-5",
    #"rv51_inpaint",
    revision="fp16",
    torch_dtype=torch.float16,
)
pipe = pipe.to("cuda")

prompt = "Face of a yellow cat, high resolution, sitting on a park bench"

images = pipe(
    prompt=[prompt]*4, 
    image=imgs, 
    mask_image=masks,
    width=512,
    height=512
).images

for i, image in enumerate(images):
    image.save(f"yiyi_test_1_out_{i}.png")

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[patrickvonplaten]

Thanks for the fix, can you also double-check if other pipelines need to be updated? E.g. controlnet inpainting and SDXL inpainting

[patrickvonplaten]

Let's also please add one fast test :-)

[patrickvonplaten]

Tests is flaky, let's merge

[landmann]

Is this merged onto main 😁 ?