Stable diffusion inpainting new version faces a bug

[FBehrad]

Describe the bug

I used to use stable diffusion inpainting legacy. When I used num_images_per_prompt = 2, the model generated two similar images, while I expected to get two different outputs.
Then, I cloned the new version of the stable diffusion inpainting to tackle this problem. However, it does not work.
The problem is in line 354:

if num_channels_latents + num_channels_mask + num_channels_masked_image != self.unet.config.in_channels:
            raise ValueError(
                f"Incorrect configuration settings! The config of `pipeline.unet`: {self.unet.config} expects"
                f" {self.unet.config.in_channels} but received `num_channels_latents`: {num_channels_latents} +"
                f" `num_channels_mask`: {num_channels_mask} + `num_channels_masked_image`: {num_channels_masked_image}"
                f" = {num_channels_latents+num_channels_masked_image+num_channels_mask}. Please verify the config of"
                " `pipeline.unet` or your `mask_image` or `image` input."
            )

My parameters:
The number of input channels in unet = 4
num_channels_latents = 4
num_channels_mask = 1
num_channels_masked_image = 4

What is the problem?

Reproduction

No response

Logs

No response

System Info

• diffusers version: 0.6.0
• Platform: Windows-10-10.0.19044-SP0
• Python version: 3.8.10
• PyTorch version (GPU?): 1.12.1+cu113 (True)
• Huggingface_hub version: 0.10.1
• Transformers version: 4.23.1
• Using GPU in script?:
• Using distributed or parallel set-up in script?:

[patrickvonplaten]

Hey @FBehrad,

Could you add a reproducible code snippet? I'm not 100% sure what code you ran 😅

Thanks!

[Kenoubi]

@FBehrad, if you want to use the new inpainting pipeline, you have to use the finetuned model for inpainting (runwayml/stable-diffusion-inpainting). The other models do not have the right number of channels in the U-Net to work with the new inpainting pipeline. I'm pretty sure this is why your code didn't work.

[RobertLucian]

@FBehrad I've also been stuck with this exact error for a while until I realized I wasn't initializing the model correctly. I was doing

pipe = StableDiffusionInpaintPipeline.from_pretrained(
    "runwayml/runwayml/stable-diffusion-v1-5",
    revision="fp16",
    torch_dtype=torch.float16,
)

Instead of

pipe = StableDiffusionInpaintPipeline.from_pretrained(
    "runwayml/stable-diffusion-inpainting",
    revision="fp16",
    torch_dtype=torch.float16,
)

[FBehrad]

Thank you so much for your quick response.
I was loading wrong weights which caused this problem. Using "runwayml/stable-diffusion-inpainting" solved my problem.

[jbmaxwell]

I just hit his error and found my way here.

Unfortunately, in my case I'm not using the generic pretrained image model, so just loading a different model won't help. Is there any guidance on how to pretrain for inpainting? Or can I fine-tune my current text-conditioned model on the inpainting task?

I notice it's the same UNet class, so perhaps it's just a different input shape?... Is the mask added to an input channel?

Any help appreciated.

EDIT: Searching around I see: https://github.com/huggingface/diffusers/tree/main/examples/research_projects/dreambooth_inpaint
Will have a deeper look.

[GoulartNogueira]

@jbmaxwell I'm in the same scenario
Did you solve it?

[jbmaxwell]

Hi @GoulartNogueira, unfortunately I just abandoned inpainting for now. Though it is obviously an interesting and useful feature, so I'd be interested in picking it up again at some stage.