Bug: Inpaint with IP Adapter (face+style)

[tin2tin]

Describe the bug

Inpaint with IP Adapter(face+style) doesn't seem to work. I'm getting this error, if I run the following code:

RuntimeError: mat1 and mat2 shapes cannot be multiplied (5140x1664 and 1280x1280)

Reproduction

import torch
from diffusers import DDIMScheduler, AutoPipelineForInpainting
from transformers import CLIPVisionModelWithProjection
from diffusers.utils import load_image

image_encoder = CLIPVisionModelWithProjection.from_pretrained(
    "h94/IP-Adapter",
    subfolder="models/image_encoder",
    torch_dtype=torch.float16,
)
pipeline = AutoPipelineForInpainting.from_pretrained("diffusers/stable-diffusion-xl-1.0-inpainting-0.1", torch_dtype=torch.float16).to("cuda")

pipeline.scheduler = DDIMScheduler.from_config(pipeline.scheduler.config)
pipeline.load_ip_adapter(
  "h94/IP-Adapter",
  subfolder="sdxl_models",
  weight_name=["ip-adapter-plus_sdxl_vit-h.safetensors", "ip-adapter-plus-face_sdxl_vit-h.safetensors"]
)
pipeline.set_ip_adapter_scale([0.7, 0.3])
pipeline.enable_model_cpu_offload()

mask_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/ip_adapter_mask.png")
image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/ip_adapter_bear_1.png")

face_image = load_image("https://huggingface.co/datasets/YiYiXu/testing-images/resolve/main/women_input.png")
style_folder = "https://huggingface.co/datasets/YiYiXu/testing-images/resolve/main/style_ziggy"
style_images = [load_image(f"{style_folder}/img{i}.png") for i in range(10)]

generator = torch.Generator(device="cpu").manual_seed(0)

image = pipeline(
    prompt="a cute wonderwoman waving",
    image=image,
    ip_adapter_image=[style_images, face_image],
    mask_image=mask_image,
    negative_prompt="monochrome, lowres, bad anatomy, worst quality, low quality",
    num_inference_steps=50, num_images_per_prompt=1,
    generator=generator,
).images[0]
image

Logs

RuntimeError: mat1 and mat2 shapes cannot be multiplied (5140x1664 and 1280x1280)

System Info

Diffusers: latest github beta.
Win: 11

Who can help?

@sayakpaul @yiyixuxu @DN6

[tolgacangoz]

It seems that you created an image_encoder object but forgot to give it to AutoPipelineForInpainting.from_pretrained(..., image_encoder=image_encoder, ...).
Also, try not to use .to("cuda") when using pipeline.enable_model_cpu_offload().

[tin2tin]

Thanks! I guess, my blindness was due to a misleading error message.