IP-Adapter for StableDiffusion3InpaintPipeline

[guiyrt]

What does this PR do?

Part of #9966. Unlike #10561, this pipeline doesn't use a controlnet for inpainting, so we can test this pipeline. I tested with stabilityai/stable-diffusion-3.5-large, which wasn't fine-tuned for inpainting, but actually got decent results. Here are a few example outputs:

Inference code

import torch
from torchvision import transforms

from diffusers import StableDiffusion3InpaintPipeline
from diffusers.utils import load_image
from transformers import SiglipVisionModel, SiglipImageProcessor

def preprocess_image(image):
    image = image.convert("RGB")
    image = transforms.CenterCrop((image.size[1] // 64 * 64, image.size[0] // 64 * 64))(image)
    image = transforms.ToTensor()(image)
    image = image.unsqueeze(0).to("cuda")
    return image

def preprocess_mask(mask):
    mask = mask.convert("L")
    mask = transforms.CenterCrop((mask.size[1] // 64 * 64, mask.size[0] // 64 * 64))(mask)
    mask = transforms.ToTensor()(mask)
    mask = mask.to("cuda")
    return mask

model_id = "stabilityai/stable-diffusion-3.5-large"
image_encoder_id = "google/siglip-so400m-patch14-384"
ip_adapter_id = "guiyrt/InstantX-SD3.5-Large-IP-Adapter-diffusers"

feature_extractor = SiglipImageProcessor.from_pretrained(
    image_encoder_id, torch_dtype=torch.float16
)

image_encoder = SiglipVisionModel.from_pretrained(
    image_encoder_id, torch_dtype=torch.float16
)

pipe = StableDiffusion3InpaintPipeline.from_pretrained(
    model_id,
    torch_dtype=torch.float16,
    feature_extractor=feature_extractor,
    image_encoder=image_encoder,
)

# Load IP Adapter
pipe.load_ip_adapter(ip_adapter_id)
pipe.set_ip_adapter_scale(0.75)
pipe._exclude_from_cpu_offload.append("image_encoder")
pipe.enable_sequential_cpu_offload()

# Input
source_image = preprocess_image(load_image("source_image.png"))
mask = preprocess_mask(load_image("mask.png"))
ip_adapter_img = load_image("cat.jpg")

# please note that SD3.5 Large is sensitive to highres generation like 1536x1536
images = pipe(
    image=source_image,
    mask_image=mask,
    width=1024,
    height=1024,
    prompt="a cat sitting on a bench",
    negative_prompt="lowres, low quality, worst quality",
    num_images_per_prompt=4,
    generator=torch.manual_seed(42),
    ip_adapter_image=ip_adapter_img,
    guidance_scale=3.5,
    num_inference_steps=60,
    strength=0.95,
).images

for i, image in enumerate(images):
    image.save(f"result_{i}.jpg")

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline?
• Did you read our philosophy doc (important for complex PRs)?
• Was this discussed/approved via a GitHub issue or the forum? Please add a link to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

@hlky @yiyixuxu

[yiyixuxu]

thanks!

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[hlky]

Thanks @guiyrt