fix torch.compile for multi-controlnet of sdxl inpaint

[yiyixuxu]

fix #7466

this now run

from diffusers import StableDiffusionXLControlNetInpaintPipeline, ControlNetModel
import torch
from PIL import Image

controlnet_canny = ControlNetModel.from_pretrained(
"diffusers/controlnet-canny-sdxl-1.0", torch_dtype=torch.float16
).to("cuda")

# inpaint
sdxl = StableDiffusionXLControlNetInpaintPipeline.from_pretrained(
"stabilityai/stable-diffusion-xl-base-1.0",
torch_dtype=torch.float16,
variant="fp16",
use_safetensors=True,
controlnet=[controlnet_canny,controlnet_canny],
vae=vae,
add_watermarker=False
).to('cuda')

torch._dynamo.config.verbose = True
sdxl.controlnet.to(memory_format=torch.channels_last)
sdxl.controlnet = torch.compile(sdxl.controlnet, mode="reduce-overhead", fullgraph=True)

out = sdxl(
    prompt="dog",
    negative_prompt="dog",
    num_inference_steps=10,
    guidance_scale=4,
    strength=1.0,
    image=Image.new(mode="RGB", size=(200, 200)),
    mask_image=Image.new(mode="RGB", size=(200, 200)),
    control_image=[Image.new(mode="RGB", size=(200, 200)), Image.new(mode="RGB", size=(200, 200))],
    controlnet_conditioning_scale= [1.0, 0.3],
    )

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[sayakpaul]

I don't understand the logical difference between these two though 🥲

[sayakpaul]

Thanks!