getting bug on diffuser example

[xalteropsx]

modelp is "dreamshaper_8Inpainting.safetensors"
https://huggingface.co/docs/diffusers/v0.29.0/en/api/pipelines/controlnet_sdxl#diffusers.StableDiffusionXLControlNetInpaintPipeline

tried compel as well it failed

using diffuser 0.29.0

[haofanwang]

dreamshaper_8Inpainting.safetensors is not a SDXL model?

[xalteropsx]

@haofanwang thnx alot bro was mistakenly though xl

its pretty weird how it able to load the model without giving us warning xD

[xalteropsx]

@haofanwang sorry for trouble u again but it not load the sdxl model rather it load sd model

[asomoza]

Hi, can you please post the code as text and not as an image, it makes it a lot harder to read and to test.

[xalteropsx]

@asomoza

import cv2
from diffusers import StableDiffusionXLControlNetInpaintPipeline, ControlNetModel, DDIMScheduler
from diffusers.utils import load_image
import numpy as np
import torch

init_image = load_image("https://huggingface.co/datasets/diffusers/test-arrays/resolve/main/stable_diffusion_inpaint/boy.png")

init_image = init_image.resize((1024, 1024))

generator = torch.Generator(device="cpu").manual_seed(1)

mask_image = load_image("https://huggingface.co/datasets/diffusers/test-arrays/resolve/main/stable_diffusion_inpaint/boy_mask.png")

mask_image = mask_image.resize((1024, 1024))
modelp = "models_painting/dreamshaperXL_lightningInpaint.safetensors"

def make_canny_condition(image):
    image = np.array(image)
    image = cv2.Canny(image, 100, 200)
    image = image[:, :, None]
    image = np.concatenate([image, image, image], axis=2)
    image = Image.fromarray(image)
    return image


control_image = make_canny_condition(init_image)

controlnet = ControlNetModel.from_pretrained("diffusers/controlnet-canny-sdxl-1.0",variant="fp16", torch_dtype=torch.float16)

pipe = StableDiffusionXLControlNetInpaintPipeline.from_single_file(modelp, controlnet=controlnet, torch_dtype=torch.float16)

pipe.enable_model_cpu_offload()

image = pipe(
    "a handsome man with ray-ban sunglasses",
    num_inference_steps=20,
    generator=generator,
    eta=1.0,
    image=init_image,
    mask_image=mask_image,
    control_image=control_image,
).images[0]

between bro it was same code of ur example was provided the link on top

havent tested with other model just low on space right now so only some inpaint model
https://civitai.com/models/403751/dreamshaper-xl-lightning-inpainting
but u can try with other xl model not sure if it work with singlefile

[asomoza]

I learned the hard way that almost never is the same code, that's why we kindly ask for the same minimal reproducible code you can share for the issues.

For example, just out of the box, you're using pillow (without the import?) and from_single_file and the example does not.

Nevertheless I tested it with the same model and with one that I know it worked before and I can reproduce this issue, it seems this happens with the from_single_file refactor and the single file inpaint checkpoints. Pinging @DN6 for this.

Probably this won't be resolved fast so I suggest you use a diffusers format one like this one or install the 0.27.2 version of diffusers.

P.S.: the pillow error is in the docs too, my mistake. cc: @stevhliu

[xalteropsx]

ah thnx bro again guess will follow ur link >.</
sorry forget

[sayakpaul]

Closing this as this seems to have been resolved. Feel free to reopen.