[0.28.0.dev0] In ControlNetModel forward function, added_cond_kwargs arg's default value should be {} rather than None

[CatLoves]

Describe the bug

Version: [0.28.0.dev0]
Bug description: In ControlNetModel forward function, added_cond_kwargs arg's default value should be {} rather than None
Reason: in the source code(controlnet.py):
elif self.config.addition_embed_type == "text_time":
if "text_embeds" not in added_cond_kwargs:
raise ValueError(
f"{self.class} has the config param addition_embed_type set to 'text_time' which requires the keyword argument text_embeds to be passed in added_cond_kwargs"
)
If the default value is None, this code would raise a ValueError
Pull request: I intend to submit a pull request for this issue.

Reproduction

from diffusers import AutoPipelineForInpainting, StableDiffusionControlNetInpaintPipeline,ControlNetModel
from diffusers.utils import load_image
import torch

controlnet_canny = ControlNetModel.from_pretrained(
"xinsir/controlnet-canny-sdxl-1.0",
torch_dtype=torch.float16
)
print(f"=> controlnet_canny is ready")
pipe = StableDiffusionControlNetInpaintPipeline.from_pretrained(
"diffusers/stable-diffusion-xl-1.0-inpainting-0.1",
controlnet=controlnet_canny,
torch_dtype=torch.float16, variant="fp16",
).to("cuda")

img_url = "https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo.png"
mask_url = "https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo_mask.png"

image = load_image(img_url).resize((1024, 1024))
mask_image = load_image(mask_url).resize((1024, 1024))
canny_image = get_canny_edge(image)

prompt = "a tiger sitting on a park bench"
generator = torch.Generator(device="cuda").manual_seed(0)

generated_image = pipe(
prompt=prompt,
image=image,
mask_image=mask_image,
control_image=canny_image,
guidance_scale=8.0,
num_inference_steps=20, # steps between 15 and 30 work well for us
strength=0.99, # make sure to use strength below 1.0
generator=generator,
).images[0]

Logs

No response

System Info

diffusers-cli env
2024-06-03 03:28:35.669077: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable TF_ENABLE_ONEDNN_OPTS=0.
2024-06-03 03:28:35.758278: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-06-03 03:28:35.758367: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-06-03 03:28:35.765096: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-06-03 03:28:35.790633: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-06-03 03:28:37.281435: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
Traceback (most recent call last):
File "/training/home/project/stable-diffusion-webui/venv/bin/diffusers-cli", line 8, in
sys.exit(main())
File "/training/home/project/stable-diffusion-webui/venv/lib/python3.9/site-packages/diffusers/commands/diffusers_cli.py", line 39, in main
service.run()
File "/training/home/project/stable-diffusion-webui/venv/lib/python3.9/site-packages/diffusers/commands/env.py", line 101, in run
bitsandbytes_version = bitsandbytes.version
AttributeError: module 'bitsandbytes' has no attribute 'version'

Who can help?

@sayakpaul @yiyixuxu

[CatLoves]

Update diffusers-cli env info:

• 🤗 Diffusers version: 0.28.0.dev0
• Platform: Linux-5.4.143-2-velinux1-amd64-x86_64-with-glibc2.31
• Running on a notebook?: No
• Running on Google Colab?: No
• Python version: 3.9.17
• PyTorch version (GPU?): 2.1.2+cu118 (True)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Huggingface_hub version: 0.23.0
• Transformers version: 4.41.2
• Accelerate version: 0.21.0
• PEFT version: not installed
• Bitsandbytes version: 0.43.1
• Safetensors version: 0.4.3
• xFormers version: 0.0.23.post1
• Accelerator: Tesla V100-SXM2-32GB, 32510 MiB
  Tesla V100-SXM2-32GB, 32510 MiB VRAM
• Using GPU in script?:
• Using distributed or parallel set-up in script?:

[sayakpaul]

I don't think we need to default to {} here because it becomes effective only when we're using SDXL models. WDYT @yiyixuxu?

[CatLoves]

I don't think we need to default to {} here because it becomes effective only when we're using SDXL models. WDYT @yiyixuxu?

Yes, you are right. Thank you for quick response! But this naturally lead to one question: when the user use SDXL model, why does the code raise the ValueError exception? The minimal replication code snippet is the following:

from diffusers import AutoPipelineForInpainting, StableDiffusionControlNetInpaintPipeline,ControlNetModel
from diffusers.utils import load_image
import torch

""" build diffusion model pipeline """
controlnet_canny = ControlNetModel.from_pretrained(
"xinsir/controlnet-canny-sdxl-1.0",
torch_dtype=torch.float16
)
print(f"=> controlnet_canny is ready")
pipe = StableDiffusionControlNetInpaintPipeline.from_pretrained(
"diffusers/stable-diffusion-xl-1.0-inpainting-0.1",
controlnet=controlnet_canny,
torch_dtype=torch.float16, variant="fp16",
).to("cuda")

img_url = "https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo.png"
mask_url = "https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo_mask.png"

image = load_image(img_url).resize((1024, 1024))
mask_image = load_image(mask_url).resize((1024, 1024))
canny_image = get_canny_edge(image)

prompt = "a tiger sitting on a park bench"
generator = torch.Generator(device="cuda").manual_seed(0)

generated_image = pipe(
prompt=prompt,
image=image,
mask_image=mask_image,
control_image=canny_image,
guidance_scale=8.0,
num_inference_steps=20, # steps between 15 and 30 work well for us
strength=0.99, # make sure to use strength below 1.0
generator=generator,
).images[0]

The above code would encounter the ValueError as added_cond_kwargs is None by default. So, currently, diffusers just doesn't support diffusers/stable-diffusion-xl-1.0-inpainting-0.1 + xinsir/controlnet-canny-sdxl-1.0 pipeline ？Or this is just a bug of diffusers ?
BTW: The above code would be OK if I just use lllyasviel/control_v11p_sd15_inpaint + lllyasviel/control_v11p_sd15_canny pipeline.

[sayakpaul]

You are not using the right class. You need to use this:

diffusers/src/diffusers/pipelines/controlnet/pipeline_controlnet_inpaint_sd_xl.py

Line 154 in 4136044

class StableDiffusionXLControlNetInpaintPipeline(

[CatLoves]

You are not using the right class. You need to use this:

diffusers/src/diffusers/pipelines/controlnet/pipeline_controlnet_inpaint_sd_xl.py

Line 154 in 4136044

class StableDiffusionXLControlNetInpaintPipeline(

Yes, you are right, after switching to StableDiffusionXLControlNetInpaintPipeline, the code just runs fine. Thanks again for your kind and quick response! I would close this issue now.