Add a ControlNet model & pipeline

[takuma104]

ControlNet by @lllyasviel is a neural network structure to control diffusion models by adding extra conditions. Discussed in #2331.

Usage Example

• Document: https://huggingface.co/docs/diffusers/main/en/api/pipelines/stable_diffusion/controlnet
• Google Colab Notebook

TODO:

• Conflict fixes
• Add utility function to generate control embedding: controlnet_hinter
• Docstring fixes
• Unit test fixes and additions to acceptable
• Test with other ControlNet models (probably subjective comparative evaluation with references): result report
  [ ] (time permitting) Investigate why pixel matches are not perfect

Discussion:

1. How to provide users with functions to create control embedding? The reference implementation is highly dependent on other libraries, and porting it without thought will increase the number of dependent libraries for Diffusers. From my observations, the most common use case is not Canny Edge conversion, but rather inputting a similar line drawing image (for example), or generating an OpenPose compatible image (for example) and then inputting that image. In such use cases, it may be acceptable to prepare only a minimal conversion from the image.
   -- one idea: controlnet_hinter
2. There is little difference between StableDiffusionPipeline and the (newly added) StableDiffusionControlNetPipeline. I think it would be preferable to change the behavior of StableDiffusionPipeline instead of creating a new StableDiffusionControlNetPipeline.
3. How much test code is required? Diffusers does a lot of coverage testing, which is great, but I'm having a hard time physically doing this amount of coding. I would appreciate it if someone could help me.
4. I currently have the converted models on my Huggingface hub, which I think is more appropriate for the @lllyasviel account. I would appreciate it if Huggingface team could support on release.

@williamberman @apolinario @xvjiarui

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[williamberman]

phenomenal @takuma104! let's collaborate on this PR :)

[takuma104]

@williamberman Thanks! Let's brush up to better code together.

[patrickvonplaten]

Alright, I think we can merge this PR now 🥳

After some testing, I would recommend to use the following settings for ControlNet:

# !pip install opencv-python transformers accelerate
from diffusers import StableDiffusionControlNetPipeline, ControlNetModel, UniPCMultistepScheduler
from diffusers.utils import load_image
import numpy as np
import torch

import cv2
from PIL import Image

# download an image
image = load_image(
    "https://hf.co/datasets/huggingface/documentation-images/resolve/main/diffusers/input_image_vermeer.png"
)
image = np.array(image)

# get canny image
image = cv2.Canny(image, 100, 200)
image = image[:, :, None]
image = np.concatenate([image, image, image], axis=2)
canny_image = Image.fromarray(image)

# load control net and stable diffusion v1-5
controlnet = ControlNetModel.from_pretrained("lllyasviel/sd-controlnet-canny", torch_dtype=torch.float16)
pipe = StableDiffusionControlNetPipeline.from_pretrained(
    "runwayml/stable-diffusion-v1-5", controlnet=controlnet, torch_dtype=torch.float16
)

# speed up diffusion process with faster scheduler and memory optimization
pipe.scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config)
# remove following line if xformers is not installed
pipe.enable_xformers_memory_efficient_attention()

pipe.enable_model_cpu_offload()

# generate image
generator = torch.manual_seed(0)
image = pipe(
    "futuristic-looking woman", num_inference_steps=20, generator=generator, image=canny_image
).images[0]

I've also updated the official example to this code snippet. The advantages:

• We use the currently fastest sampler (UniPCMultistepScheduler) which usually only need 20-25 steps
• We use smart model offloading which doesn't hurt speed performance really
• We use memory efficient attention a.k.a x-formers.

=> This allows us to run the code in ~3 seconds on a V100 with max 4.5GB RAM (< 4GB VRAM). Think this should make the model very useable for the community!

Amazing work @takuma104 - it's super cool that you led the integration here and iterated with us on the design choices.

[patrickvonplaten]

@takuma104 - left a comment here for future readers for credit

[takuma104]

@patrickvonplaten Thank you so much! It is a great honor!

[patrickvonplaten]

@takuma104 , I've changed the official checkpoints to the ones that reside in the ControlNet's authors repo: https://huggingface.co/lllyasviel instead of using the ones of your namespace - I hope this is ok! I've made sure to mark down here: https://github.com/huggingface/diffusers/pull/2407/files#r1123177670 that you contributed the model.
I think it's better to just have the controlnet weights in the individual models to make it trivial to switch out different controlnets with SD-v1-5, e.g.:

from diffusers import StableDiffusionControlNetPipeline, ControlNetModel
import torch

controlnet = ControlNetModel.from_pretrained("lllyasviel/sd-controlnet-canny", torch_dtype=torch.float16)
pipe = StableDiffusionControlNetPipeline.from_pretrained(
    "runwayml/stable-diffusion-v1-5", controlnet=controlnet, torch_dtype=torch.float16
)

# load new controlnet
pipe.controlnet = ControlNetModel.from_pretrained("lllyasviel/sd-controlnet-normal", torch_dtype=torch.float16)

Also, we'll make sure to link to your Twitter - ok to use this handle: https://twitter.com/takuma104 ?

Great job again on getting this done!

[takuma104]

@patrickvonplaten

I've changed the official checkpoints to the ones that reside in the ControlNet's authors repo: https://huggingface.co/lllyasviel instead of using the ones of your namespace - I hope this is ok!

Yes, that was my intention from the beginning. I'm glad it turned out as hoped. I think there is still a link to my Colab in the documentation (although it can be removed if necessary), so once that is rewritten, I plan to make my takuma104/control_sd_xxx private.

I think it's better to just have the controlnet weights in the individual models to make it trivial to switch out different controlnets with SD-v1-5, e.g.:

I agree. I think it's a very good method. It has become easier to understand than the previous method, and there is no unnecessary loading.

Also, we'll make sure to link to your Twitter - ok to use this handle: https://twitter.com/takuma104 ?

I'm no longer using that Twitter account, so if possible, I would like to link to my GitHub account instead. https://github.com/takuma104

[takuma104]

To the Diffusers team, thank you! I believe that Diffusers is one of the most high-quality libraries among OSS. I realized that your attention to detail is what supports it. It was a very inspiring experience. I would be happy to help with anything in the future.

[patrickvonplaten]

We should not move to CUDA when enabling CPU offload cc @williamberman

[patrickvonplaten]

1e-4 won't pass on CI

[takuma104]

I have updated the Colab notebook for the merged version in main. It also serves as a simple test for the functionality, and everything works fine. It is based on the sample code at https://huggingface.co/lllyasviel/sd-controlnet-canny

@patrickvonplaten While I would like to keep Colab for the convenience of users, I think it's better for me to let it go. Could you please copy the Colab file and use it as a separate Colab notebook?
I had trouble installing human_pose with pip install (I think it's still a work in progress), so I have included controlnet_hinter, but it's okay to replace it with human_pose or others.

[patrickvonplaten]

I've actually changed the libraries name and made it a pypi package:
https://github.com/patrickvonplaten/controlnet_aux

[yaqing01]

hi,

I have some troubles while doing backpropagation under controlnet (torch==1.13.1):

RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.HalfTensor [24, 1280, 8, 8]], which is output 0 of AddBackward0, is at version 1; expected version 0 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!

it's solved when I changed all the inplace operations in controlnet.py and unet_2d_condition.py from x += y to x = x + y. I didn't know whether it is a bug or not since I am new to pytorch.

[Suhail]

Does anyone know how to do the control net mode conversion correctly or can point me to where the right directory to dl the necessary files?

I tried doing:

python scripts/convert_original_stable_diffusion_to_diffusers.py --controlnet --checkpoint_path ../models/control_sd15_canny.pth --dump_path ../models/control_sd15_canny --device cpu --original_config_file ./cldm_v15.yaml

It worked but I only got this:

I am trying to attempt this: https://github.com/haofanwang/ControlNet-for-Diffusers#controlnet--inpainting but diffusers is already out of date with that README

[yiyixuxu]

@Suhail I think if you pass --controlnet it will only save the controlnet element

cc @williamberman should know more

[Suhail]

Got it. Ok that somewhat solved my problem.

I was able to run this instead (without --controlnet):

python scripts/convert_original_stable_diffusion_to_diffusers.py --checkpoint_path ../models/control_sd15_canny.pth --dump_path ../models/control_sd15_canny --device cpu --original_config_file ./cldm_v15.yaml

Getting this dir:

However I am still met with this error:

Full code here: https://github.com/haofanwang/ControlNet-for-Diffusers/blob/main/pipeline_stable_diffusion_controlnet_inpaint_img2img.py#L850

Ah I see something got rearchitected I guess:

Old code:

New code:

Updating that gets this error:

I am not totally sure what in_channels to set here I guess.

[Suhail]

I got it working! Yay. I had a small bug with latent_mode_input in the wrong location:

[williamberman]

@Suhail https://github.com/haofanwang/ControlNet-for-Diffusers/ looks like it was based off of an older commit of this PR.

It might be easiest to work off the version currently in main and using the already converted controlnet models i.e. https://huggingface.co/lllyasviel/sd-controlnet-canny

I haven't looked deeply at the other repo but if there's a particular feature you want from it please let us know and I can see if we can add the feature directly in diffusers :)

[Suhail]

Hey @williamberman I am looking to do img2img with inpainting which I think diffusers doesn't quite support. Someone at HF encouraged me to look at the https://github.com/haofanwang/ControlNet-for-Diffusers/ repo as a result

[williamberman]

Hey @williamberman I am looking to do img2img with inpainting which I think diffusers doesn't quite support. Someone at HF encouraged me to look at the https://github.com/haofanwang/ControlNet-for-Diffusers/ repo as a result

ah ok, not sure if we'll officially support that but I can throw together a community pipeline for you :)

[williamberman]

@Suhail we have a PR here #2561 for the community pipelines