How should I create a ControlNet without a model using diffusers and specify its preprocessor as reference_adain+attn

[vvlookman]

Is your feature request related to a problem? Please describe.
I need to implement the same function in diffusers as ControlNet in SD WebUI with reference_adain+attn preprocessor specified, but I can't find the available method.

Describe the solution you'd like
I can create a ControlNet without a model and I can specify its preprocessor as reference_adain+attn.

[kadirnar]

https://github.com/huggingface/diffusers/tree/main/examples/community#stable-diffusion-controlnet-reference

[vvlookman]

https://github.com/huggingface/diffusers/tree/main/examples/community#stable-diffusion-controlnet-reference

Thank you for your reply!

I tried the new class StableDiffusionControlNetReferencePipeline, but got an error:

TypeError: Transformer2DModel.forward() got an unexpected keyword argument 'attention_mask'

Since the StableDiffusionControlNetReferencePipeline that the example depends on is in the main branch and not yet officially released, I used diffusers from v0.16.1 plus this class, not sure if there are any version inconsistencies.

[vvlookman]

https://github.com/huggingface/diffusers/tree/main/examples/community#stable-diffusion-controlnet-reference

And there is another problem, the pipeline also needs to specify that the controlnet parameter, but the reference preproccessor is model free and I do not want to use a pretrained controlnet model.

I see that there is another StableDiffusionReferencePipeline class, is this consistent with the controlnet in the SD WebUI that does not specify a model?

[sayakpaul]

I don't understand your questions fully.

You can initialize a ControlNetModel like so:

from diffusers import ControlNetModel


controlnet = ControlNetModel.from_pretrained("lllyasviel/sd-controlnet-canny")

If you don't want to initialize from a pre-trained checkpoint, that is also possible. You can check out the tests from here:

diffusers/tests/pipelines/controlnet/test_controlnet.py

Line 117 in 4f14b36

controlnet = ControlNetModel(

But if the model needs modification, then you will have to start with https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/controlnet.py

[kadirnar]

https://github.com/huggingface/diffusers/tree/main/examples/community#stable-diffusion-controlnet-reference

And there is another problem, the pipeline also needs to specify that the controlnet parameter, but the reference preproccessor is model free and I do not want to use a pretrained controlnet model.

I see that there is another StableDiffusionReferencePipeline class, is this consistent with the controlnet in the SD WebUI that does not specify a model?

You have to use controlnet.
Mikubill/sd-webui-controlnet#1236

[vvlookman]

https://github.com/huggingface/diffusers/tree/main/examples/community#stable-diffusion-controlnet-reference

And there is another problem, the pipeline also needs to specify that the controlnet parameter, but the reference preproccessor is model free and I do not want to use a pretrained controlnet model.
I see that there is another StableDiffusionReferencePipeline class, is this consistent with the controlnet in the SD WebUI that does not specify a model?

You have to use controlnet. Mikubill/sd-webui-controlnet#1236

So how should I create this ControlNet? A pre-trained model does not need to be specified in the ControlNet tab in the SD WebUI:

[kadirnar]

Controlnet + Canny Preprocces + Reference_attn + Reference_adain:

import cv2
import torch
import numpy as np
from PIL import Image
from diffusers import UniPCMultistepScheduler
from diffusers.utils import load_image

input_image = load_image("https://hf.co/datasets/huggingface/documentation-images/resolve/main/diffusers/input_image_vermeer.png")

# get canny image
image = cv2.Canny(np.array(input_image), 100, 200)
image = image[:, :, None]
image = np.concatenate([image, image, image], axis=2)
canny_image = Image.fromarray(image)

controlnet = ControlNetModel.from_pretrained("lllyasviel/control_v11p_sd15_canny", torch_dtype=torch.float16)
pipe = StableDiffusionControlNetReferencePipeline.from_pretrained(
       "runwayml/stable-diffusion-v1-5",
       controlnet=controlnet,
       safety_checker=None,
       torch_dtype=torch.float16
       ).to('cuda:0')

pipe.scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config)

result_img = pipe(ref_image=input_image,
      prompt="1girl",
      image=canny_image,
      num_inference_steps=20,
      reference_attn=True,
      reference_adain=True).images[0]

Controlnet Other Modules:
https://huggingface.co/lllyasviel

[vvlookman]

So what should I do if I want to remove the "Canny model" in this sample?

When I use SD WebUI, no pre-trained model need to be specified in the ControlNet tab if choose reference preprocessor

[sayakpaul]

You can pass a ControlNetModel of your choosing as long as it's compatible. You will have to also ensure that you're using the right preprocessor to preprocess the conditioning image.

Please refer to the docs for more details:
https://huggingface.co/docs/diffusers/main/en/api/pipelines/stable_diffusion/controlnet

[vvlookman]

I think I may still not have described the problem clearly.

My question is, when I select the reference_adain+attn preprocessor in the ControNet tab in the SD WebUI, how should I create this StableDiffusionControlNetReferencePipeline corresponding to diffusers? Obviously there is no pre-trained model for the ControlNet specified here.

If I need to build the ControlNetModel manually, how should I set the parameters, I want to be able to match the effect set in the SD WebUI.

[sayakpaul]

Cc @williamberman for the above.

[bharathithal]

Can anybody tell how to resolve the issue @vvlookman rasied about
TypeError: Transformer2DModel.forward() got an unexpected keyword argument 'attention_mask' ? Any version incompatibility or something? I'm trying to run this on a google colab notebook! I am using the same example shown in the community

[vvlookman]

Can anybody tell how to resolve the issue @vvlookman rasied about TypeError: Transformer2DModel.forward() got an unexpected keyword argument 'attention_mask' ? Any version incompatibility or something? I'm trying to run this on a google colab notebook! I am using the same example shown in the community

Installing diffusers directly from the master branch can work around this problem

[williamberman]

Sorry I'm not familiar with reference_adain+attn preprocessor is this webui using diffusers or the original controlnet implementationg?