Controlnet support?

[fyepi]

Any plan for it?
This project is great but without controlnet, many advanced users might be disappointed.

[afeezaziz]

@saketh12 @superadi04 please enable your account to receive monthly support. I will donate on monthly basis so that this can be done/run.

[superadi04]

Hi @fyepi, this is the next step we are planning to add. @afeezaziz Thank you for the donations, we will set it up and get back to you

[saketh12]

Hi @afeezaziz I just opened up my Github sponsors account. We appreciate any donations! As requested, we are working on making ControlNet now and expanding the project.

[superadi04]

Hey, i just added controlnet. Right now it only works with fp32, looking to add fp16 support soon. Also, I would really appreciate any contributors for this issue.

[AlexMKramer]

Just a note; the instructions for Controlnet are a little confusing and possibly incorrect in some places. This section in the Readme says to use "ControlNetModel(model=" for the control net model, but I am seeing "model_path" as the expected variable.

from auto1111sdk import StableDiffusionPipeline
from auto1111sdk import ControlNetModel

model = ControlNetModel(model="<THE CONTROLNET MODEL FILE NAME (WITHOUT EXTENSION)>", 
                   image="<PATH TO IMAGE>")

pipe = StableDiffusionPipeline("<Path to your local safetensors or checkpoint file>", controlnet=model)

prompt = "a picture of a brown dog"
output = pipe.generate_txt2img(prompt = prompt, height = 1024, width = 768, steps = 10)

output[0].save("image.png")

The Gitbook also is inconsistent showing the instruction above, but shows "ControlNet(model=" in the example.

Not sure what the proper way to set the model is either way, as setting it with "model=" says unexpected argument, "model_path" doesn't seem to set the model properly, and "ControlNet("model=" is unrecognized.

[superadi04]

hey could you send me the code you are running exaclty

[AlexMKramer]

Here is an example of the code I am running:

import os
import time
from auto1111sdk import StableDiffusionPipeline, ControlNetModel


def controlnet(prompt="a man eating an ice cream cone",
               negative_prompt="",
               model_path="models/checkpoints/epicPhotogasm.safetensors",
               num_images=4,
               height=512,
               width=512,
               steps=10,
               cfg_scale=7,
               sampler_name="DPM++ 2M Karras",
               clip_skip=2,
               controlnet_name="controlnet_depth"
               ):
    print(model_path)
    # If a controlnet is specified, run the image through the controlnet first
    controlnet_path = os.path.join(os.getcwd(), "models/controlnets/" + controlnet_name)
    print(controlnet_path)
    controlnet_model = ControlNetModel(controlnet_path, image="attached_image.png")
    pipe = StableDiffusionPipeline(model_path, controlnet=controlnet_model, clip_skip=clip_skip)
    print(f"Using {controlnet_name} to process the image")
    output = pipe.generate_txt2img(prompt=prompt,
                                   negative_prompt=negative_prompt,
                                   num_images=num_images,
                                   height=height,
                                   width=width,
                                   steps=steps,
                                   cfg_scale=cfg_scale,
                                   sampler_name=sampler_name,
                                   seed=-1
                                   )
    file_paths = []
    for i in range(num_images):
        # Save the output to a file in the output folder using the time and date as the file name
        file_name = f"ImageGenBot-{time.strftime('%Y-%m-%d-%H-%M-%S')}-{i}.png"
        save_path = os.path.join(os.getcwd(), "output/")
        output[i].save(f"{save_path}{file_name}")
        print(f"Saved {file_name}")
        file_paths.append(f"{save_path}{file_name}")

    pipe.set_vae(None)
    del pipe
    print("Finished generating controlnet images.")


controlnet()
print("Done!")

And here is the output:

(base) alexk@AlexHomeServer:~/Desktop/ImageGenBot$ venv/bin/python test.py 
models/checkpoints/epicPhotogasm.safetensors
/home/alexk/Desktop/ImageGenBot/models/controlnets/controlnet_depth
None
2024-03-20 07:56:59,104 - ControlNet - INFO - ControlNet v1.1.440
ControlNet preprocessor location: /home/alexk/Desktop/ImageGenBot/venv/lib/python3.11/site-packages/auto1111sdk/extensions/controlnet/annotator/downloads
2024-03-20 07:57:00,435 - ControlNet - INFO - ControlNet v1.1.440
./
None
Applying attention optimization: Doggettx... done.
Using controlnet_depth to process the image
2024-03-20 07:57:02,898 - ControlNet - INFO - unit_separate = False, style_align = False
*** Error running process: /home/alexk/Desktop/ImageGenBot/venv/lib/python3.11/site-packages/auto1111sdk/extensions/controlnet/scripts/controlnet.py
    Traceback (most recent call last):
      File "/home/alexk/Desktop/ImageGenBot/venv/lib/python3.11/site-packages/auto1111sdk/modules/scripts.py", line 715, in process
        script.process(p, *script_args)
      File "/home/alexk/Desktop/ImageGenBot/venv/lib/python3.11/site-packages/auto1111sdk/extensions/controlnet/scripts/controlnet.py", line 1191, in process
        self.controlnet_hack(p)
      File "/home/alexk/Desktop/ImageGenBot/venv/lib/python3.11/site-packages/auto1111sdk/extensions/controlnet/scripts/controlnet.py", line 1176, in controlnet_hack
        self.controlnet_main_entry(p)
      File "/home/alexk/Desktop/ImageGenBot/venv/lib/python3.11/site-packages/auto1111sdk/extensions/controlnet/scripts/controlnet.py", line 941, in controlnet_main_entry
        model_net, control_model_type = Script.load_control_model(p, unet, unit.model)
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/alexk/Desktop/ImageGenBot/venv/lib/python3.11/site-packages/auto1111sdk/extensions/controlnet/scripts/controlnet.py", line 408, in load_control_model
        control_model = Script.build_control_model(p, unet, model)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/alexk/Desktop/ImageGenBot/venv/lib/python3.11/site-packages/auto1111sdk/extensions/controlnet/scripts/controlnet.py", line 427, in build_control_model
        raise RuntimeError(f"model not found: {model}")
    RuntimeError: model not found: None

---
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:01<00:00,  6.52it/s]
Total progress: 10it [00:01,  8.28it/s]
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:01<00:00,  8.19it/s]
Total progress: 10it [00:01,  8.46it/s]
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:01<00:00,  8.16it/s]
Total progress: 10it [00:01,  8.45it/s]
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:01<00:00,  8.22it/s]
Total progress: 10it [00:01,  8.44it/s]
Saved ImageGenBot-2024-03-20-07-57-08-0.png
Saved ImageGenBot-2024-03-20-07-57-08-1.png
Saved ImageGenBot-2024-03-20-07-57-08-2.png
Saved ImageGenBot-2024-03-20-07-57-09-3.png
Finished generating controlnet images.
Done!

If it is something I am doing incorrectly, let me know!