[1.1.454] ControlNet union SDXL model

[huchenlei]

ControlNet Union is a new type of ControlNet that can handle multiple input conditions

• https://huggingface.co/xinsir/controlnet-union-sdxl-1.0
• https://github.com/xinsir6/ControlNetPlus

How to use

Download model

Model can be downloaded here: https://huggingface.co/xinsir/controlnet-union-sdxl-1.0/tree/main. After download, it is recommended to rename the file as controlnet++_union_sdxl so that the control type filter can correctly handle it.

Openpose

Do not use key map with face and hand as the union controlnet does not seem to be trained with hand/face annotation

Canny

Depth

SoftEdge

Normal

Segmentation

Note: ofade20k preprocessor does not seem to work

Multi-ControlNet

Currently the multi-controlnet is not working in the optimal way described in the original paper, but you can still try use it, as it can help you save VRAM by avoid loading another controlnet for different type of control.

Unit1 setup

Unit 2 setup

Generation result

Appendix

Input images

Mask

Generation parameters (Parameters including seed are fixed for all examples)

1girl, sitting on throne,, absurdres, highres, (masterpiece:1.2), (best quality, highest quality),
Negative prompt: (lowres, low quality, worst quality:1.2), (text:1.2), watermark, (frame:1.2), deformed, ugly, deformed eyes, blur, out of focus, blurry, deformed cat, deformed, photo, anthropomorphic cat, monochrome, photo, pet collar, gun, weapon, blue, 3d, drones, drone, buildings in background, green
Steps: 30, Sampler: DPM++ 2M, Schedule type: Karras, CFG scale: 7, Seed: 1413879340, Size: 1024x1024, Model hash: 5d973c1f8a, Model: animagine-xl-2.0, ControlNet 0: "Module: none, Model: controlnet++_union_sdxl [15e6ad5d], Weight: 1.0, Resize Mode: Crop and Resize, Processor Res: 512, Threshold A: 0.5, Threshold B: 0.5, Guidance Start: 0.0, Guidance End: 1.0, Pixel Perfect: False, Control Mode: Balanced", ControlNet 1: "Module: canny, Model: controlnet++_union_sdxl [15e6ad5d], Weight: 1.0, Resize Mode: Crop and Resize, Processor Res: 512, Threshold A: 100.0, Threshold B: 200.0, Guidance Start: 0.0, Guidance End: 1.0, Pixel Perfect: False, Control Mode: Balanced", Version: v1.9.4-249-g93c00b2a

[sdbds]

xinsir6/ControlNetPlus#1
According to my issue, union used mobile sam random split instead of ADE20K

Considering the controlnet_aux repository is now hosted by huggingface, and more new research papers will use the controlnet_aux package, I think we can talk to @Fannovel16 about unifying the preprocessor parts of the three projects to update controlnet_aux.

[Fannovel16]

@sdbds It's quite challenging as comfyui_controlnet_aux has a different DWPose implementation and embeds many third party packages's fork

[sdbds]

Yes, I'm going to start experimenting with preprocessors that don't rely on third-party packages, and I'll just use what's in the repository for the rest of it.
Also controlnet_aux's DWpose implementation seems to be faster and more suitable for video frame conversion, in my previous tests.

[slashedstar]

CAn we have a update for the promax version with tile and inpainting? 👀

[ali0une]

Yes please @huchenlei !

i currently have this error with the promax version :

2024-07-14 17:47:35,187 - ControlNet - INFO - Loading model: controlnet++_union_sdxl-promax-1.0 [9460e4db]
2024-07-14 17:47:35,196 - ControlNet - INFO - Loaded state_dict from [/whatever/stable-diffusion-webui/models/ControlNet/controlnet++_union_sdxl-promax-1.0.safetensors]
2024-07-14 17:47:35,201 - ControlNet - INFO - controlnet_sdxl_config
*** Error running process: /whatever/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/controlnet.py
    Traceback (most recent call last):
      File "/whatever/stable-diffusion-webui/modules/scripts.py", line 825, in process
        script.process(p, *script_args)
      File "/whatever/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/controlnet.py", line 1228, in process
        self.controlnet_hack(p)
      File "/whatever/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/controlnet.py", line 1213, in controlnet_hack
        self.controlnet_main_entry(p)
      File "/whatever/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/controlnet.py", line 919, in controlnet_main_entry
        model_net, control_model_type = Script.load_control_model(p, unet, unit.model)
      File "/whatever/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/controlnet.py", line 436, in load_control_model
        control_model = Script.build_control_model(p, unet, model)
      File "/whatever/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/controlnet.py", line 465, in build_control_model
        control_model = build_model_by_guess(state_dict, unet, model_path)
      File "/whatever/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/controlnet_model_guess.py", line 240, in build_model_by_guess
        network = PlugableControlModel(config, state_dict)
      File "/whatever/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/cldm.py", line 24, in __init__
        self.control_model.load_state_dict(state_dict, strict=False)
      File "/whatever/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 2189, in load_state_dict
        raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
    RuntimeError: Error(s) in loading state_dict for ControlNet:
    	size mismatch for task_embedding: copying a param with shape torch.Size([8, 320]) from checkpoint, the shape in current model is torch.Size([6, 320]).
    	size mismatch for control_add_embedding.linear_1.weight: copying a param with shape torch.Size([1280, 2048]) from checkpoint, the shape in current model is torch.Size([1280, 1536]).

[3blackbar]

You have to fill issue for this

[huchenlei]

Tile supported, but inpaint still has issue. #2998

Need to wait official inference script for inpaint from union controlnet author to see what is wrong for inpaint.

[TKoestlerx]

Edit: Outpaint is working with the preprocessor set to "none".
Inpaint is not working correctly as it seems to draw information from the masked out area.

Edit2: You can do inpainting if you use the "Effective Region Mask" Feature to upload the mask instead of the default inpainting map.

[Gavr728]

I tried to do outpainting with the preprocessor set to none, but surprisingly it also changed colors on the original picture while preserving all the other details in tact.

[3blackbar]

OK while the comit that introduced this new union works fine cause it installs opencv properly - the last one is impossible to install on clean install of newest auto11... so exepect lot of issues if ppl will bother to fill them in before the fix.
Somehow this is not applied in recent ones - installing sd-webui-controlnet requirement: changing opencv-python version from 4.10.0.84 to 4.8.0
This must be applied to all recent installs

Also in only masked inpaint union is not working if cnet is more important or balanced - overbaked result but if i select - prompt is more important, it kinda works ... of course preprocessor must be None which is strange ...
Also... theres one culprit in some sdxl cnet models this one included - resolutions divisible by 64.... if you pick res that is not divisible by 64 - you get crap or nans, black image whatever, so... maybe cnet should under the hood round the values so they are always divisble by 64 ,cause this is whats causing nans - fancy resolutions .I get nans black screen with this if i dont have preprocessor set to none

Great news is that this union inpaint works with pony

With more tests i got to conclusion that it is not working with its full potential yet even with the settings im using, cant inpaint big solid chunks into something else, for example i cant inpaint this into mountains panorama , all i get is split pics

[TKoestlerx]

I have had success with replacing big parts with inpainting, but only if i remove the area i want to replace in the src img too.
So it becomes sort of inward outpainting ;)