ControlNet support

[kabachuha]

ControlNet is an especially powerful new feature which can be used to guide image generation.

It is going to be extremely useful for Deforum animation creation, so it's top priority to integrate it into Deforum.

Since it's applicable both to txt2img and img2img, it can be feed similarly to video masking. Fortunately, we already have the composable mask mechanism.

Meanwhile, another lucky thing is that someone's doing an API layer on top of the CN extension. If we base Deforum support on top of the functions there, it will greatly speedup the transition process.

Mikubill/sd-webui-controlnet#194

[kabachuha]

I added all you because you have expertise in video parts insertion. I'm going to push a draft version soon and then we need to make it like the second 'hybrid mode' or something like that

[kabachuha]

The PR is live! (although still a bit raw)

[screan]

after updating, deforum tab is missing and console throwing this error:

Error executing callback ui_tabs_callback for E:\stable-diffusion-webui\extensions\deforum-for-automatic1111-webui\scripts\deforum.py
Traceback (most recent call last):
  File "E:\stable-diffusion-webui\modules\script_callbacks.py", line 119, in ui_tabs_callback
    res += c.callback() or []
  File "E:\stable-diffusion-webui\extensions\deforum-for-automatic1111-webui\scripts\deforum.py", line 198, in on_ui_tabs
    components = deforum_args.setup_deforum_setting_dictionary(None, True, True)
  File "E:\stable-diffusion-webui/extensions/deforum-for-automatic1111-webui/scripts\deforum_helpers\args.py", line 683, in setup_deforum_setting_dictionary
    controlnet_dict = setup_controlnet_ui()
  File "E:\stable-diffusion-webui/extensions/deforum-for-automatic1111-webui/scripts\deforum_helpers\deforum_controlnet.py", line 171, in setup_controlnet_ui
    return setup_controlnet_ui_raw()
  File "E:\stable-diffusion-webui/extensions/deforum-for-automatic1111-webui/scripts\deforum_helpers\deforum_controlnet.py", line 89, in setup_controlnet_ui_raw
    from scripts.processor import canny, midas, midas_normal, leres, hed, mlsd, openpose, pidinet, simple_scribble, fake_scribble, uniformer
ImportError: cannot import name 'leres' from 'scripts.processor' (E:\stable-diffusion-webui/extensions/sd-webui-controlnet\scripts\processor.py)

[kabachuha]

@screan disabled leres in 147b2cb. Try launching now

[screan]

Error executing callback ui_tabs_callback for E:\stable-diffusion-webui\extensions\deforum-for-automatic1111-webui\scripts\deforum.py
Traceback (most recent call last):
  File "E:\stable-diffusion-webui\modules\script_callbacks.py", line 119, in ui_tabs_callback
    res += c.callback() or []
  File "E:\stable-diffusion-webui\extensions\deforum-for-automatic1111-webui\scripts\deforum.py", line 198, in on_ui_tabs
    components = deforum_args.setup_deforum_setting_dictionary(None, True, True)
  File "E:\stable-diffusion-webui/extensions/deforum-for-automatic1111-webui/scripts\deforum_helpers\args.py", line 686, in setup_deforum_setting_dictionary
    controlnet_dict = setup_controlnet_ui()
  File "E:\stable-diffusion-webui/extensions/deforum-for-automatic1111-webui/scripts\deforum_helpers\deforum_controlnet.py", line 171, in setup_controlnet_ui
    return setup_controlnet_ui_raw()
  File "E:\stable-diffusion-webui/extensions/deforum-for-automatic1111-webui/scripts\deforum_helpers\deforum_controlnet.py", line 89, in setup_controlnet_ui_raw
    from scripts.processor import canny, midas, midas_normal, hed, mlsd, openpose, pidinet, simple_scribble, fake_scribble, uniformer
ImportError: cannot import name 'pidinet' from 'scripts.processor' (E:\stable-diffusion-webui/extensions/sd-webui-controlnet\scripts\processor.py)

[kabachuha]

Upgrade or Downgrade your ControlNet version to a24089a62e70a7fae44b7bf35b51fd584dd55e25 — it's the one used in the integration development

[screan]

thanks @kabachuha !

[kabachuha]

Glad it helped!

I'll look into other versions support too, but now I need to get some sleep.

@screan, send your results when you get them. I'm eager to see how it works for you 🙂

[kabachuha]

The recommended version is https://github.com/Mikubill/sd-webui-controlnet/archive/c9340671d6d59e5a79fc404f78f747f969f87374.zip now