[Feature] Support for SD3 ControlNet

[Fannovel16]

Chads from InstantX (who created InstantID) has made several ControlNet for SD3-Medium, including:

• InstantX/SD3-Controlnet-Canny
• InstantX/SD3-Controlnet-Pose
• InstantX/SD3-Controlnet-Tile
• InstantX/SD3-Controlnet-Inpainting

Their implementation can be found at huggingface/diffusers#8566

[kadirnar]

+1

[sanjay-uzu]

But these models are not working in comfyui

[Fannovel16]

@sanjay-uzu That is why we need an issue

[zsxkib]

"Chads from InstantX" lol @Fannovel16

[geroldmeisinger]

controlnet support merged into diffusers
huggingface/diffusers#8566 (comment)

[geroldmeisinger]

state of the model seems subpar atm https://huggingface.co/InstantX/SD3-Controlnet-Canny/discussions/4
but I guess a 1.0 version will follow soon
anyone had better results yet?

[kijai]

state of the model seems subpar atm https://huggingface.co/InstantX/SD3-Controlnet-Canny/discussions/4 but I guess a 1.0 version will follow soon anyone had better results yet?

Here's some of my tests, post and tile work pretty good. This is using diffusers with a wrapper node I made to test this, I added some of the standard CN features that was missing (start/end, guess_mode). Canny seems to work better with more defined input, I found using lineart preproc better. I do definitely think it's worth implementing!

[al-swaiti]

its here but in diffusers mode
https://github.com/ZHO-ZHO-ZHO/ComfyUI-SD3-Medium-CN-Diffusers
waiting for normal one

[mcmonkey4eva]

Added to Comfy yesterday 264caca use ControlNetApplySD3 node

[geroldmeisinger]

@kijai can you please try it again with something non-human and non-architectural, like an animal. the controlnet seems to have an effect and working but i'm not getting any good results with the dog2.png test image of the original controlnet :/

dog2 square-cropped and upscaled to 1024x1024:

I trained canny controlnets on my own and this result looks to me as if a) the CN didn't yet fully converge or b) the model collapsed at some point.