ControlNet-XS support

[scarbain]

Model/Pipeline/Scheduler description

Hello all,

Are you planning on supporting ControlNet-XS ? It's promising a better architecture with a much lower number of parameters for the same quality of results.

Thanks !

Open source status

• The model implementation is available
• The model weights are available (Only relevant if addition is not a scheduler).

Provide useful links for the implementation

Code repository : https://github.com/vislearn/ControlNet-XS

Project page : https://vislearn.github.io/ControlNet-XS/

Weights : https://huggingface.co/CVL-Heidelberg/ControlNet-XS (available for Canny and Depth for both SDXL and SD2.1)

[scarbain]

I forgot to add the handle of the original author : @Sipirius

[patrickvonplaten]

@williamberman @patil-suraj @yiyixuxu anybody interested in porting the controlnets here?

[UmerHA]

@patrickvonplaten If okay for you, I'd be happy to do it!

[patrickvonplaten]

Yeah that would be great !

[ayushtues]

Hey @UmerHA let me know if you want any help on this, happy to collaborate on this!

[UmerHA]

@ayushtues sure! I think I'll be done with the main implementation & first pipeline by ~mid next week. Then we can qa & split up the remaining pipelines. How should I contact you?

[UmerHA]

To give a quick update: I'm working on it (see https://github.com/UmerHA/diffusers/tree/controlnet-xs) and estimate it will take 1-2 more weeks. But am learning a lot! :)

[UmerHA]

Another quick update: I'm making steady progress, but can't estimate well when I'll be done.

I have:

• Understood the paper
• Converted the paper artifacts into diffusers artifacts
• Translated the code into diffusers architecture and made it run

I am now:

• Debugging why the diffusers version doesn't produce the same output as the paper code. But I'm getting closer and closer :)

I still need to:

• Do that for the other pipelines (img2img, sd [not sdxl], inpainting)
• Clean up (test, documentation)

Still learning a lot! And thanks for the opportunity to let me do this!

[UmerHA]

I'm getting close to being done :)

See here for a working example

@patrickvonplaten I have opened a draft PR (#5827) and would love feedback!

The SDXL canny versions work.

Still to do:

• Add other versions of ControlNet-XS: SD canny, SD depth, SDXL depth
• Add documentation
• A few other (iiuc) minor things

I think it should be done be mid/end next week.

[UmerHA]

PR is now marked ready for review!