LoRA + ControlNet?

[andreemic]

Hey!
Do you think it's possible to adapt ControlNet onto a LoRA network?
I think it could be really powerful for real-world tasks where a very specific output is desired. So potentially anyone could take a LoRA model from https://civitai.com and append a ControlNet to it.

Let me know if you have any pointers and I'll take it from there.

Thanks!

[alitan]

What do you mean? You can do that already (if i am understanding you correctly). Just add LoRA into your prompts while using ControlNet

[andreemic]

How would you do that? LoRA models are separate small models that are trained separately from SD models and need to be attached at runtime, so as far as I understand you need to write a bit of connector code.

And I'm not even sure if it makes sense conceptually.

[lllyasviel]

This is why some people in research community are clearly lagging behind industry.
People have been using LoRAs with ControlNets and Multi-ControlNets for months in industry-oriented A1111 and results are usually many times better than what all reseach papers and old-fashined research labs can achieve.