LoRA for Conv2d layer, script to convert kohya_ss LoRA to PEFT

[kovalexal]

Hi!

Thank you for your awesome library, it pushes the limits of tuning large-scale models on consumer-like hardware even further 🤟

Some researches have shown, that allowing Conv2d layers to be also trained in LoRA helps to get even better results for SD models.

I've made a small modification, which allows to train Conv2d layers, and also made a script to convert basic checkpoints from kohya_ss to be used in PEFT.

It would be beneficial if it will get eventually merged and will allow anyone to use some of pretrained LoRAs in diffusers framework.

[ElleLeonne]

Wow, I was literally just looking for something like this. Nice! Hopefully it gets pulled soon

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[kovalexal]

Hi!
I've fixed some code style issues, would be great if somebody could run the workflow again.

[kovalexal]

Hey, @pacman100, would highly appreciate it if you could have a look at this PR. Thanks in advance!

[pacman100]

Thank you @kovalexal for adding support for Conv2D layers with LoRA method and converting the Kohya ckpts to PEFT format, LGTM! 🤗🚀🔥

[zyshin]

Hi @kovalexal ,

Thanks for your nice code to make LoRA work better!

For those who want to use PEFT LoRAs in Automatic1111 WebUI, do you have any suggestion on how to convert LoRA in PEFT format back to the Kohya ckpts or safetensors?

I know the question is not quite related to this PR (and there is #212) but I can't find any answer. Thank you very much!

[kovalexal]

Hi @zyshin!

Thanks for your feedback, I definitely agree that such a script would be really helpful for debugging purposes and backward compatibility with webui, so I have created PR for this #596.