Added additional parameters to mixing multiple LoRAs through SVD, added ability to mix LoRAs through concatenation

[kovalexal]

Hi!

This PR introduces the following changes:

1. Fixes an error during SVD decomposition when Conv2d layers are present in LoRAs (caused by the lack of fan_in_fan_out in Conv2d LoRALayer)
   

   peft/src/peft/tuners/lora.py

   Lines 659 to 661 in 412d7bc

   	if target.fan_in_fan_out:
   	delta_weight = delta_weight.T

2. Adds an ability to combine multiple adapters with concatenation of their A and B matrices. It's super fast compared to SVD decomposition (e.g. on my GPU SVD takes about 30+secs, and concatenation takes less than 1sec), and is also mathematically equivalent to successive applying of multple LoRAs. When using this method, one should be aware that mixing LoRAs may result in target LoRA being too large, so in this case SVD decomposition should be preferred.
3. Adds an ability to provide the desired target SVD rank when performing SVD decomposition. So, one can find a balance between inference speed and loss in precision.
4. Adds an ability to provide additional parameters to torch.linalg.svd like driver to use during decomposition.
5. Adds an ability to disable clamping matrices values after SVD decomposition. From my experiments I see that clamping this values for Stable Diffusion LoRAs also influences results compared to Automatic1111 webui.

You can see my previous results here #643.

webui output:

SVD decomposed mixture of 2 LoRAs, clamped:

SVD decomposed mixture of 2 LoRAs, unclamped:

@pacman100 your review is kindly appreciated.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

[BenjaminBossan]

Thanks a lot for the fixes and the new method, this looks pretty solid.

I'm not an expert on this topic (@pacman100 will know better), but the implementation looks quite good. However, I would like to see the unit tests extended to cover the new combination type.

[pacman100]

Thank you so much @kovalexal for adding the concat method from our previous discussions and improving upon the svd method, you rock 🔥. LGTM! 🤗

[BenjaminBossan]

Looks good, thanks a lot for addressing my comments.