Integarting RandLoRA a full rank PEFT algorithm

[PaulAlbert31]

Feature request

Include RandLoRA as an option for PEFT
Code: https://github.com/PaulAlbert31/RandLoRA/tree/main/peft/src/peft/tuners/randlora
Paper: https://openreview.net/pdf?id=Hn5eoTunHN

Motivation

RandLoRA is a recently published paper https://openreview.net/forum?id=Hn5eoTunHN that challenges the limitations of LoRA's low-rank setting. RandLoRA combines random bases to create a full rank parameter-efficient updates. An custom gradient function is proposed to optimize memory use.

In summary:
RandLoRA can outperform LoRA for complex tasks (e.g. vision-language or LLM finetuning)
RandLoRA reduces memory usage over LoRA thanks to custom gradient functions that share non-trainable random bases across layers
RandLoRA's parameter count is dictated by the rank of the random basis.
RandLoRA can take slightly longer to train when scaling up the amount of trainable parameters.

To illustrate RandLoRA's capabilities, here are key results from the paper:

CLIP results (vision-language classification) with GPU VRAM usage during training:

Loss landscape connectivity of LoRA vs RandLoRA vs Std. Finetuning for CLIP. RandLoRA achieves a deeper minima than LoRA for equal trainable parameters.

Your contribution

I have implemented RandLoRA as part of a local copy of the peft package https://github.com/PaulAlbert31/RandLoRA
More specifically: https://github.com/PaulAlbert31/RandLoRA/tree/main/peft/src/peft/tuners/randlora

I am happy to help or refactor part of the existing code to fit the current PEFT structure.

[githubnemo]

Hey, thanks for the suggestion, at the first glance this looks promising.

We're currently in the process of deciding how to implement such variations, starting with SineLoRA. We'll get back to you, hopefully quite soon, on how to implement this and/or with follow-up questions.

[BenjaminBossan]

We took a closer look at the paper and the existing implementation. This looks like a nice extension of PEFT and we'd be happy to add to the library. Please feel free to create a (draft) PR, your existing code should already tick many necessary boxes, we can iterate from there.

Btw. this paper also reminded me a little bit of VB-LoRA, although overall it's looks closer to VeRA. Still, it could be something worth citing.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

[BenjaminBossan]

not stale

[PaulAlbert31]

Hi @BenjaminBossan,
Thanks a lot for your time, help and feedback for merging the RandLora code into main, glad it is now done.

I have initiated a new draft request to add documentation and example usages, see #2524 as per your recommendation.

I'd also be happy to add RandLora to a potential PEFT leaderboard if you have one. Let me know if this is possible.

Thanks !

[githubnemo]

I think this is complete. :)

Thanks again @PaulAlbert31 for your time and excellent work! It is much appreciated.