Feature Request: Loading PeFT - LoRA adapters during runtime without prior merging

[niranjanakella]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

Hello @ggerganov, this is a feature request to support LoRA adapters loading during runtime. In the current flow, we need to merge the adapter weights to the base model weights and convert them into GGUF to infer the model. But what if there is a way to independently convert the base model and the adapters to GGUF so that during runtime, the desired adapter could be mounted over the base model.

Motivation

This can hugely benefit memory-efficient computing not just for low-end developers but even for commercial startups who wish to build quick serverless applications. Many developers today train multiple adapters for a wide range of applications utilizing the same base model.

Possible Implementation

I am not a CPP guy but with the basic understanding that I have, I think there should be a way to independently convert the LoRA adapters to GGUF format considering the rank, alpha and other parameters used and which architecture it was used upon to train initially. Then during runtime, all we need to do is to mount these adapter parameters over the base model at specific layers.

[DewEfresh]

I was just looking for this. All i found was how to train a lora in llama.cpp https://github.com/ggerganov/llama.cpp/tree/master/examples/finetune . I just saw a demo of Lorax https://github.com/predibase/lorax ,which lets use multiple lora on the fly and to turn them off independently. In llama.ccp documention it says you can adjust the scaling of the lora which is nice. It would be great if you could adjust them when sending the prompt. Wishful thinking, but to be able to convert loras already out there would be great.