Is it possible to dynamically switch multiple LoRA adapters?

[nai-kon]

In the transformers library, we can load multiple adapters to the original model by load_adapter then switch the specified adapter with set_adapter like below.

# base model
model = AutoModelForCausalLM.from_pretrained(
    model_name,
)

# load multiple adapters
model.load_adapter("model/adapter1/", "adapter1")
model.load_adapter("model/adapter2/", "adapter2")

# switch adapter
model.set_adapter("adapter2")

Is it possible to do the same thing with llama cpp python?
I know constructor of Llama() has lora_base and lora_path parameters so one adapter can be applied when loading the model.
But my question is switch multiple adapters after model loading.

[abetlen]

@nai-kon not supported in the high-level API as I believe it just keeps applying the LoRA each time the low level function calls are made. To "switch" you'll need to re-load the full model.

[Theodotus1243]

Cool feature, VLLM has it, but for some reason not working for me