multi-LoRA as extra models in OpenAI server

[jvmncs]

closes #2600

how to serve the loras (mimicking the multilora inference example):

$ export LORA_PATH=~/.cache/huggingface/hub/models--yard1--llama-2-7b-sql-lora-test/
$ python -m vllm.entrypoints.api_server \
 --model meta-llama/Llama-2-7b-hf \
 --enable-lora \
 --lora-modules sql-lora=$LORA_PATH sql-lora2=$LORA_PATH

the above server will list 3 separate values if the user queries /models: one for the base served model, and one each for the specified lora modules. in this case sql-lora and sql-lora2 point to the same underlying lora, but this need not be the case. lora config values take the same values they do in EngineArgs

no work has been done here to scope client permissions to specific models

[jvmncs]

was planning to add a test in tests/entrypoints/test_openai_server.py, but it's unclear which loras are usable with the existing zephyr model from that file. happy to change that out with some different model/lora defaults from huggingface, if that's desirable

[simon-mo]

@jvmncs, thanks for the PR. I think you can change the base model to other that works. Any suggestions? If not we can also create a new test file just to test LoRA support.

[jvmncs]

this one looks good: https://huggingface.co/typeof/zephyr-7b-beta-lora

[jvmncs]

for some reason this test was failing for all cases when I switched to MODEL_NAME="mistralai/Mistral-7B-v0.1". model was consistently emitting "1999" no matter the prompt/temperature I tried. not really sure why that's the case but reverting to the zephyr model fixed it

[jvmncs]

latest commit should be ready for a review, assuming CI passes as it did on my machine

[jvmncs]

@simon-mo @Yard1 bumping this for review

[Yard1]

This looks pretty good to me, could we add an example of starting an API server with loras?

[jvmncs]

could we add an example of starting an API server with loras

sure @Yard1, what kind of example were you thinking? other than the command snippet in my original comment, I'm not sure what that would look like

[Yard1]

Hmm, I guess we don't really have an example where we start the server - in that case just extending the docs (https://github.com/vllm-project/vllm/blob/main/docs/source/models/lora.rst) with a snippet of how to run the OpenAI server with lora should be enough!

[jvmncs]

@Yard1 should be good to go

[Yard1]

Thanks, looks good!

[Wizmak9]

@Yard1 can we do hot swap of lora like without restarting the base model again and changing lora adapters to that base model on the fly.

[ajaychinni]

@Wizmak9 You must initiate your VLLM only using the OpenAI server, following the guidelines provided in the doumnetation

python -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Llama-2-7b-hf \
    --enable-lora \
    --lora-modules sql-lora=~/.cache/huggingface/hub/models--yard1--llama-2-7b-sql-lora-test/ sql-lora2=~/.cache/huggingface/hub/models--yard1--llama-2-7b-sql-lora-test2/

In the above command just for example loaded two lora modules sql-lora and sql-lora2

Now, when making a request, you can specify which LoRA you wish to use. In subsequent requests, you can select different LoRA adapters on the fly, without needing to restart the model.

Python code to simulate the request using sql-lora. You can comment out this model and use the base model or another LoRA model in subsequent requests:

from openai import OpenAI
openai_api_key = "EMPTY"
openai_api_base = "http://localhost:8000/v1"

client = OpenAI(
    api_key=openai_api_key,
    base_url=openai_api_base,
)

chat_response = client.chat.completions.create(
    # model =  "meta-llama/Llama-2-7b-hf ",
    model = "sql-lora" ,
    # model = "sql-lora2",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Write a SQL query to answer the question based on the table schema.\n\n context: CREATE TABLE table_name_74 (icao VARCHAR, airport VARCHAR)\n\n question: Name the ICAO for lilongwe international airport "},
    ]
)
print("Chat response:", chat_response)