[Usage]: different models in the /vllm/model directory #15660
Description
Your current environment
If I’ve downloaded different models in the /vllm/model directory, how can I set up vLLM to only load the models when a user selects one in another LLM tool, instead of loading them when the Docker container starts?
How would you like to use vllm
I want to run inference of a [specific model](put link here). I don't know how to integrate it with vllm.
Before submitting a new issue...
- Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.