Sharing my vLLM Docker Container Image

[samos123]

Repo: https://github.com/substratusai/vllm-docker

This container image runs the OpenAI API server of vLLM.

Image URL: ghcr.io/substratusai/vllm

Quickstart

Deploy Mistral 7B Instruct:

docker run -d -p 8080:8080 --gpus=all \
  -e MODEL=mistralai/Mistral-7B-Instruct-v0.1 \
  ghcr.io/substratusai/vllm

Configuration Options

The following configuration options are available by using environment
variables:

Env Name	Description
MODEL	REQUIRED, The model ID to serve. This can be in the form of hf_org/model or utilize a path to point to a local model. Example value: mistralai/Mistral-7B-Instruct-v0.1
GPU_MEMORY_UTILIZATION	OPTIONAL, the max memory allowed to be utilized, default is 0.85
PORT	OPTIONAL, the port to use for serving, default is 8080

The container image automatically detects the number of GPUs and sets
--tensor-parallel-size to be equal to number of GPUs available. The
gpu-count.py script is used to detect number of GPUs.

Building

docker build -t ghcr.io/substratusai/vllm .

[gabriead]

Can you explain how I can interact with the docker container with Langchain for instance? Or can I use the HF text-inference framework and call this as an API endpoint?

[cybrtooth]

When you run this command it will create a endpoint that you can then call. You can use python for this or curl by sending a post request to the model. See the vllm documentation here for examples on how to call the model once it is spun up in a container (you can also spin it up with python, that is in this documentation): https://docs.vllm.ai/en/latest/getting_started/quickstart.html