chore: update the vLLM inference impl to use OpenAIMixin for openai-compat functions #3404
Conversation
…ompat functions
inference recordings from Qwen3-0.6B and vLLM 0.8.3 -
```
docker run --gpus all -v ~/.cache/huggingface:/root/.cache/huggingface -p 8000:8000 --ipc=host \
vllm/vllm-openai:latest \
--model Qwen/Qwen3-0.6B --enable-auto-tool-choice --tool-call-parser hermes
```
test with -
```
./scripts/integration-tests.sh --stack-config server:ci-tests --setup vllm --subdirs inference
```
meta-cla
bot
added
the
CLA Signed
mattf
requested review from
ashwinb,
yanxi0830,
hardikjshah,
raghotham,
ehhuang,
terrytangyuan,
leseb,
bbrowning,
reluctantfuturist and
slekkala1
as code owners
|
wait, we need to agree about the models we use for vllm testing with others who do this also -- cc @derekhiggins @bbrowning if there are thoughts |
With #3128 my intent was to test the provider code, so long the the model used can fit into the CI workers and is smart enough pass the integration tests I've no particular preference. In the CI job I've been hoping to merge I'm using Llama-3.2-1B-Instruct but I'm not against changing it if another model makes more sense ( the model is baked into the vllm image) |
what i could have used was a way to override the suite model. |
…ompat functions (llamastack#3404) # What does this PR do? update vLLM inference provider to use OpenAIMixin for openai-compat functions inference recordings from Qwen3-0.6B and vLLM 0.8.3 - ``` docker run --gpus all -v ~/.cache/huggingface:/root/.cache/huggingface -p 8000:8000 --ipc=host \ vllm/vllm-openai:latest \ --model Qwen/Qwen3-0.6B --enable-auto-tool-choice --tool-call-parser hermes ``` ## Test Plan ``` ./scripts/integration-tests.sh --stack-config server:ci-tests --setup vllm --subdirs inference ```
What does this PR do?
update vLLM inference provider to use OpenAIMixin for openai-compat functions
inference recordings from Qwen3-0.6B and vLLM 0.8.3 -
Test Plan