Your current environment
Environment / Docker image
The issue occurs when running vLLM inside the following Docker image:
Image: lmcache/vllm-openai:build-latest
Relevant characteristics
- Base OS: Ubuntu 24.04
- Python: 3.12
- CUDA: 12.8.1
- Driver compatibility: 570.124.06
- NCCL: 2.25.1
- cuDNN: 9.8.0
- TensorRT: 10.9.0
- vLLM: nightly (built from source via build.txt)
- Architecture: amd64
- NVIDIA_VISIBLE_DEVICES=all
- TORCH_CUDA_ARCH_LIST=10.0+PTX 12.0+PTX
🐛 Describe the bug
🐛 Describe the bug
When serving Olmo-3.1-32B via vLLM, the model never emits structured tool calls, even when tool calling is explicitly enabled.
Instead of returning a tool invocation compatible with the olmo3 tool call parser, the model produces only natural-language reasoning describing how it would call a tool. As a result, vLLM never triggers any tool execution, even though the model otherwise generates correct text responses and does not crash or error.
The model is served using the following command:
vllm serve cyankiwi/Olmo-3.1-32B-Think-AWQ-4bit
--port 8000
--tensor-parallel-size 1
--gpu-memory-utilization 0.9
--max-model-len 80000
--tool-call-parser olmo3
--enable-auto-tool-choice
--trust-remote-code
No explicit prompt template is provided (as not asked to do so via VLLM website to tool calling)
A minimal example that reproduces the issue is the prompt:
write file A.txt in root.
The observed output consists only of free-form reasoning text such as “Okay, the user wants me to create a file named A.txt in the root directory. I should use the write_file tool…”, but no structured tool call is emitted.
Expected behavior
The model should emit a structured tool call (for example, JSON) that can be parsed by vLLM and executed by the tool runtime. (that will work for agent mode)
Actual behavior
- Only free-form reasoning text is generated
- No tool call is returned (it even says it has nothing available, but can list any of them by words)
- The
olmo3 tool call parser is never triggered or it model prevents the token to call
- Auto tool choice has no observable effect
Additional observations
- The issue reproduces consistently
It is currently unclear whether this behavior is caused by a missing or required prompt template for Olmo-3, a mismatch between Olmo-3’s tool-calling format and vLLM’s olmo3 tool parser, or a vLLM integration issue.
Before submitting a new issue...
Your current environment
Environment / Docker image
The issue occurs when running vLLM inside the following Docker image:
Image: lmcache/vllm-openai:build-latest
Relevant characteristics
🐛 Describe the bug
🐛 Describe the bug
When serving Olmo-3.1-32B via vLLM, the model never emits structured tool calls, even when tool calling is explicitly enabled.
Instead of returning a tool invocation compatible with the
olmo3tool call parser, the model produces only natural-language reasoning describing how it would call a tool. As a result, vLLM never triggers any tool execution, even though the model otherwise generates correct text responses and does not crash or error.The model is served using the following command:
vllm serve cyankiwi/Olmo-3.1-32B-Think-AWQ-4bit
--port 8000
--tensor-parallel-size 1
--gpu-memory-utilization 0.9
--max-model-len 80000
--tool-call-parser olmo3
--enable-auto-tool-choice
--trust-remote-code
No explicit prompt template is provided (as not asked to do so via VLLM website to tool calling)
A minimal example that reproduces the issue is the prompt:
write file A.txt in root.
The observed output consists only of free-form reasoning text such as “Okay, the user wants me to create a file named A.txt in the root directory. I should use the write_file tool…”, but no structured tool call is emitted.
Expected behavior
The model should emit a structured tool call (for example, JSON) that can be parsed by vLLM and executed by the tool runtime. (that will work for agent mode)
Actual behavior
olmo3tool call parser is never triggered or it model prevents the token to callAdditional observations
It is currently unclear whether this behavior is caused by a missing or required prompt template for Olmo-3, a mismatch between Olmo-3’s tool-calling format and vLLM’s
olmo3tool parser, or a vLLM integration issue.Before submitting a new issue...