Hello vLLM team,
We are currently running the Qwen/Qwen3-32B model using vLLM v0.8.5 with full context length and tool-calling enabled via the Hermes parser, as documented.
However, during inference via the OpenAI-compatible endpoint, we consistently encounter the following error from the Hermes tool parser:
`ERROR [hermes_tool_parser.py:110] Error in extracting tool call from response.
Traceback (most recent call last):
File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/tool_parsers/hermes_tool_parser.py", line 88, in extract_tool_calls
json.loads(match[0] if match[0] else match[1])
File "/usr/lib/python3.12/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3.12/json/decoder.py", line 341, in decode
raise JSONDecodeError("Extra data", s, end)
json.decoder.JSONDecodeError: Extra data: line 3 column 1 (char 527)`
This seems to indicate that the Hermes parser is failing to extract and decode the tool call due to extra data in the string passed to json.loads.
**Our deployment configuration includes the following relevant arguments:**
`args:
- --host
- 0.0.0.0
- --port
- "8000"
- --model
- Qwen/Qwen3-32B
- --swap-space
- "16"
- --disable-log-requests
- --tensor-parallel-size
- "8"
- --gpu-memory-utilization
- "0.95"
- --max-model-len
- "131072"
- --max-seq-len-to-capture
- "131072"
- --max-num-seqs
- "5"
- --enable-chunked-prefill
- --tool-call-parser
- hermes
- --enable-auto-tool-choice`
We suspect the issue may be caused by the tool call extraction logic in hermes_tool_parser.py, particularly if the model output includes multiple JSON blocks or additional text beyond the expected tool call.
**Steps to reproduce:**
Deploy Qwen/Qwen3-32B with the above config.
Query the OpenAI-compatible endpoint with a tool-calling prompt.
Observe the traceback from hermes_tool_parser.
**Expected behavior:** Hermes parser should correctly extract the tool call or fail gracefully with clearer diagnostics if the output is malformed.
Please let us know if this is a known issue or if additional formatting or post-processing is needed for compatibility with Hermes.
Thank you for your support and for maintaining vLLM!
I want to run inference of a [specific model](put link here). I don't know how to integrate it with vllm.
Your current environment
How would you like to use vllm
I want to run inference of a [specific model](put link here). I don't know how to integrate it with vllm.
Before submitting a new issue...