Tool Call Issues with Qwen2.5-VL Models (7B & 72B) under vLLM

[edwardzjl]

Hi there, thanks for the awesome work!

I've been experimenting with the Qwen2.5-VL models to build agent workflows, but I’ve run into a few issues when combining the VLMs with tools.

Chat Template Compatibility

First of all, it seems that the default chat templates provided for Qwen2.5-VL-7B-Instruct, Qwen2.5-VL-32B-Instruct or Qwen2.5-VL-72B-Instruct don't seem to support tool usage out of the box.

To work around this, I created a custom chat template that merges the multi-modal input format with the tool call part used in the non-VLM Qwen models (like this one from tokenizer_config.json). Based on some unit tests, I believe it should render prompts correctly (we’ll see the results later).

Test Setup

I deployed the model using vllm (0.8.2) with the following command (simplified for clarity):

vllm serve Qwen/Qwen2.5-VL-7B-Instruct --enable-auto-tool-choice --tool-call-parser hermes --chat-template /etc/qwen2.5-vl-7b-instruct/chat_template.jinja

Then I ran this basic test case, trying both with and without the sampling args:

from openai import OpenAI

client = OpenAI(base_url="http://qwen2dot5-vl-72b-instruct.myns.svc/v1", api_key="dummy")

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get current temperature for a given location.",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "City and country e.g. Bogotá, Colombia"
                    }
                },
                "required": ["location"],
                "additionalProperties": "false"
            }
        }
    }
]

msgs = [
    {"role": "user", "content": "What is the weather like in London today?"}
]

response = client.chat.completions.create(
    model=client.models.list().data[0].id,
    messages=msgs,
    tools=tools
    # temperature=0.7,
    # top_p=0.8,
    # extra_body={"repetition_penalty": 1.05}
)

response.choices[0].message.content

Unexpected Behavior

Unfortunately, neither the 7B nor the 72B model handles tool usage properly:

• 7B model ignores the tools entirely and always replies with something like:

  <tool_call>\n addCriterion\n<tool_call>\n\n\n\n\n\n\n\n\n\n
  

• 72B model seems to recognize the tool but includes emojis and always without the closing XML tag:

  <tool_call>\n📐\n{"name": "get_weather", "arguments": {"location": "London, UK"}}\n📐
  

Additional Debug Info

Here’s the system prompt after the template is rendered (from server logs):

<|im_start|>system
You are Qwen, created by Alibaba Cloud. You are a helpful assistant.

# Tools

You may call one or more functions to assist with the user query.

You are provided with function signatures within <tools></tools> XML tags:
<tools>
{"type": "function", "function": {"name": "get_weather", "description": "Get current temperature for a given location.", "parameters": {"type": "object", "properties": {"location": {"type": "string", "description": "City and country e.g. Bogotá, Colombia"}}, "required": ["location"], "additionalProperties": "false"}}}
</tools>

For each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:
<tool_call>
{"name": <function-name>, "arguments": <args-json-object>}
</tool_call><|im_end|>
<|im_start|>user
What is the weather like in London today?<|im_end|>
<|im_start|>assistant

full log entry

INFO 04-10 18:48:20 [logger.py:39] Received request chatcmpl-7acf794f-d260-4ff0-9ff4-8d881055f507: prompt: '<|im_start|>system\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.\n\n# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>\n{"type": "function", "function": {"name": "get_weather", "description": "Get current temperature for a given location.", "parameters": {"type": "object", "properties": {"location": {"type": "string", "description": "City and country e.g. Bogotá, Colombia"}}, "required": ["location"], "additionalProperties": "false"}}}\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{"name": <function-name>, "arguments": <args-json-object>}\n</tool_call><|im_end|>\n<|im_start|>user\nWhat is the weather like in London today?<|im_end|>\n<|im_start|>assistant\n', params: SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.05, temperature=1.0, top_p=0.001, top_k=1, min_p=0.0, seed=None, stop=[], stop_token_ids=[], bad_words=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=32573, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, guided_decoding=None, extra_args=None), prompt_token_ids: None, lora_request: None, prompt_adapter_request: None.

---

Any ideas on what might be going wrong? Are there any recommended best practices or known limitations when using Qwen2.5-VL models with tools in this way?

Thanks again for the great work!

[edwardzjl]

I also created a simple test case based on the example provided in the cookbook:

from openai import OpenAI
from qwen_agent.llm.fncall_prompts.nous_fncall_prompt import (
    NousFnCallPrompt,
    Message,
    ContentItem,
)
from qwen_agent.tools.base import BaseTool, register_tool

@register_tool("weather_tool")
class WeatherTool(BaseTool):
    @property
    def description(self) -> str:
        return "Get current temperature for a given location."

    parameters = {
        "type": "object",
        "properties": {
            "location": {
                "type": "string",
                "description": "City and country e.g. Bogotá, Colombia"
            }
        },
        "required": ["location"]
    }

    def __init__(self, cfg=None):
        super().__init__(cfg)

    def call(self, params: str | dict, **kwargs):
        params = self._verify_json_format_args(params)
        ...

weather_tool = WeatherTool()

client = OpenAI(base_url="http://qwen2dot5-vl-72b-instruct.myns.svc/v1", api_key="dummy")

system_message = NousFnCallPrompt().preprocess_fncall_messages(
    messages=[
        Message(role="system", content=[ContentItem(text="You are a helpful assistant.")]),
    ],
    functions=[weather_tool.function],
    lang=None,
)
system_message = system_message[0].model_dump()

messages=[
    {
        "role": "system",
        "content": [
            {"type": "text", "text": msg["text"]} for msg in system_message["content"]
        ],
    },
    {
        "role": "user",
        "content": [
            {"type": "text", "text": "What is the weather like in London today?"},
        ],
    }
]
completion = client.chat.completions.create(
    model = client.models.list().data[0].id,
    messages = messages,
)

completion.choices[0].message.content

and observed the same result. Here's the response from the 72B model:

<tool_call>\n📐\n{"name": "weather_tool", "arguments": {"location": "London, UK"}}\n📐

[edwardzjl]

Also, the exact same setup works correctly with Qwen2.5-7B-Instruct and Qwen2.5-72B-Instruct (non-VLM versions).

[eav-solution]

Try set temperature like 0.0001

[edwardzjl]

Try set temperature like 0.0001

Thanks for the suggestion! I tried setting temperature=0.0001, but unfortunately the output is still the same — the tool call format is not correct.

[aditya1709]

If you look closely at the HF model they have made edits to their tokenizer_config.json which removed tool_calling related tokens.
This changed the behavior of our system completely and tool calls started failing, I don't know if its the same problem as what you are seeing but its something worth looking into. If you use the earlier version of the tokenizer_config tool_use works well with some problems.

[edwardzjl]

@aditya1709 Do you mean this commit? If so, I am not using that chat template. I'm using a custom chat template that based on both qwen2.5-instruct and qwen2.5-vl-instruct, designed to support both tool usage and multi-modal input.

You can see the "Additional Debug Info" part of my original post to see the prompt after applying my chat template, It should be correct for tool use.

[tadangkhoa1999]

@edwardzjl Thanks for providing the chat template, before I couldn't use function call with Qwen/Qwen2.5-VL-32B-Instruct, but after applying your chat template I was able to use function call.

Strangely I didn't get the error you described, although I applied your method.

[YANGTUOMAO]

it appears frequntlly when i use it to run a agent task.

[edwardzjl]

@tadangkhoa1999 I just tested the VL-32B model, and surprisingly, it sometimes works. However, it still tends to fail when handling more complex scenarios -- such as intricate tools or complex instructions. That said, the VL-32B seems to perform better than the VL-72B when it comes to tool usage.

[aditya1709]

@edwardzjl I tried your chat template but it seems to be struggling on some usecases more than the template used in the commit I mentioned. Of course this is anecdotal since I do not have quantitative numbers.

[tadangkhoa1999]

@edwardzjl @aditya1709
I think it's no coincidence that the developer's published chat template has the tool use section omitted.
Maybe the model was trained to focus on achieving good results on image benchmark sets.
We can try to change the prompt so that the model can handle it better.

[edwardzjl]

@aditya1709 My point is that my custom template should render prompts in the same way as the templates used in other Qwen-2.5-Instruct models with tool support, as well as the template before the commit I mentioned. You can check my unit test here to verify: https://github.com/edwardzjl/chat-templates/blob/30561e775c867ed6f74f930f64bcca53ce97eb29/qwen2_5/tests/test_system_messages.py#L69-L102

That said, it’s possible my tests are too simple and I might be missing something. Still, the VL-32B model performs slightly better than both VL-7B and VL-72B when using the same template.

@tadangkhoa1999 What confuses me is that in their model card, they describe Qwen2.5-VL as:

• Being agentic: Qwen2.5-VL directly plays as a visual agent that can reason and dynamically direct tools, which is capable of computer use and phone use.

And in the tech report there's also a section about agent benchmarks.

[jingrsu]

I also created a simple test case based on the example provided in the cookbook:

from openai import OpenAI
from qwen_agent.llm.fncall_prompts.nous_fncall_prompt import (
NousFnCallPrompt,
Message,
ContentItem,
)
from qwen_agent.tools.base import BaseTool, register_tool

@register_tool("weather_tool")
class WeatherTool(BaseTool):
@Property
def description(self) -> str:
return "Get current temperature for a given location."

parameters = {
    "type": "object",
    "properties": {
        "location": {
            "type": "string",
            "description": "City and country e.g. Bogotá, Colombia"
        }
    },
    "required": ["location"]
}

def __init__(self, cfg=None):
    super().__init__(cfg)

def call(self, params: str | dict, **kwargs):
    params = self._verify_json_format_args(params)
    ...

weather_tool = WeatherTool()

client = OpenAI(base_url="http://qwen2dot5-vl-72b-instruct.myns.svc/v1", api_key="dummy")

system_message = NousFnCallPrompt().preprocess_fncall_messages(
messages=[
Message(role="system", content=[ContentItem(text="You are a helpful assistant.")]),
],
functions=[weather_tool.function],
lang=None,
)
system_message = system_message[0].model_dump()

messages=[
{
"role": "system",
"content": [
{"type": "text", "text": msg["text"]} for msg in system_message["content"]
],
},
{
"role": "user",
"content": [
{"type": "text", "text": "What is the weather like in London today?"},
],
}
]
completion = client.chat.completions.create(
model = client.models.list().data[0].id,
messages = messages,
)

completion.choices[0].message.content
and observed the same result. Here's the response from the 72B model:

<tool_call>\n📐\n{"name": "weather_tool", "arguments": {"location": "London, UK"}}\n📐

@edwardzjl Hi, I observed the same issue when using the vLLM server. However, when loading the model directly with from_pretrained via Hugging Face, everything seems to work fine.

What confuses me is that after the vLLM server processes any query containing an image, trying your example again works perfectly fine. Maybe some issues on the vLLM side?

[fgolemo]

That is so weird. @jingrsu I just tried and confirmed your observation! Following the setup of @edwardzjl above, all on the same instance of Qwen/Qwen2.5-VL-7B-Instruct

• run a tool use example directly: 👎 bad output
• run an image but without a tool prompt, then a tool example (i.e. with 2 different chat histories): 👎 bad output
• run an image with a tool prompt, then a tool use example: 👍 good output

Here's a full example to show the tool use working on the 7B model

import base64
import json
import os
from datetime import datetime
from openai import OpenAI

# Constants for OpenAI client configuration
OPENAI_API_KEY = "EMPTY"
OPENAI_API_BASE = "http://SERVER_IP:8000/v1"

# Initialize OpenAI client
client = OpenAI(api_key=OPENAI_API_KEY, base_url=OPENAI_API_BASE)

# Model configuration
MODEL_NAME = "Qwen/Qwen2.5-VL-7B-Instruct"

EXAMPLE_IMG = "https://modelscope.oss-cn-beijing.aliyuncs.com/resource/qwen.png"

# Function to get current temperature

def get_current_temperature(location: str, unit: str = "celsius") -> dict:
    """Get current temperature at a location."""
    return {
        "temperature": 26.1,
        "location": location,
        "unit": unit,
    }

# Function to get temperature on a specific date

def get_temperature_date(location: str, date: str, unit: str = "celsius") -> dict:
    """Get temperature at a location and date."""
    return {
        "temperature": 25.9,
        "location": location,
        "date": date,
        "unit": unit,
    }

# Function to retrieve a function by name

def get_function_by_name(name: str):
    """Retrieve a function by its name."""
    functions = {
        "get_current_temperature": get_current_temperature,
        "get_temperature_date": get_temperature_date,
    }
    return functions.get(name)

# Tool definitions
TOOLS = [
    {
        "type": "function",
        "function": {
            "name": "get_current_temperature",
            "description": "Get current temperature at a location.",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": 'The location to get the temperature for, in the format "City, State, Country".',
                    },
                    "unit": {
                        "type": "string",
                        "enum": ["celsius", "fahrenheit"],
                        "description": 'The unit to return the temperature in. Defaults to "celsius".',
                    },
                },
                "required": ["location"],
            },
        },
    },
    {
        "type": "function",
        "function": {
            "name": "get_temperature_date",
            "description": "Get temperature at a location and date.",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": 'The location to get the temperature for, in the format "City, State, Country".',
                    },
                    "date": {
                        "type": "string",
                        "description": 'The date to get the temperature for, in the format "Year-Month-Day".',
                    },
                    "unit": {
                        "type": "string",
                        "enum": ["celsius", "fahrenheit"],
                        "description": 'The unit to return the temperature in. Defaults to "celsius".',
                    },
                },
                "required": ["location", "date"],
            },
        },
    },
]

# Initial system message
MESSAGES = [
    {"role": "system", "content": f"You are Qwen, created by Alibaba Cloud. You are a helpful assistant.\n\nCurrent Date: {datetime.now().strftime('%Y-%m-%d')}"},
    {"role": "user",  "content": "What's the temperature in San Francisco now? How about tomorrow?"},
]

# Messages with image
messages_img = [
    {"role": "system", "content": "You are a helpful assistant."},
    {
        "role": "user",
        "content": [
            {
                "type": "image_url",
                "image_url": {
                    "url": EXAMPLE_IMG
                },
            },
            {"type": "text", "text": "Please describe the image."},
        ],
    },
]

# Copy image messages
messages = messages_img.copy()

# Create chat completion with image
response = client.chat.completions.create(
    model=MODEL_NAME,
    messages=messages,
    tools=TOOLS,
    stream=True,
)

# Print response chunks
for chunk in response:
    print(chunk.choices[0].delta.content, end="", flush=True)

# Add temperature messages
messages += MESSAGES

# Create chat completion with temperature
response = client.chat.completions.create(
    model=MODEL_NAME,
    messages=messages,
    tools=TOOLS,    
)

# Print response
print(response)

# Append response to messages
messages.append(response.choices[0].message.model_dump())

# Handle tool calls
if tool_calls := messages[-1].get("tool_calls", None):
    for tool_call in tool_calls:
        call_id: str = tool_call["id"]
        if fn_call := tool_call.get("function"):
            fn_name: str = fn_call["name"]
            fn_args: dict = json.loads(fn_call["arguments"])
            fn_res: str = json.dumps(get_function_by_name(fn_name)(**fn_args))
            messages.append({
                "role": "tool",
                "content": fn_res,
                "tool_call_id": call_id,
            })

# Print new messages
print("new messages:")
print(messages)

# Final chat completion
response = client.chat.completions.create(
    model=MODEL_NAME,
    messages=messages,
    tools=TOOLS,
)

# Print final response
print(response.choices[0].message.content)

And this is how I start the VLLM server (on 2x Titan RTX):
vllm serve Qwen/Qwen2.5-VL-7B-Instruct --port 8000 --host 0.0.0.0 --dtype=float16 --tensor-parallel-size 2 --gpu_memory_utilization=0.95 --max-model-len 32768 --enable-auto-tool-choice --tool-call-parser hermes --limit-mm-per-prompt image=5,video=0 --chat-template ~/chat-templates/gwen-vl-tool.jinja

[cyjin-yl]

Also meet this problem on Qwen2.5-VL-Instruct-72B. Output 📐 and ⚗ instead of </tool_call>