Does qwen2.5-coder support function calling?

[Muuut]

According to the documentation, I use VLLM to build a openai compatible server for qwen2.5-coder-7b-instruct.

That is my code

messages = [
    {
        "role": "user",
        "content": "Calculate how many r's are in the word 'strawberry'"
    }
]

# Define the tools
tools = [{
    "type": "function",
    "function": {
        "name": "execute_python",
        "description": "Execute python code in a Jupyter notebook cell and return result",
        "parameters": {
            "type": "object",
            "properties": {
                "code": {
                    "type": "string",
                    "description": "The python code to execute in a single cell"
                }
            },
            "required": ["code"]
        }
    }
}]

# Generate text with OpenAI
response = client.chat.completions.create(
    model="qwen2.5-7B-coder",
    messages=messages,
    tools=tools,
)

However, I got this:

The finish_reason is "stop" instead of "tool_calls", and the tool_calls is empty.
So I wonder whether qwen2.5-coder series support function calling?

[sagarey]

same problem recurred on Qwen2.5-Coder-32B-instruct deployed by vllm, while Qwen2.5-32B-instruct support function calling as expected.

[Muuut]

same problem recurred on Qwen2.5-Coder-32B-instruct deployed by vllm, while Qwen2.5-32B-instruct support function calling as expected.

I did not set the system content in the message before. After setting, function calling works normally.

[sagarey]

same problem recurred on Qwen2.5-Coder-32B-instruct deployed by vllm, while Qwen2.5-32B-instruct support function calling as expected.

I did not set the system content in the message before. After setting, function calling works normally.

Default qwen system prompt? I tried it, and still the problem that tool_calls is empty and content returns tool call string. Whether it is Qwen2.5-Coder-32B-instruct or Qwen2.5-Coder-7B-instruct, without any adjustments, only model switch to Qwen2.5-32B-instruct or Qwen2.5-7B-instruct can tool_calls be obtained correctly.

[sagarey]

When Qwen2.5-Coder deployed with Ollama, it can return tool_calls as expected.

[ethal0n]

same problem

if run the example from the documentation through vllm, the call function works with the usual Qwen2.5-7B-instruct and does not work with the Qwen2.5-Coder-32B-instruct-AWQ

Qwen2.5-7B-instruct return "Choice(finish_reason='tool_calls'....."
Qwen2.5-Coder-32B-instruct-AWQ return "Choice(finish_reason='stop'....."

and yes, this example from the documentation contains "role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant.\n\nCurrent Date: 2024-09-30" in messages

[huang-jl]

Same problem as @ethal0n on vllm with Qwen2.5-Coder-3B-Instruct, with the same prompt/messages as documentation.

[ethal0n]

I managed to make the coder models (tried 14b and 32b) work with calling functions via Qwen-Agent

[huang-jl]

I have read the code of Qwen-Agent and vLLM about their function calling logic. I find that vLLM uses the nous or hermes style. However, in your links to Qwen-Agent, it uses 'qwen' by default. Actually, I think it is the old function calling template that Qwen2 support instead of Qwen2.5 function call template.

(In short, I think the Qwen2.5-Coder models should support newest hermes-style template, which is supported in vLLM. If Qwen2.5-Coder models only support its Qwen-2 style template, then it explains why function calling works in Qwen-Agent, but not works in vLLM).

So what's value of fncall_prompt_type passed here in your code when using Qwen-Agent, @ethal0n ?

[ethal0n]

vllm run as usual without any prefixes to call functions

[huang-jl]

Thanks for you quick reply. When using Qwen-Agent, you need to pass an argument called fncall_prompt_type. For example (this is copied from your links about Qwen-Agent):

llm = get_chat_model({
        # Use the model service provided by DashScope:
        'model': 'qwen2.5-72b-instruct',
        'model_server': 'dashscope',
        'api_key': os.getenv('DASHSCOPE_API_KEY'),
        'generate_cfg': {
            'fncall_prompt_type': fncall_prompt_type # NOTE HERE!!!
        },
    })

Do you pass this argument, or what is the value you pass, when you make the coder models works with function calling, @ethal0n ?

[ethal0n]

I haven't used it. According to the description it's for DashScope, in the example with vllm it's not there

[huang-jl]

That make sense. I think the conclusions of this issue are:

1. The function calling does not works for Qwen2.5 coder models in vLLM, because it does not support hermes-style function calling.
2. The function calling works for normal Qwen2.5 models (e.g., Qwen2.5-14B-Instruct) in vLLM, as in Qwen docs.
3. The function calling works for coder models in Qwen-Agent. Since Qwen-Agent by default uses Qwen2-style function calling. This Qwen2-style is not support in vLLM, but is supported by the Qwen2.5 coder model.

I think maybe Qwen2.5 coder models does not trained the dataset with hermes-style function calling as normal Qwen2.5 models. If you would like to use the function call in vLLM:

1. Use normal Qwen2.5 models (this is the easiest method).
2. Write a template and tool call parser for vLLM (you can refer to here), which parse the LLM output according to the Qwen2-style.
3. Uses Qwen-Agent with the built-in support for Qwen2-style.

[aep]

having difficulties getting this working. the qwen2-style docs seems a bit dubious?
the system prompt instructs the model to include RESULT in the tool call, which it does, by hallucinating the result.

[WinPooh32]

Found working trick for Ollama: replace <tool_call> by [tool_call] at the model template.