[Bug]: Is vllm support function call mode? #6631
Description
Your current environment
Device: Nvidia GeForce 4090
software: vllm 0.5.2 + openai 1.30.5 + transformes 4.42.4
🐛 Describe the bug
I use OpenAI api and vllm to deploy local Qwen2 llm, But vllm function call mode does not work. The OpenAI interface correctly passed the tools info parameters to vllm, but vllm did not use it. If I enable ' tool_choice="auto" 'parameter, I will encounter with 400 error code.
---------------------------------------------------------------------server script-------------------------------------------------------------
python entrypoints/openai/api_server.py --model="xxx/Qwen2-1.5B-Instruct" --trust-remote-code --host "localhost" --port 8000 --dtype auto
-------------------------------------------------------------client code ------------------------------------------------------------------
from openai import OpenAI
tools = [
{
"type": "function",
"function": {
"name": "get_current_weather",
"description": "Get the current weather in a given location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city and state, e.g. San Francisco, CA",
},
"unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
},
"required": ["location"],
},
},
}
]
openai_api_key = "xxx"
openai_api_base = "http://localhost:8000/v1/"
client = OpenAI(
api_key=openai_api_key,
base_url=openai_api_base,
)
models = client.models.list()
model = models.data[0].id
chat_completion = client.chat.completions.create(
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "help me query currnet weather in BeiJing."},
],
model=model,
tools=tools,
# tool_choice="auto"
)
print("response: ", chat_completion.choices[0].message)
--------------------------------------------extra info && response ----------------------------------------------
response: ChatCompletionMessage(content='xxx', role='assistant', function_call=None, tool_calls=[])
---------------------------------- enable tool_choice="auto" parameter ----------------------------------------
openai.BadRequestError: Error code: 400 - {'object': 'error', 'message': "[{'type': 'value_error', 'loc': ('body',), 'msg': 'Value error, Currently only named tools are supported.', 'input': {'messages': [{'role': 'system', 'content': 'You are a helpful assistant.'}, {'role': 'user', 'content': 'help me query currnet weather in San Francisco.'}], 'model': '/home/zhangfan/deep_learning/Qwen2/examples/sft/Qwen2-1.5B-Instruct', 'tool_choice': 'auto', 'tools': [{'type': 'function', 'function': {'name': 'get_current_weather', 'description': 'Get the current weather in a given location', 'parameters': {'type': 'object', 'properties': {'location': {'type': 'string', 'description': 'The city and state, e.g. San Francisco, CA'}, 'unit': {'type': 'string', 'enum': ['celsius', 'fahrenheit']}}, 'required': ['location']}}}]}, 'ctx': {'error': ValueError('Currently only named tools are supported.')}}]", 'type': 'BadRequestError', 'param': None, 'code': 400}