[Bug]: Hermes tool parser stream output error in Qwen3 case

[LiuLi1998]

Your current environment

Details

DEBUG 06-03 06:00:03.465 [init.py:28] No plugins for group vllm.platform_plugins found.
DEBUG 06-03 06:00:03.465 [init.py:34] Checking if TPU platform is available.
DEBUG 06-03 06:00:03.465 [init.py:44] TPU platform is not available because: No module named 'libtpu'
DEBUG 06-03 06:00:03.465 [init.py:52] Checking if CUDA platform is available.
DEBUG 06-03 06:00:03.484 [init.py:72] Confirmed CUDA platform is available.
DEBUG 06-03 06:00:03.485 [init.py:100] Checking if ROCm platform is available.
DEBUG 06-03 06:00:03.486 [init.py:114] ROCm platform is not available because: No module named 'amdsmi'
DEBUG 06-03 06:00:03.486 [init.py:122] Checking if HPU platform is available.
DEBUG 06-03 06:00:03.486 [init.py:129] HPU platform is not available because habana_frameworks is not found.
DEBUG 06-03 06:00:03.486 [init.py:140] Checking if XPU platform is available.
DEBUG 06-03 06:00:03.486 [init.py:150] XPU platform is not available because: No module named 'intel_extension_for_pytorch'
DEBUG 06-03 06:00:03.486 [init.py:158] Checking if CPU platform is available.
DEBUG 06-03 06:00:03.486 [init.py:181] Checking if Neuron platform is available.
DEBUG 06-03 06:00:03.487 [init.py:52] Checking if CUDA platform is available.
DEBUG 06-03 06:00:03.498 [init.py:72] Confirmed CUDA platform is available.
INFO 06-03 06:00:03.499 [init.py:248] Automatically detected platform cuda.
Collecting environment information...
/usr/local/lib/python3.10/dist-packages/_distutils_hack/init.py:30: UserWarning: Setuptools is replacing distutils. Support for replacing an already imported distutils is deprecated. In the future, this condition will fail. Register concerns at https://github.com/pypa/setuptools/issues/new?template=distutils-deprecation.yml
warnings.warn(
Traceback (most recent call last):
File "/workspace/collect_env.py", line 818, in
main()
File "/workspace/collect_env.py", line 797, in main
output = get_pretty_env_info()
File "/workspace/collect_env.py", line 792, in get_pretty_env_info
return pretty_str(get_env_info())
File "/workspace/collect_env.py", line 598, in get_env_info
vllm_version = get_vllm_version()
File "/workspace/collect_env.py", line 285, in get_vllm_version
version_str = version_tuple[-1]
TypeError: 'generator' object is not subscriptable
root@ll-qwen360-fc5fc5f95-zptzk:/workspace# python3 collect_env.py
DEBUG 06-03 06:01:15.959 [init.py:28] No plugins for group vllm.platform_plugins found.
DEBUG 06-03 06:01:15.959 [init.py:34] Checking if TPU platform is available.
DEBUG 06-03 06:01:15.959 [init.py:44] TPU platform is not available because: No module named 'libtpu'
DEBUG 06-03 06:01:15.959 [init.py:52] Checking if CUDA platform is available.
DEBUG 06-03 06:01:15.977 [init.py:72] Confirmed CUDA platform is available.
DEBUG 06-03 06:01:15.978 [init.py:100] Checking if ROCm platform is available.
DEBUG 06-03 06:01:15.978 [init.py:114] ROCm platform is not available because: No module named 'amdsmi'
DEBUG 06-03 06:01:15.978 [init.py:122] Checking if HPU platform is available.
DEBUG 06-03 06:01:15.978 [init.py:129] HPU platform is not available because habana_frameworks is not found.
DEBUG 06-03 06:01:15.978 [init.py:140] Checking if XPU platform is available.
DEBUG 06-03 06:01:15.978 [init.py:150] XPU platform is not available because: No module named 'intel_extension_for_pytorch'
DEBUG 06-03 06:01:15.978 [init.py:158] Checking if CPU platform is available.
DEBUG 06-03 06:01:15.979 [init.py:181] Checking if Neuron platform is available.
DEBUG 06-03 06:01:15.979 [init.py:52] Checking if CUDA platform is available.
DEBUG 06-03 06:01:15.990 [init.py:72] Confirmed CUDA platform is available.
INFO 06-03 06:01:15.991 [init.py:248] Automatically detected platform cuda.
Collecting environment information...
/usr/local/lib/python3.10/dist-packages/_distutils_hack/init.py:30: UserWarning: Setuptools is replacing distutils. Support for replacing an already imported distutils is deprecated. In the future, this condition will fail. Register concerns at https://github.com/pypa/setuptools/issues/new?template=distutils-deprecation.yml
warnings.warn(
Traceback (most recent call last):
File "/workspace/collect_env.py", line 818, in
main()
File "/workspace/collect_env.py", line 797, in main
output = get_pretty_env_info()
File "/workspace/collect_env.py", line 792, in get_pretty_env_info
return pretty_str(get_env_info())
File "/workspace/collect_env.py", line 598, in get_env_info
vllm_version = get_vllm_version()
File "/workspace/collect_env.py", line 285, in get_vllm_version
version_str = version_tuple[-1]
TypeError: 'generator' object is not subscriptable

🐛 Describe the bug

The extract_tool_calls_streaming component of the Hermes tool parser improperly handled a specific token during the parsing of streaming tool function outputs when the Qwen3 generates a specific token.

case: Qwen3 tokenizer vocab has a token "}, id 9207
if Qwen3 models output this token, hermes extract_tool_calls_streaming method will truncate this token to " in which case } will be discarded, further led to broken tool call's json schema output

LLM

Qwen3 series

vLLM serve log

[2025-06-03 13:09:22] DEBUG 06-03 05:09:22.878 [hermes_tool_parser.py:127] delta_text: "}
[2025-06-03 13:09:22] DEBUG 06-03 05:09:22.878 [hermes_tool_parser.py:128] delta_token_ids: [9207]
[2025-06-03 13:09:22] DEBUG 06-03 05:09:22.878 [hermes_tool_parser.py:243] Parsed tool call {'name': 'summary', 'arguments': {'query': 'DeepSeek-R1-0528具体有哪些提升', 'title': 'DeepSeek-R1-0528版本升级亮点解析', 'style': '研究报告', 'outline': [{'h2': '一、版本迭代背景', 'guideline': '阐述DeepSeek-R1系列研发历程及0528版本开发目标'}, {'h2': '二、核心技术突破', 'guideline': '对比分析模型架构优化、训练数据升级、算法改进等核心提升点'}, {'h2': '三、性能指标提升', 'guideline': '从推理速度、准确率、多语言支持度等维度量化展示提升幅度'}, {'h2': '四、应用场景拓展', 'guideline': '说明在代码生成、内容创作、逻辑推理等场景的专项优化'}, {'h2': '五、用户使用体验', 'guideline': '描述实际应用中的响应速度、交互流畅度等体验改善'}]}}
[2025-06-03 13:09:22] DEBUG 06-03 05:09:22.878 [hermes_tool_parser.py:282] Trying to parse current tool call with ID 0
[2025-06-03 13:09:22] DEBUG 06-03 05:09:22.878 [hermes_tool_parser.py:296] diffing old arguments: {'query': 'DeepSeek-R1-0528具体有哪些提升', 'title': 'DeepSeek-R1-0528版本升级亮点解析', 'style': '研究报告', 'outline': [{'h2': '一、版本迭代背景', 'guideline': '阐述DeepSeek-R1系列研发历程及0528版本开发目标'}, {'h2': '二、核心技术突破', 'guideline': '对比分析模型架构优化、训练数据升级、算法改进等核心提升点'}, {'h2': '三、性能指标提升', 'guideline': '从推理速度、准确率、多语言支持度等维度量化展示提升幅度'}, {'h2': '四、应用场景拓展', 'guideline': '说明在代码生成、内容创作、逻辑推理等场景的专项优化'}, {'h2': '五、用户使用体验', 'guideline': '描述实际应用中的响应速度、交互流畅度等体验改善'}]}
[2025-06-03 13:09:22] DEBUG 06-03 05:09:22.878 [hermes_tool_parser.py:297] against new ones: {'query': 'DeepSeek-R1-0528具体有哪些提升', 'title': 'DeepSeek-R1-0528版本升级亮点解析', 'style': '研究报告', 'outline': [{'h2': '一、版本迭代背景', 'guideline': '阐述DeepSeek-R1系列研发历程及0528版本开发目标'}, {'h2': '二、核心技术突破', 'guideline': '对比分析模型架构优化、训练数据升级、算法改进等核心提升点'}, {'h2': '三、性能指标提升', 'guideline': '从推理速度、准确率、多语言支持度等维度量化展示提升幅度'}, {'h2': '四、应用场景拓展', 'guideline': '说明在代码生成、内容创作、逻辑推理等场景的专项优化'}, {'h2': '五、用户使用体验', 'guideline': '描述实际应用中的响应速度、交互流畅度等体验改善'}]}
[2025-06-03 13:09:22] DEBUG 06-03 05:09:22.878 [hermes_tool_parser.py:347] got diff "

Client code

import openai

api_key = ""
api_base = ""
client = openai.OpenAI(
    api_key=api_key,
    base_url=api_base,
)

tools = [
    {
        "type": "function",
        "function": {
            "name": "summary",
            "description": "总结工具，用于在资料准备完成后生成回答，并展示给用户。",
            "parameters": {
                "type": "object",
                "properties": {
                    "query": {
                        "description": "用户的问题",
                        "type": "string"
                    },
                    "title": {
                        "description": "回答的标题，与用户问题密切相关，不超过15个字。",
                        "type": "string"
                    },
                    "style": {
                        "description": "回答风格。可能的回答风格包含: 自媒体文章，短视频脚本，媒体稿件，论文，研究报告等",
                        "enum": [
                            "自媒体文章",
                            "短视频脚本",
                            "媒体稿件",
                            "论文",
                            "研究报告"
                        ],
                        "type": "string"
                    },
                    "outline": {
                        "description": "回答的大纲",
                        "type": "array",
                        "items": {
                            "h2": {
                                "type": "string",
                                "description": "大纲的二级标题"
                            },
                            "guideline": {
                                "type": "string",
                                "description": "如何填充该二级标题的指南"
                            }
                        }
                    },
                    "search_result_index_list": {
                        "type": "array",
                        "items": {
                            "type": "string"
                        }
                    },
                    "literature_paper_search_result": {
                        "type": "array",
                        "items": {
                            "type": "object",
                            "properties": {
                                "title": {
                                    "type": "string"
                                },
                                "paper_url": {
                                    "type": "string",
                                    "description": "文献pdf/word URL"
                                },
                                "abstract": {
                                    "type": "string",
                                    "description": "论文检索摘要原文，不要对文献摘要修改"
                                },
                                "content": {
                                    "type": "string",
                                    "description": "文献与检索相关内容原文，不要对内容精简"
                                }
                            }
                        }
                    },
                    "literature_image_list": {
                        "type": "array",
                        "items": {
                            "type": "object",
                            "properties": {
                                "url": {
                                    "type": "string"
                                },
                                "image_description": {
                                    "type": "string"
                                }
                            }
                        }
                    }
                },
                "required": [
                    "query",
                    "title",
                    "style",
                    "outline",
                    "search_result_index_list",
                    "literature_paper_search_result"
                ]
            }
        }
    },
]
try:
    models = client.models.list()
    model = models.data[0].id
    stream = True
    response = client.chat.completions.create(
        model=model,
        messages=[
            {
                "role": "user",
                "content": "充分理解用户问题并创建一个plan。 \n\n用户问题：DeepSeek-R1-0528具体有哪些提升，请生成一份详细的报告\n\nplan ID：plan_1748576400\n\n参考资料：\n无"
            }
        ],
        stream=stream,
        max_tokens=4096,
        tool_choice="auto",
        tools=tools,
        extra_body={"chat_template_kwargs": {"enable_thinking": True}},
    )
    if not stream:
        print(response.choices[0].message.content)
        print(response.choices[0].message.tool_calls)
    else:
        full_response = ""
        tool_calls_list = []
        for chunk in response:
            if chunk.choices[0].delta:
                if chunk.choices[0].delta.content:
                    full_response += chunk.choices[0].delta.content
                if chunk.choices[0].delta.tool_calls:
                    tool_calls_list.append(chunk.choices[0].delta.tool_calls)
        if tool_calls_list:
            merged_tool_call = {}
            for tool_call_chunk in tool_calls_list:
                for tool_call in tool_call_chunk:
                    print(tool_call)
                    merged_tool_call.setdefault(tool_call.index, {})
                    merged_tool_call[tool_call.index].setdefault("name", tool_call.function.name)
                    merged_tool_call[tool_call.index].setdefault("arguments", "")

                    if tool_call.function.arguments is not None:
                        merged_tool_call[tool_call.index]["arguments"] = merged_tool_call[tool_call.index][
                                                                          "arguments"] + tool_call.function.arguments

            print("Merged Tool Call:", merged_tool_call)
        print(full_response)

except Exception as e:
    print(f"请求失败: {e}")

Client log

ChoiceDeltaToolCall(index=0, id='chatcmpl-tool-ca0a10ac646744be850c62b1301ff39b', function=ChoiceDeltaToolCallFunction(arguments=None, name='summary'), type='function')
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='{"query": "', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='Deep', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='Seek', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='-R', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='1', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='-', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='0', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='5', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='2', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='8', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='具体', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='有哪些', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='提升', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='",', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments=' "', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='title', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='":', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments=' "', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='Deep', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='Seek', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='-R', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='1', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='-', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='0', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='5', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='2', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='8', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='版本', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='升级', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='亮点', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='解析', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='",', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments=' "', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='style', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='":', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments=' "', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='研究报告', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='",', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments=' "', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='outline', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='":', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments=' [{"', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='h', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='2', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='":', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments=' "', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='一', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='、', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='版本', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='迭代', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='背景', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='",', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments=' "', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='guid', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='eline', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='":', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments=' "', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='阐述', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='Deep', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='Seek', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='-R', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='1', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='系列', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='研发', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='历程', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='及', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='0', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='5', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='2', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='8', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='版本', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='开发', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='目标', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='"},', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments=' {"', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='h', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='2', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='":', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments=' "', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='二', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='、', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='核心技术', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='突破', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='",', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments=' "', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='guid', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='eline', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='":', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments=' "', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='对比', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='分析', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='模型', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='架构', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='优化', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='、', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='训练', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='数据', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='升级', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='、', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='算法', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='改进', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='等', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='核心', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='提升', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='点', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='"},', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments=' {"', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='h', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='2', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='":', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments=' "', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='三', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='、', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='性能', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='指标', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='提升', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='",', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments=' "', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='guid', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='eline', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='":', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments=' "', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='从', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='推理', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='速度', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='、', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='准确', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='率', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='、', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='多', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='语言', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='支持', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='度', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='等', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='维度', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='量化', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='展示', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='提升', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='幅度', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='"},', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments=' {"', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='h', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='2', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='":', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments=' "', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='四', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='、', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='应用场景', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='拓展', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='",', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments=' "', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='guid', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='eline', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='":', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments=' "', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='说明', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='在', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='代码', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='生成', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='、', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='内容', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='创作', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='、', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='逻辑', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='推理', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='等', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='场景', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='的', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='专项', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='优化', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='"},', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments=' {"', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='h', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='2', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='":', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments=' "', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='五', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='、', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='用户', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='使用', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='体验', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='",', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments=' "', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='guid', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='eline', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='":', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments=' "', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='描述', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='实际', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='应用', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='中的', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='响应', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='速度', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='、', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='交互', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='流畅', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='度', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='等', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='体验', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='改善', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='"', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments=']', name=None), type=None)
ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='}', name=None), type=None)
Merged Tool Call: {'chatcmpl-tool-ca0a10ac646744be850c62b1301ff39b': {'name': 'summary', 'arguments': ''}, None: {'name': None, 'arguments': '{"query": "DeepSeek-R1-0528具体有哪些提升", "title": "DeepSeek-R1-0528版本升级亮点解析", "style": "研究报告", "outline": [{"h2": "一、版本迭代背景", "guideline": "阐述DeepSeek-R1系列研发历程及0528版本开发目标"}, {"h2": "二、核心技术突破", "guideline": "对比分析模型架构优化、训练数据升级、算法改进等核心提升点"}, {"h2": "三、性能指标提升", "guideline": "从推理速度、准确率、多语言支持度等维度量化展示提升幅度"}, {"h2": "四、应用场景拓展", "guideline": "说明在代码生成、内容创作、逻辑推理等场景的专项优化"}, {"h2": "五、用户使用体验", "guideline": "描述实际应用中的响应速度、交互流畅度等体验改善"]}'}}

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[chaunceyjiang]

Can you test this PR #17506 to see if it resolves your issue?

[LiuLi1998]

Can you test this PR #17506 to see if it resolves your issue?

Thanks, I think when i set tool_choice="auto", tool_choice_function_name will be none, so the code will not implement the logic of this PR.

[LiuLi1998]

Can you test this PR #17506 to see if it resolves your issue?

My start cmd:
vllm
serve
/model
--host 0.0.0.0
--port "$SERVICE_PORT"
--gpu-memory-utilization 0.9
-tp "$gpu_count"
--block-size 256
--enable-chunked-prefill
--max_num_batched_tokens 4096
--max-model-len 131072
--max_seq_len_to_capture 131072
--enable-auto-tool-choice
--tool-call-parser hermes

[chaunceyjiang]

Could you wrap your client code with ``` ``` so I can try to reproduce the issue locally?

[LiuLi1998]

Could you wrap your client code with so I can try to reproduce the issue locally?

I have updated this issue

[vladrad]

Can you test this PR #17506 to see if it resolves your issue?

I tested this with thinking on/off and no go.

[LiuLi1998]

Can you test this PR #17506 to see if it resolves your issue?

I tested this with thinking on/off and no go.

you can test #19058, works for me