Question about how to reproduce Qwen's result using vLLM

[haochengxi]

Hi authors, thanks for this great work! When I am trying to reproduce the results, I can successfully reproduce the GLM-9B and GPT-4o's result. But when benchmarking the Qwen models using vLLM, the performance is much lower than expected. I can only get 1% Overall Success Rate and 45.12% Overall Call Acc, which is much lower than your official number (40.10% and 58.32%)

My environments uses vllm==0.6.3, transformers==4.48.3, torch==2.4.0, and I am benchmarking Qwen2.5-72B-Instruct model.
To extend the model's context length to 131072, following there official repo, I manually add the "rope_scaling": { "factor": 4.0, "original_max_position_embeddings": 32768, "type": "yarn" } parameter to the models config.json, and store it under checkpoints/Qwen2.5-72B-Instruct-YaRN directory.

The vllm command I use is

vllm serve checkpoints/Qwen2.5-72B-Instruct-YaRN --tensor-parallel-size 4 --gpu-memory-utilization 0.8 \
    --max_model_len 131072 --trust-remote-code --port 4008 \
    --enable-auto-tool-choice --tool-call-parser hermes

My evaluation code is

CUDA_VISIBLE_DEVICES=0,1,2,3 python evaluation.py \
    --model_name checkpoints/Qwen2.5-72B-Instruct-YaRN \
    --proc_num 10 \
    --vllm_url http://0.0.0.0:4008/v1
```.

Could you share insights on how to achieve accurate results with the Qwen model? Additionally, it would be great if you could include the code from your repository for serving Qwen models using vLLM.

[lucenzhong]

Hi authors, thanks for this great work! When I am trying to reproduce the results, I can successfully reproduce the GLM-9B and GPT-4o's result. But when benchmarking the Qwen models using vLLM, the performance is much lower than expected. I can only get 1% Overall Success Rate and 45.12% Overall Call Acc, which is much lower than your official number (40.10% and 58.32%)

My environments uses vllm==0.6.3, transformers==4.48.3, torch==2.4.0, and I am benchmarking Qwen2.5-72B-Instruct model. To extend the model's context length to 131072, following there official repo, I manually add the "rope_scaling": { "factor": 4.0, "original_max_position_embeddings": 32768, "type": "yarn" } parameter to the models config.json, and store it under checkpoints/Qwen2.5-72B-Instruct-YaRN directory.

The vllm command I use is

vllm serve checkpoints/Qwen2.5-72B-Instruct-YaRN --tensor-parallel-size 4 --gpu-memory-utilization 0.8 \
    --max_model_len 131072 --trust-remote-code --port 4008 \
    --enable-auto-tool-choice --tool-call-parser hermes

My evaluation code is

CUDA_VISIBLE_DEVICES=0,1,2,3 python evaluation.py \
    --model_name checkpoints/Qwen2.5-72B-Instruct-YaRN \
    --proc_num 10 \
    --vllm_url http://0.0.0.0:4008/v1
```.

Could you share insights on how to achieve accurate results with the Qwen model? Additionally, it would be great if you could include the code from your repository for serving Qwen models using vLLM.

I think the performance gap is mainly due to a bug in vllm's tool calling support. I got the following message.content from vllm while it is supposed to decode as message.tool_calls.

{"name": "Get_Flight_Details", "arguments": {"token": "d7699_H4sIAAAAAAAA_1VR247aMBD9GvapcWI75LKSVVECW9iELLmxPFkQQghlMY29CuTrOyGo22pGM-ec8Yw89kGpi3zW9f2pKg9Kap8SlUKJcqMKlIsPfV9D2ArxqzqX-qaqdX_0_jaKsBNP3r4BiMIAux0GACWDhONEx7oGlj-vvxdXpck6Z0_VtkAbprmu20OZM_xANbOQaduvs_VyeZdyoZiJzDSM6cLy0x-9WLOXRTQNnTvbsWDcNL43kUHc3HxvZvd8JB_ZDjyJwzYlYRtlCQ7a5JhNo3Zt-9k89sc4TqvmukgdAjNw0q7lanL62eHUW8vuTNjeZ6kgFdJPghv0LLOOx9csKgfUA79fpchhFQQbuP1CYiMfq4lcMdvo4U6xeO44mdVTxbAxtK2-eGWYENdCmFJCn2RxKnJVifNrcWPxwh26jgbPXVyIqSUcBIta_wnUoqYGn_AlYIf-I_jv2KZDDb4mVyCsQDCNLzog4-Px97lxAAzoCLzkBvprwA93jsEIIsA3fGKhdIhe5l0cTZGfgLrldtecczuEtOPYINTs2gvexT03IVYcP4Ye-WwFqeX4tr2Ij8Mf8cY_OIkCAAA."}}
</tool_call>

{"name": "Get_Flight_Details", "arguments": {"token": "d7699_H4sIAAAAAAAA_1VR247aMBD9GvapcWI75LKSVVECW9iELLmxPFkQQghlMY29CuTrOyGo22pGM-ec8Yw89kGpi3zW9f2pKg9Kap8SlUKJcqMKlIsPfV9D2ArxqzqX-qaqdX_0_jaKsBNP3r4BiMIAux0GACWDhONEx7oGlj-vvxdXpck6Z0_VtkAbprmu20OZM_xANbOQaduvs_VyeZdyoZiJzDSM6cLy0x-9WLOXRTQNnTvbsWDcNL43kUHc3HxvZvd8JB_ZDjyJwzYlYRtlCQ7a5JhNo3Zt-9k89sc4TqvmukgdAjNw0q7lanL62eHUW8vuTNjeZ6kgFdJPghv0LLOOx9csKgfUA79fpchhFQQbuP1CYiMfq4lcMdvo4U6xeO44mdVTxbAxtK2-eGWYENdCmFJCn2RxKnJVifNrcWPxwh26jgbPXVyIqSUcBIta_wnUoqYGn_AlYIf-I_jv2KZDDb4mVyCsQDCNLzog4-Px97lxAAzoCLzkBvprwA93jsEIIsA3fGKhdIhe5l0cTZGfgLrldtecczuEtOPYINTs2gvexT03IVYcP4Ye-WwFqeX4tr2Ij8Mf8cY_OIkCAAA."}}
</tool_response>