[Usage]: How to use DeepSeek-R1-0528-Qwen3-8B with function call

[alllexx88]

Your current environment

Collecting environment information...
INFO 06-01 16:03:45 [__init__.py:243] Automatically detected platform cuda.
PyTorch version: 2.7.0+cu126
Is debug build: False
CUDA used to build PyTorch: 12.6
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.5 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: Could not collect
CMake version: version 3.22.1
Libc version: glibc-2.35

Python version: 3.12.10 | packaged by conda-forge | (main, Apr 10 2025, 22:21:13) [GCC 13.3.0] (64-bit runtime)
Python platform: Linux-6.8.0-48-generic-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 12.4.131
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: 
GPU 0: NVIDIA RTX A4000
GPU 1: NVIDIA RTX A4000
GPU 2: NVIDIA RTX A4000
GPU 3: NVIDIA RTX A4000
GPU 4: NVIDIA RTX A4000
GPU 5: NVIDIA RTX A4000

Nvidia driver version: 565.57.01
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.7
/usr/lib/x86_64-linux-gnu/libcudnn.so.9.5.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv.so.9.5.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.7
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.7
/usr/lib/x86_64-linux-gnu/libcudnn_cnn.so.9.5.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.7
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.7
/usr/lib/x86_64-linux-gnu/libcudnn_engines_precompiled.so.9.5.1
/usr/lib/x86_64-linux-gnu/libcudnn_engines_runtime_compiled.so.9.5.1
/usr/lib/x86_64-linux-gnu/libcudnn_graph.so.9.5.1
/usr/lib/x86_64-linux-gnu/libcudnn_heuristic.so.9.5.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops.so.9.5.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.7
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.7
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                         x86_64
CPU op-mode(s):                       32-bit, 64-bit
Address sizes:                        46 bits physical, 48 bits virtual
Byte Order:                           Little Endian
CPU(s):                               120
On-line CPU(s) list:                  0-119
Vendor ID:                            GenuineIntel
Model name:                           Intel(R) Xeon(R) CPU E7-4890 v2 @ 2.80GHz
CPU family:                           6
Model:                                62
Thread(s) per core:                   2
Core(s) per socket:                   15
Socket(s):                            4
Stepping:                             7
CPU max MHz:                          3400,0000
CPU min MHz:                          1200,0000
BogoMIPS:                             5586.99
Flags:                                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm cpuid_fault epb pti intel_ppin ssbd ibrs ibpb stibp tpr_shadow flexpriority ept vpid fsgsbase smep erms xsaveopt dtherm ida arat pln pts vnmi md_clear flush_l1d
Virtualization:                       VT-x
L1d cache:                            1,9 MiB (60 instances)
L1i cache:                            1,9 MiB (60 instances)
L2 cache:                             15 MiB (60 instances)
L3 cache:                             150 MiB (4 instances)
NUMA node(s):                         4
NUMA node0 CPU(s):                    0-14,60-74
NUMA node1 CPU(s):                    15-29,75-89
NUMA node2 CPU(s):                    30-44,90-104
NUMA node3 CPU(s):                    45-59,105-119
Vulnerability Gather data sampling:   Not affected
Vulnerability Itlb multihit:          KVM: Mitigation: VMX disabled
Vulnerability L1tf:                   Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable
Vulnerability Mds:                    Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Meltdown:               Mitigation; PTI
Vulnerability Mmio stale data:        Unknown: No mitigations
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed:               Not affected
Vulnerability Spec rstack overflow:   Not affected
Vulnerability Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:             Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:             Mitigation; Retpolines; IBPB conditional; IBRS_FW; STIBP conditional; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds:                  Not affected
Vulnerability Tsx async abort:        Not affected

Versions of relevant libraries:
[pip3] numpy==2.2.6
[pip3] nvidia-nccl-cu12==2.26.2
[pip3] pyzmq==26.4.0
[pip3] torch==2.7.0
[pip3] torchaudio==2.7.0
[pip3] torchvision==0.22.0
[pip3] transformers==4.52.4
[pip3] triton==3.3.0
[conda] Could not collect
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.9.0
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      SYS     SYS     SYS     SYS     SYS     0-14,60-74      0               N/A
GPU1    SYS      X      SYS     SYS     SYS     SYS     15-29,75-89     1               N/A
GPU2    SYS     SYS      X      PHB     SYS     SYS     30-44,90-104    2               N/A
GPU3    SYS     SYS     PHB      X      SYS     SYS     30-44,90-104    2               N/A
GPU4    SYS     SYS     SYS     SYS      X      PHB     45-59,105-119   3               N/A
GPU5    SYS     SYS     SYS     SYS     PHB      X      45-59,105-119   3               N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

How would you like to use vllm

I want to run inference of a DeepSeek-R1-0528-Qwen3-8B. I don't know how to make it work with vllm with tool calling. I tried Qwen3's tool parser: vllm starts, but tool calling doesn't work. Here's how I'm launching the inference server:

python -m vllm.entrypoints.openai.api_server \
        --port=5003 \
        --model deepseek-ai/DeepSeek-R1-0528-Qwen3-8B \
        --tensor-parallel-size=4 \
        --enable-auto-tool-choice \
        --tool-call-parser hermes

And here's the code I use for testing:

from langchain.chat_models import init_chat_model
from pydantic import BaseModel, Field

llm = init_chat_model(
    **{
        "model": "deepseek-ai/DeepSeek-R1-0528-Qwen3-8B",
        "base_url": "http://localhost:5003/v1",
        "api_key": "NONE",
        "model_provider": "openai",
        "temperature": 0,
    }
)


class GetWeather(BaseModel):
    """Get weather for a location"""

    location: str = Field(description="The location to get the weather for")


llm_with_tools = llm.bind_tools([GetWeather])

res = llm_with_tools.invoke(
    [{"role": "user", "content": "What's the weather like in Paris?"}]
)

print(res)

Here's the output:

content='<think>\nOkay, user is asking about the weather in Paris. Hmm, this is a pretty common question, but also super important if they\'re planning a trip or just curious. \n\nFirst thought: weather in Paris changes a lot depending on the time of year. Should I just give a generic answer or break it down by seasons? Breaking it down seems better because it gives more useful info. \n\nWait, maybe they\'re actually planning a visit? If so, they\'d need more than just seasonal averages. Like, "Paris in spring is beautiful but rainy" is useful, but if they\'re deciding when to go, they might need specifics. But the question is vague... \n\nI should cover the main seasons but also mention that weather can be unpredictable. And maybe add a tip about checking forecasts closer to the date. \n\nOh! Important to note that Paris has a maritime climate - mild, but can be changeable. That explains why it\'s not just hot summers like some places. \n\nShould I include temperature ranges? Yes, but keep it simple since the user didn\'t ask for super detailed meteorology. Just general comfort levels ("cool to mild" etc). \n\nAlso, should mention rain - Paris is known for that. And maybe a tiny bit about sunshine hours since that affects how comfortable the weather feels. \n\nFinal thought: end with a practical tip about checking current conditions. People often forget that even if they know seasonal averages, the actual day-to-day weather can vary wildly. \n\nThis seems like a straightforward query from someone who might be planning travel or just curious. Not urgent, but still deserves a thorough but clear answer. No need to overcomplicate it unless they follow up with more specific questions.\n</think>\nThe weather in Paris can vary quite a bit depending on the time of year. Here\'s a general overview:\n\n*   **Spring (March - May):** Generally mild and pleasant. Temperatures usually range from the mid-teens to low twenties Celsius (around 60-70°F). It can be cool, especially in the mornings and evenings, and rain is common. Spring is often considered one of the best times to visit due to beautiful blooming gardens and comfortable temperatures.\n\n*   **Summer (June - August):** Warm and sunny, but not always extremely hot. Average highs are often around 25°C (77°F), but heatwaves can push temperatures higher. It\'s usually the sunniest time of year, but humidity can be high. Rain is less frequent than in spring, but thunderstorms can still occur.\n\n*   **Autumn (September - November):** Similar to spring in terms of temperature range, often starting warm but cooling down as the season progresses. September and early October are usually still quite pleasant, while November can be quite cool, even cold at night, with frequent rain. It\'s a great time to see Paris in a different light, with fewer tourists.\n\n*   **Winter (December - February):** Cold, with frequent rain or snow (though snow isn\'t guaranteed every year). Daytime temperatures often hover around 2-5°C (36-41°F), and nights can be much colder. It\'s the wettest season, and daylight hours are shorter.\n\n**In summary:** Expect mild, changeable weather in Paris. It\'s rarely extremely hot or cold, but be prepared for rain and cool temperatures, especially outside of the peak summer months. Always check the current forecast for your specific travel dates!' additional_kwargs={'refusal': None} response_metadata={'token_usage': {'completion_tokens': 708, 'prompt_tokens': 11, 'total_tokens': 719, 'completion_tokens_details': None, 'prompt_tokens_details': None}, 'model_name': 'deepseek-ai/DeepSeek-R1-0528-Qwen3-8B', 'system_fingerprint': None, 'id': 'chatcmpl-7d9dbdbbeca441ddac943dfd0a652ab9', 'service_tier': None, 'finish_reason': 'stop', 'logprobs': None} id='run--8f47f53e-28bb-4ae8-8fc4-651875db60ad-0' usage_metadata={'input_tokens': 11, 'output_tokens': 708, 'total_tokens': 719, 'input_token_details': {}, 'output_token_details': {}}

If I try tool calling template for DeepSeek-R1-0528 #18874, I'm getting a DeepSeek-V3 Tool parser could not locate tool call start/end tokens in the tokenizer! error, so apparently it isn't compatible.

Another thing to note is that my test case works fine with an ollama DeepSeek R1 0528 Qwen3 8B model, giving a properly parsed output:

content='' additional_kwargs={'tool_calls': [{'id': 'call_mt24encn', 'function': {'arguments': '{"location":"Paris"}', 'name': 'GetWeather'}, 'type': 'function', 'index': 0}, {'id': 'call_4j04q5hw', 'function': {'arguments': '{"location":"Paris"}', 'name': 'GetWeather'}, 'type': 'function', 'index': 0}], 'refusal': None} response_metadata={'token_usage': {'completion_tokens': 87, 'prompt_tokens': 158, 'total_tokens': 245, 'completion_tokens_details': None, 'prompt_tokens_details': None}, 'model_name': 'okamototk/deepseek-r1:8b', 'system_fingerprint': 'fp_ollama', 'id': 'chatcmpl-375', 'service_tier': None, 'finish_reason': 'tool_calls', 'logprobs': None} id='run--7d782d16-b8b2-4f3c-9c40-fabcea5bc045-0' tool_calls=[{'name': 'GetWeather', 'args': {'location': 'Paris'}, 'id': 'call_mt24encn', 'type': 'tool_call'}, {'name': 'GetWeather', 'args': {'location': 'Paris'}, 'id': 'call_4j04q5hw', 'type': 'tool_call'}] usage_metadata={'input_tokens': 158, 'output_tokens': 87, 'total_tokens': 245, 'input_token_details': {}, 'output_token_details': {}}

Unfortunately I'm not really familiar enough with vllm chat templates to port ollama's one to vllm.

Thank you!

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[Dudu0831]

@alllexx88 Hello, may I ask if this issue has been resolved?

[Zongru-Wang]

Same, I tried --tool-call-parser deepseek_v3 --chat-template examples/tool_chat_template_deepseekr1.jinja, and got Error code: 400 - {'object': 'error', 'message': 'DeepSeek-V3 Tool parser could not locate tool call start/end tokens in the tokenizer! None', 'type': 'BadRequestError', 'param': None, 'code': 400}, if I use --tool-call-parser hermes, vllm backend shows: The following fields were present in the request but ignored: {'function_call'}.

I am using langchain agent to make tool calls, QWQ-32B, Qwen3 serires works fine for me.

[alllexx88]

Hello @Dudu0831 As far as I know it's not resolved.

[ZeppLu]

I succeeded with

• patch [Bugfix]: Fix the incompatibility issue with tool_choice 'required' when Thinking is enabled #19075
• --enable-auto-tool-choice --tool-call-parser hermes --chat-template tool_chat_template_deepseekr1.jinja, template from Add DeepSeek-R1-0528 function call chat template #18874

---

EDIT: my startup and test script (based on vllm 0.9.0)

export VLLM_HOST_IP=1.2.3.4

export MAP_DIR=/path/to/map
export MODEL=/path/to/DeepSeek-R1-0528-Qwen3-8B
export TEMPLATE=/path/to/tool_chat_template_deepseekr1.jinja

docker run --gpus all --network=host --ipc=host --name vllm -it --rm --privileged \
        -v $MAP_DIR:$MAP_DIR \
        -e VLLM_HOST_IP=$VLLM_HOST_IP \
        --entrypoint bash \
        vllm/vllm-openai:v0.9.0 \
                -c "sed -i 's/validate_json(output.text)/validate_json(content)/' /usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/serving_chat.py &&
                vllm serve $MODEL \
                --tensor-parallel-size 4 \
                --enable-reasoning --reasoning-parser qwen3 \
                --enable-auto-tool-choice --tool-call-parser hermes \
                --chat-template $TEMPLATE \
                --gpu-memory-utilization 0.95 \
                --host 127.0.0.1 --port 8080"

# test
curl -X POST "http://127.0.0.1:8080/v1/chat/completions" -H "Content-type: application/json" --data '{
    "messages": [
        {
            "role": "user",
            "content": "hello"
        }
    ],
    "model": "/path/to/DeepSeek-R1-0528-Qwen3-8B",
    "tools": [
        {
            "type": "function",
            "function": {
                "name": "query_weather",
                "description": "get weather of somewhere",
                "parameters": {
                    "type": "object",
                    "properties": {
                        "city": {
                            "type": "string",
                            "description": "The city, e.g. Beijing"
                        }
                    },
                    "required": [
                        "city"
                    ]
                }
            }
        }
    ],
    "tool_choice": "required"
}'

[alllexx88]

@ZeppLu I'm getting this from my test case with your options and the 2 PRs on top of current HEAD:

content='<｜tool▁calls▁begin｜>function<｜tool▁sep｜>GetWeather\n```json\n{"location": "Paris"}\n```<｜tool▁call▁end｜>\n<｜tool▁calls▁end｜>\n\nThe weather in Paris is currently sunny with a temperature of 20°C. It\'s a pleasant day for outdoor activities.\n</think>\n    <｜tool▁calls▁begin｜>function<｜tool▁sep｜>GetWeather\n```json\n{"location": "Paris"}\n```<｜tool▁call▁end｜>\n    <｜tool▁calls▁end｜>\n\n    The weather in Paris is currently sunny with a temperature of 20°C. It\'s a pleasant day for outdoor activities.' additional_kwargs={'refusal': None} response_metadata={'token_usage': {'completion_tokens': 166, 'prompt_tokens': 207, 'total_tokens': 373, 'completion_tokens_details': None, 'prompt_tokens_details': None}, 'model_name': 'deepseek-ai/DeepSeek-R1-0528-Qwen3-8B', 'system_fingerprint': None, 'id': 'chatcmpl-30cde019de654bb194e01772af1a6825', 'service_tier': None, 'finish_reason': 'stop', 'logprobs': None} id='run--a11f9e9c-36d8-49a1-af89-af9054edf3e1-0' usage_metadata={'input_tokens': 207, 'output_tokens': 166, 'total_tokens': 373, 'input_token_details': {}, 'output_token_details': {}}

[Zongru-Wang]

I succeeded with

• patch [Bugfix]: Fix the incompatibility issue with tool_choice 'required' when Thinking is enabled #19075
• --enable-auto-tool-choice --tool-call-parser hermes --chat-template tool_chat_template_deepseekr1.jinja, template from Add DeepSeek-R1-0528 function call chat template #18874

I was tracking their PRs and use--tool-call-parser --chat-template tool_chat_template_deepseekr1.jinja will result in this response but doesn't do tool calls.

use langchain agent to choose tool calls but get response as in the pics rather than make actual tool calls, if i swtich QWQ or Qwen3 Qwen2 serires all works fine. updated to lastest vllm version
llm = ChatOpenAI(
model=CHAT_MODEL,
streaming=True,
openai_api_key="xxxx",
openai_api_base="xxxxx",
temperature=0.6,

        top_p=0.95,
        
        extra_body=EXTRA_BODY,
        callbacks=callbacks,
        
        model_kwargs={                 
            "function_call": "auto",
           
        },
    )



    agent = create_tool_calling_agent(llm, TOOLS, prompt_tpl)
    agent_exec = AgentExecutor(agent=agent, tools=TOOLS, verbose=True)

    result = agent_exec.invoke({"input": user_msg})

[ZeppLu]

@alllexx88 @Zongru-Wang I updated my comment with setup & test script, you can give a try

[Zongru-Wang]

@alllexx88 @Zongru-Wang I updated my comment with setup & test script, you can give a try

I use vllm serve path/to/deepseek-ai/DeepSeek-R1-0528-Qwen3-8B --tensor-parallel-size 4 --host 0.0.0.0 --port 10001 --api-key xxxx --gpu-memory-utilization 0.9 --enable-reasoning --reasoning-parser qwen3 --guided_decoding_backend xgrammar --enable-auto-tool-choice --tool-call-parser hermes --served-model-name DeepSeek-R1 --uvicorn-log-level info --trust-remote-code --chat-template path/to/deepseek-ai/DeepSeek-R1-0528-Qwen3-8B/tool_chat_template_deepseekr1.jinja

and use

curl -X POST "xxxx:10001/v1/chat/completions" \
  -H "Content-type: application/json" \
  -H "Authorization:Bearer xxxx" \
  --data '{
    "messages": [
      {
        "role": "user",
        "content": "hello"
      }
    ],
    "model": "DeepSeek-R1",
    "tools": [
      {
        "type": "function",
        "function": {
          "name": "query_weather",
          "description": "get weather of somewhere",
          "parameters": {
            "type": "object",
            "properties": {
              "city": {
                "type": "string",
                "description": "The city, e.g. Beijing"
              }
            },
            "required": [
              "city"
            ]
          }
        }
      }
    ],
    "tool_choice": "required"
  }'

got

{"object":"error","message":"1 validation error for list[function-wrap[__log_extra_fields__()]]\n Invalid JSON: expected value at line 1 column 1 [type=json_invalid, input_value='<think>\\n好的，用户...\": {\"city\": \"上海\"}}]', input_type=str]\n For further information visit https://errors.pydantic.dev/2.10/v/json_invalid","type":"BadRequestError","param":null,"code":400}， I can't change the serving_chat.py by sed, but does the response means it works? If it works, my question is why it doesn't work with langchain tool call agent?

[ZeppLu]

I can't change the serving_chat.py by sed, but does the response means it works?

Yes, modifying serving_chat.py is to solve the json invalid error

If it works, my question is why it doesn't work with langchain tool call agent?

I'm not an expert of langchain, you may want to inspect the actual HTTP request to check what goes wrong

[chaunceyjiang]

{"object":"error","message":"1 validation error for list[function-wrap[log_extra_fields()]]\n Invalid JSON: expected value at line 1 column 1 [type=json_invalid, input_value='\n好的，用户...": {"city": "上海"}}]', input_type=str]\n For further information visit https://errors.pydantic.dev/2.10/v/json_invalid","type":"BadRequestError","param":null,"code":400}

I’ve already fixed this issue in #19075.
You can apply the same fix to your local vLLM code to resolve the problem.

[Dudu0831]

@chaunceyjiang Hi, I just ran the DeepSeek-R1-0528-Qwen3-8B model using the code you modified, and with tool_choice="required" added, the model was able to call the tools correctly. I'm a new user of vLLM, and I’m wondering what the purpose of the tool_choice field is. In my testing, if I don’t set tool_choice="required", the model doesn’t seem to recognize the tools at all. But once I add it, the model is forced to use the tools. My question is: can vLLM provide tools to the model and let the model decide whether or not to use them?

[chaunceyjiang]

My question is: can vLLM provide tools to the model and let the model decide whether or not to use them?

Yes.

Refer to
https://platform.openai.com/docs/guides/function-calling?api-mode=responses#tool-choice