[Bug]: Gemma4 + MTP speculative decoding drops first tool-call arguments in streaming multi-tool auto-tool-choice

[paolo-losi]

The output of python collect_env.py

Collecting environment information...
==============================
        System Info
==============================
OS                           : Ubuntu 22.04.5 LTS (x86_64)
GCC version                  : (Ubuntu 11.4.0-1ubuntu1~22.04.3) 11.4.0
Clang version                : Could not collect
CMake version                : Could not collect
Libc version                 : glibc-2.35

==============================
       PyTorch Info
==============================
PyTorch version              : 2.11.0+cu130
Is debug build               : False
CUDA used to build PyTorch   : 13.0
ROCM used to build PyTorch   : N/A
XPU used to build PyTorch    : N/A

==============================
      Python Environment
==============================
Python version               : 3.12.13 (main, Mar  4 2026, 09:23:07) [GCC 11.4.0] (64-bit runtime)
Python platform              : Linux-6.17.0-1013-aws-x86_64-with-glibc2.35
    
==============================
       CUDA / GPU Info
==============================
Is CUDA available            : True
CUDA runtime version         : 13.0.88
CUDA_MODULE_LOADING set to   : 
GPU models and configuration : GPU 0: NVIDIA RTX PRO 6000 Blackwell Server Edition
Nvidia driver version        : 595.58.03
cuDNN version                : Could not collect
HIP runtime version          : N/A
MIOpen runtime version       : N/A
Is XNNPACK available         : True

==============================
          CPU Info
==============================
Architecture:                            x86_64
CPU op-mode(s):                          32-bit, 64-bit
Address sizes:                           46 bits physical, 48 bits virtual
Byte Order:                              Little Endian
CPU(s):                                  8
On-line CPU(s) list:                     0-7
Vendor ID:                               GenuineIntel
Model name:                              Intel(R) Xeon(R) Platinum 8559C
CPU family:                              6
Model:                                   207
Thread(s) per core:                      2
Core(s) per socket:                      4
Socket(s):                               1
Stepping:                                2
BogoMIPS:                                4800.00
Hypervisor vendor:                       KVM
Virtualization type:                     full

==============================
Versions of relevant libraries
==============================
[pip3] flashinfer-python==0.6.8.post1
[pip3] numpy==2.2.6
[pip3] nvidia-cublas==13.1.0.3
[pip3] nvidia-cuda-cupti==13.0.85
[pip3] nvidia-cuda-nvrtc==13.0.88
[pip3] nvidia-cuda-runtime==13.0.96
[pip3] nvidia-cudnn-cu13==9.19.0.56
[pip3] nvidia-cudnn-frontend==1.18.0
[pip3] nvidia-cufft==12.0.0.61
[pip3] nvidia-cufile==1.15.1.6
[pip3] nvidia-curand==10.4.0.35
[pip3] nvidia-cusolver==12.0.4.66
[pip3] nvidia-cusparse==12.6.3.3
[pip3] nvidia-cusparselt-cu13==0.8.0
[pip3] nvidia-cutlass-dsl==4.4.2
[pip3] nvidia-cutlass-dsl-libs-base==4.4.2
[pip3] nvidia-ml-py==13.595.45
[pip3] nvidia-nccl-cu13==2.28.9
[pip3] nvidia-nvjitlink==13.0.88
[pip3] nvidia-nvshmem-cu13==3.4.5
[pip3] nvidia-nvtx==13.0.85
[pip3] pyzmq==27.1.0
[pip3] torch==2.11.0+cu130
[pip3] torch_c_dlpack_ext==0.1.5
[pip3] torchaudio==2.11.0+cu130
[pip3] torchvision==0.26.0+cu130
[pip3] transformers==5.8.0
[pip3] triton==3.6.0
[conda] Could not collect

==============================
         vLLM Info
==============================
ROCM Version                 : Could not collect
vLLM Version                 : 0.20.2rc1.dev49+g9b4e83934 (git sha: 9b4e83934)
vLLM Build Flags:
  CUDA Archs: 7.5 8.0 8.6 8.9 9.0 10.0 12.0+PTX; ROCm: Disabled; XPU: Disabled

==============================
     Environment Variables
==============================
CUDA_VERSION=13.0.2
NVIDIA_VISIBLE_DEVICES=all
NVIDIA_DRIVER_CAPABILITIES=compute,utility
VLLM_ENABLE_CUDA_COMPATIBILITY=0
TORCH_CUDA_ARCH_LIST=7.5 8.0 8.6 8.9 9.0 10.0 12.0+PTX
VLLM_USAGE_SOURCE=production-docker-image
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_root

🐛 Describe the bug

I hit a streaming tool-calling corruption bug with Gemma4 when MTP speculative decoding is enabled.

Setup:

• Docker image: vllm/vllm-openai:gemma4-0505-cu130
• vLLM version reported by the server: 0.20.2rc1.dev49+g9b4e83934
• model: google/gemma-4-31B-it
• server args include:
  • --enable-auto-tool-choice
  • --tool-call-parser gemma4
  • --reasoning-parser gemma4
  • --speculative-config '{"model": "google/gemma-4-31B-it-assistant", "num_speculative_tokens": 4}'

Observed behavior:

• with stream=true, when the model emits two tool calls in one turn, the first tool call's arguments are corrupted / dropped in the streamed deltas
• with stream=false, the same request returns the correct final two tool calls
• if I restart the same server without --speculative-config, the streaming output becomes correct across repeated runs

So this appears to be a speculative-decoding + streaming + multi-tool-call interaction, not a base-model issue.

I searched and found related issues, but this one seems distinct:

• [Bug]: tool_choice="required" + speculative decoding with lukealonso/Qwen3.5-397B-A17B-NVFP4 leads to failed tool calls. #38106 (tool_choice=required + speculative decoding + failed tool calls)
• [Bug]: ngram speculative decoding default prompt_lookup_min=2 causes tool-call output corruption on Qwen3-class models with structured output (config-only fix: prompt_lookup_min=8) #40875 (ngram speculative decoding corrupting structured outputs / tool calls)
• [Bug]: hermes_tool_parser drops final JSON brace during streaming when tokens are chunked (e.g., MTP enabled) #32152 (streaming tool parser dropping final JSON brace under chunking)

Minimal repro

Server launch (the important part is the speculative config + gemma4 parser):

vllm serve google/gemma-4-31B-it \
  --tensor-parallel-size 1 \
  --served-model-name gemma-4-31B-it \
  --host 0.0.0.0 \
  --max-model-len 185000 \
  --max-num-batched-tokens 8192 \
  --gpu-memory-utilization 0.95 \
  --limit-mm-per-prompt '{"image": 0}' \
  --enable-prefix-caching \
  --enable-prompt-tokens-details \
  --enable-auto-tool-choice \
  --tool-call-parser gemma4 \
  --reasoning-parser gemma4 \
  --default-chat-template-kwargs '{"enable_thinking": false}' \
  --chat-template examples/tool_chat_template_gemma4.jinja \
  --speculative-config '{"model": "google/gemma-4-31B-it-assistant", "num_speculative_tokens": 4}'

Repro request:

curl -N http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer <REDACTED>' \
  -d '{
    "model": "gemma-4-31B-it",
    "temperature": 0,
    "stream": true,
    "messages": [
      {
        "role": "user",
        "content": "Cerco i treni disponibili da Milano a Piacenza per oggi"
      }
    ],
    "tools": [
      {
        "type": "function",
        "function": {
          "name": "getStationInfo",
          "description": "Get station info for a city",
          "parameters": {
            "type": "object",
            "properties": {
              "location": {"type": "string"}
            },
            "required": ["location"]
          }
        }
      }
    ]
  }'

Broken stream=true output with speculative decoding enabled

I reproduced this repeatedly. Example streamed deltas:

[
  {
    "id": "chatcmpl-tool-95240dbe1f928781",
    "type": "function",
    "index": 0,
    "function": {"name": "getStationInfo", "arguments": ""}
  },
  {
    "id": "chatcmpl-tool-93dace5003b7bfb3",
    "type": "function",
    "index": 1,
    "function": {"name": "getStationInfo", "arguments": ""}
  },
  {
    "index": 1,
    "function": {"arguments": "{\"location\": \"Piacenza\"}"}
  }
]

The first call (Milano) never receives any argument delta.

Correct stream=false output for the same request

[
  {
    "id": "chatcmpl-tool-a05437a71d9cce91",
    "type": "function",
    "function": {
      "name": "getStationInfo",
      "arguments": "{\"location\": \"Milano\"}"
    }
  },
  {
    "id": "chatcmpl-tool-a534c6879f562a1b",
    "type": "function",
    "function": {
      "name": "getStationInfo",
      "arguments": "{\"location\": \"Piacenza\"}"
    }
  }
]

Strong signal that speculative decoding is the trigger

I restarted the same server with the same args except without --speculative-config and reran the same streaming request 5 times.

In all 5 runs, both tool calls streamed correctly, e.g.:

[
  {"id": "chatcmpl-tool-86d52864723d5ec0", "type": "function", "index": 0, "function": {"name": "getStationInfo", "arguments": ""}},
  {"index": 0, "function": {"arguments": "{\"location\": "}},
  {"index": 0, "function": {"arguments": "\"Mil"}},
  {"index": 0, "function": {"arguments": "ano"}},
  {"index": 0, "function": {"arguments": "\"}"}},
  {"id": "chatcmpl-tool-a85ab10d70df516f", "type": "function", "index": 1, "function": {"name": "getStationInfo", "arguments": ""}},
  {"index": 1, "function": {"arguments": "{\"location\": "}},
  {"index": 1, "function": {"arguments": "\"P"}},
  {"index": 1, "function": {"arguments": "iac"}},
  {"index": 1, "function": {"arguments": "enza"}},
  {"index": 1, "function": {"arguments": "\"}"}}
]

Notes

• parallel_tool_calls=false is not a useful workaround here. In my testing it just reduces the result to a single tool call, and in streaming mode I only got the empty header delta.
• Single-tool streaming worked for me even with speculative decoding enabled. The corruption showed up when the model emitted multiple tool calls in one assistant turn.

If helpful, I can also provide the exact raw SSE chunks from both the broken (speculative enabled) and working (speculative disabled) runs.

[764942]

We are also facing the same issue. Any workarounds for this other than turning of MTP and streaming ?

[m4r1k]

Same here, any updates?

[Kimahriman]

It looks like the DiffusionGemma updates may have fixed this

[yasu-oh]

Thanks for looking into this, but I do not think #45163 fully resolves #41967.

I tested the Gemma4 tool parser again using the regression test code from #44741, and the multi-boundary streaming/MTP case still fails:

FAILED tests/tool_parsers/test_gemma4_tool_parser.py::TestStreamingExtraction::test_streaming_mtp_chunk_with_multiple_tool_boundaries
E       assert {0} == {0, 1}

In this case, a single speculative/MTP-sized streaming delta contains both the end of the first tool call and the start of the second one:

location:<|"|>Milano<|"|>}<tool_call|><|tool_call>call:getStationInfo{

The parser should emit tool call deltas for both index 0 and index 1 in that same delta, but it only emits index 0. As a result, the second tool-call boundary inside the same chunk is still not handled correctly.

So while #45163 may improve DiffusionGemma support and some large-delta parser behavior, the specific issue in #41967 — handling multiple tool-call boundaries within a single streaming/MTP chunk — still appears to be unresolved.

Since this result is based on the regression test added in #44741, I think #44741 is still needed, or at least this exact test case should be added before considering #41967 fixed.

[Kimahriman]

Using a stream interval of 12 to induce the behavior, I get the stream deltas

data: {"id":"chatcmpl-ab86c139e7381956","object":"chat.completion.chunk","created":1781279433,"model":"mlx-community/gemma-4-26B-A4B-it-OptiQ-4bit","choices":[{"index":0,"delta":{"role":"assistant","content":""},"logprobs":null,"finish_reason":null}],"prompt_token_ids":null,"prompt_text":null}

data: {"id":"chatcmpl-ab86c139e7381956","object":"chat.completion.chunk","created":1781279433,"model":"mlx-community/gemma-4-26B-A4B-it-OptiQ-4bit","choices":[{"index":0,"delta":{"tool_calls":[{"id":"chatcmpl-tool-840cae60d53e7ec6","type":"function","index":0,"function":{"name":"getStationInfo","arguments":""}}]},"logprobs":{"content":[{"token":"call","logprob":0.0,"bytes":[99,97,108,108],"top_logprobs":[]},{"token":":","logprob":0.0,"bytes":[58],"top_logprobs":[]},{"token":"get","logprob":-0.00003123234637314454,"bytes":[103,101,116],"top_logprobs":[]},{"token":"Station","logprob":-2.0265558760002023e-6,"bytes":[83,116,97,116,105,111,110],"top_logprobs":[]},{"token":"Info","logprob":0.0,"bytes":[73,110,102,111],"top_logprobs":[]},{"token":"{","logprob":0.0,"bytes":[123],"top_logprobs":[]},{"token":"location","logprob":0.0,"bytes":[108,111,99,97,116,105,111,110],"top_logprobs":[]},{"token":":","logprob":0.0,"bytes":[58],"top_logprobs":[]},{"token":"<|\"|>","logprob":0.0,"bytes":[60,124,34,124,62],"top_logprobs":[]},{"token":"Mil","logprob":-0.00002062299427052494,"bytes":[77,105,108],"top_logprobs":[]},{"token":"ano","logprob":-1.6689286894688848e-6,"bytes":[97,110,111],"top_logprobs":[]},{"token":"<|\"|>","logprob":-0.000010847986231965479,"bytes":[60,124,34,124,62],"top_logprobs":[]}]},"finish_reason":null,"token_ids":null}]}

data: {"id":"chatcmpl-ab86c139e7381956","object":"chat.completion.chunk","created":1781279433,"model":"mlx-community/gemma-4-26B-A4B-it-OptiQ-4bit","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"{\"location\": \"Milano\"}"}}]},"logprobs":{"content":[{"token":"}","logprob":0.0,"bytes":[125],"top_logprobs":[]},{"token":"<tool_call|>","logprob":0.0,"bytes":[60,116,111,111,108,95,99,97,108,108,124,62],"top_logprobs":[]},{"token":"<|tool_call>","logprob":-0.009809503331780434,"bytes":[60,124,116,111,111,108,95,99,97,108,108,62],"top_logprobs":[]},{"token":"call","logprob":-1.1920928244535389e-7,"bytes":[99,97,108,108],"top_logprobs":[]},{"token":":","logprob":0.0,"bytes":[58],"top_logprobs":[]},{"token":"get","logprob":0.0,"bytes":[103,101,116],"top_logprobs":[]},{"token":"Station","logprob":0.0,"bytes":[83,116,97,116,105,111,110],"top_logprobs":[]},{"token":"Info","logprob":0.0,"bytes":[73,110,102,111],"top_logprobs":[]},{"token":"{","logprob":0.0,"bytes":[123],"top_logprobs":[]},{"token":"location","logprob":0.0,"bytes":[108,111,99,97,116,105,111,110],"top_logprobs":[]},{"token":":","logprob":0.0,"bytes":[58],"top_logprobs":[]},{"token":"<|\"|>","logprob":0.0,"bytes":[60,124,34,124,62],"top_logprobs":[]}]},"finish_reason":null,"token_ids":null}]}

data: {"id":"chatcmpl-ab86c139e7381956","object":"chat.completion.chunk","created":1781279433,"model":"mlx-community/gemma-4-26B-A4B-it-OptiQ-4bit","choices":[{"index":0,"delta":{"tool_calls":[{"id":"chatcmpl-tool-a7a2be4066628630","type":"function","index":1,"function":{"name":"getStationInfo","arguments":"{\"location\": \"Piacenza\"}"}}]},"logprobs":{"content":[{"token":"P","logprob":0.0,"bytes":[80],"top_logprobs":[]},{"token":"iac","logprob":0.0,"bytes":[105,97,99],"top_logprobs":[]},{"token":"enza","logprob":-1.1920928244535389e-7,"bytes":[101,110,122,97],"top_logprobs":[]},{"token":"<|\"|>","logprob":-1.6689286894688848e-6,"bytes":[60,124,34,124,62],"top_logprobs":[]},{"token":"}","logprob":0.0,"bytes":[125],"top_logprobs":[]},{"token":"<tool_call|>","logprob":0.0,"bytes":[60,116,111,111,108,95,99,97,108,108,124,62],"top_logprobs":[]},{"token":"<|tool_response>","logprob":-1.1920928244535389e-7,"bytes":[60,124,116,111,111,108,95,114,101,115,112,111,110,115,101,62],"top_logprobs":[]}]},"finish_reason":"tool_calls","stop_reason":50,"token_ids":null}],"system_fingerprint":"vllm-0.22.1rc1.dev461+ga37b4a940.d20260612-2f2727df"}

data: [DONE]

The second delta has the first function name, the third delta has the first function args, and the fourth delta has the second function name and args. That might be different than your test expects, but is that still valid output?

[yasu-oh]

Thanks for checking this and sharing the actual stream output.

I agree with your interpretation that this looks valid from an API consumer perspective. The second tool call is eventually emitted in the following delta, and the reconstructed final tool calls appear to be correct.

My earlier comment was focused more narrowly on the parser-level edge case covered by the regression test from #44741: a single streaming/MTP-sized chunk containing multiple tool-call boundaries. In that test, the crossing chunk already includes the end of the first tool call and the start of the second one, but the parser delta only contains index 0.

That said, I agree the test expectation may be too strict if the intended contract allows the parser to buffer the second tool call and emit it in a later delta. I do not want to argue that the stream you showed is invalid.

My only concern is that before we consider #41967 fully fixed, it would be good to have a regression test covering this multi-boundary chunk case, either with the current #44741 expectation or with an adjusted expectation that allows delayed emission but still verifies that the second tool call is not dropped, duplicated, or corrupted.

So I think #44741, or an equivalent regression test, is still useful even if the exact expected delta shape needs to be updated.

[m4r1k]

Adding two things here: a runtime check of the fix, and a PR for the regression test @yasu-oh asked for.

On the runtime side, I ran the #41967 repro on google/gemma-4-31B-it (fp8, MTP draft google/gemma-4-31B-it-assistant, num_speculative_tokens=8, --tool-call-parser gemma4 --reasoning-parser gemma4) on a single RTX PRO 6000, on a build with the #45163 parser (eb28452b1). The multi-tool streaming case reconstructs correctly: both tool calls come back with their arguments, 5/5 runs. tool_choice="required" and named tool choice under MTP work too now, and no longer crash EngineCore.

On @yasu-oh's point, I reproduced it. A test mirroring #44741's crossing case, where one MTP-sized delta ends the first tool call and starts the second, does fail the strict "both indices in the single crossing delta" assertion against eb28452b1: the parser emits index 0 in the crossing delta and index 1's header in the next one. That matches @Kimahriman's trace. The reconstructed result is still correct, so it's delta placement, not a dropped, duplicated, or corrupted call.

For the test itself, I opened #45427 with a few cases that pass against eb28452b1

• test_streaming_mtp_multi_boundary_delta: one delta ends call 0 and starts call 1; checks the final reconstruction has both indices, each name once, args correct, nothing dropped/duplicated/corrupted. PASS.
• test_streaming_mtp_multi_boundary_split_delimiter: same, but the <tool_call|> boundary is split across chunks, so the delta-buffering path runs too. PASS.
• test_streaming_mtp_multi_boundary_same_delta: the strict same-delta expectation, kept as xfail(strict=True) to document that it isn't part of the current contract.

Full gemma4 parser suite: 55 passed, 1 xfailed against eb28452b1. #45427 overlaps with #44741, so happy to fold it into #44741 instead if that keeps it to one change.