Gemma-4 tool call parser leaks raw tokens (<|" and "|>) into streaming response instead of parsing to standard tool_calls JSON

[elseyu]

Your current environment

The output of python collect_env.py

==============================
        System Info
==============================
OS                           : Ubuntu 22.04.5 LTS (x86_64)
GCC version                  : (Ubuntu 11.4.0-1ubuntu1~22.04.3) 11.4.0
Clang version                : Could not collect
CMake version                : Could not collect
Libc version                 : glibc-2.35

==============================
       PyTorch Info
==============================
PyTorch version              : 2.11.0+cu130
Is debug build               : False
CUDA used to build PyTorch   : 13.0
ROCM used to build PyTorch   : N/A
XPU used to build PyTorch    : N/A

==============================
      Python Environment
==============================
Python version               : 3.12.13 (main, Mar  4 2026, 09:23:07) [GCC 11.4.0] (64-bit runtime)
Python platform              : Linux-6.8.0-1045-ibm-x86_64-with-glibc2.35
    
==============================
       CUDA / GPU Info
==============================
Is CUDA available            : True
CUDA runtime version         : 13.0.88
CUDA_MODULE_LOADING set to   : 
GPU models and configuration : 
GPU 0: NVIDIA H100 80GB HBM3
GPU 1: NVIDIA H100 80GB HBM3
GPU 2: NVIDIA H100 80GB HBM3
GPU 3: NVIDIA H100 80GB HBM3

Nvidia driver version        : 580.126.20
cuDNN version                : Could not collect
HIP runtime version          : N/A
MIOpen runtime version       : N/A
Is XNNPACK available         : True

==============================
          CPU Info
==============================
Architecture:                            x86_64
CPU op-mode(s):                          32-bit, 64-bit
Address sizes:                           46 bits physical, 57 bits virtual
Byte Order:                              Little Endian
CPU(s):                                  160
On-line CPU(s) list:                     0-159
Vendor ID:                               GenuineIntel
Model name:                              Intel Xeon Processor (SapphireRapids)
CPU family:                              6
Model:                                   143
Thread(s) per core:                      2
Core(s) per socket:                      40
Socket(s):                               2
Stepping:                                4
BogoMIPS:                                4200.00
Flags:                                   fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology cpuid tsc_known_freq pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch cpuid_fault ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves avx_vnni avx512_bf16 wbnoinvd arat vnmi avx512vbmi umip pku ospke waitpkg avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq la57 rdpid bus_lock_detect cldemote movdiri movdir64b fsrm md_clear serialize tsxldtrk amx_bf16 avx512_fp16 amx_tile amx_int8 arch_capabilities
Virtualization:                          VT-x
Hypervisor vendor:                       KVM
Virtualization type:                     full
L1d cache:                               5 MiB (160 instances)
L1i cache:                               5 MiB (160 instances)
L2 cache:                                320 MiB (80 instances)
L3 cache:                                32 MiB (2 instances)
NUMA node(s):                            2
NUMA node0 CPU(s):                       0-79
NUMA node1 CPU(s):                       80-159
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Vulnerable
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Unknown: No mitigations
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; Enhanced / Automatic IBRS; IBPB conditional; PBRSB-eIBRS SW sequence; BHI SW loop, KVM SW loop
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

==============================
Versions of relevant libraries
==============================
[pip3] flashinfer-python==0.6.11.post2
[pip3] numpy==2.2.6
[pip3] nvidia-cublas==13.1.0.3
[pip3] nvidia-cuda-cccl==13.3.3.3.1
[pip3] nvidia-cuda-crt==13.3.33
[pip3] nvidia-cuda-cupti==13.0.85
[pip3] nvidia-cuda-nvcc==13.3.33
[pip3] nvidia-cuda-nvrtc==13.0.88
[pip3] nvidia-cuda-runtime==13.0.96
[pip3] nvidia-cudnn-cu13==9.19.0.56
[pip3] nvidia-cudnn-frontend==1.18.0
[pip3] nvidia-cufft==12.0.0.61
[pip3] nvidia-cufile==1.15.1.6
[pip3] nvidia-curand==10.4.0.35
[pip3] nvidia-cusolver==12.0.4.66
[pip3] nvidia-cusparse==12.6.3.3
[pip3] nvidia-cusparselt-cu13==0.8.0
[pip3] nvidia-cutlass-dsl==4.5.2
[pip3] nvidia-cutlass-dsl-libs-base==4.5.2
[pip3] nvidia-cutlass-dsl-libs-cu13==4.5.2
[pip3] nvidia-ml-py==13.595.45
[pip3] nvidia-nccl-cu13==2.28.9
[pip3] nvidia-nvjitlink==13.0.88
[pip3] nvidia-nvshmem-cu13==3.4.5
[pip3] nvidia-nvtx==13.0.85
[pip3] nvidia-nvvm==13.3.33
[pip3] pyzmq==27.1.0
[pip3] tokenspeed-triton==3.7.10.post20260505
[pip3] torch==2.11.0+cu130
[pip3] torch_c_dlpack_ext==0.1.5
[pip3] torchaudio==2.11.0+cu130
[pip3] torchvision==0.26.0+cu130
[pip3] transformers==5.9.0
[pip3] triton==3.6.0
[conda] Could not collect

==============================
         vLLM Info
==============================
ROCM Version                 : Could not collect
vLLM Version                 : 0.22.0
vLLM Build Flags:
  CUDA Archs: 7.5 8.0 8.6 8.9 9.0 10.0 12.0+PTX; ROCm: Disabled; XPU: Disabled
GPU Topology:
        GPU0    GPU1    GPU2    GPU3    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      NV18    NV18    NV18    0-79    0               N/A
GPU1    NV18     X      NV18    NV18    0-79    0               N/A
GPU2    NV18    NV18     X      NV18    80-159  1               N/A
GPU3    NV18    NV18    NV18     X      80-159  1               N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

==============================
     Environment Variables
==============================
NVIDIA_VISIBLE_DEVICES=void
NVIDIA_REQUIRE_CUDA=cuda>=13.0 brand=unknown,driver>=535,driver<536 brand=grid,driver>=535,driver<536 brand=tesla,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=vapps,driver>=535,driver<536 brand=vpc,driver>=535,driver<536 brand=vcs,driver>=535,driver<536 brand=vws,driver>=535,driver<536 brand=cloudgaming,driver>=535,driver<536 brand=unknown,driver>=550,driver<551 brand=grid,driver>=550,driver<551 brand=tesla,driver>=550,driver<551 brand=nvidia,driver>=550,driver<551 brand=quadro,driver>=550,driver<551 brand=quadrortx,driver>=550,driver<551 brand=nvidiartx,driver>=550,driver<551 brand=vapps,driver>=550,driver<551 brand=vpc,driver>=550,driver<551 brand=vcs,driver>=550,driver<551 brand=vws,driver>=550,driver<551 brand=cloudgaming,driver>=550,driver<551 brand=unknown,driver>=565,driver<566 brand=grid,driver>=565,driver<566 brand=tesla,driver>=565,driver<566 brand=nvidia,driver>=565,driver<566 brand=quadro,driver>=565,driver<566 brand=quadrortx,driver>=565,driver<566 brand=nvidiartx,driver>=565,driver<566 brand=vapps,driver>=565,driver<566 brand=vpc,driver>=565,driver<566 brand=vcs,driver>=565,driver<566 brand=vws,driver>=565,driver<566 brand=cloudgaming,driver>=565,driver<566 brand=unknown,driver>=570,driver<571 brand=grid,driver>=570,driver<571 brand=tesla,driver>=570,driver<571 brand=nvidia,driver>=570,driver<571 brand=quadro,driver>=570,driver<571 brand=quadrortx,driver>=570,driver<571 brand=nvidiartx,driver>=570,driver<571 brand=vapps,driver>=570,driver<571 brand=vpc,driver>=570,driver<571 brand=vcs,driver>=570,driver<571 brand=vws,driver>=570,driver<571 brand=cloudgaming,driver>=570,driver<571 brand=unknown,driver>=575,driver<576 brand=grid,driver>=575,driver<576 brand=tesla,driver>=575,driver<576 brand=nvidia,driver>=575,driver<576 brand=quadro,driver>=575,driver<576 brand=quadrortx,driver>=575,driver<576 brand=nvidiartx,driver>=575,driver<576 brand=vapps,driver>=575,driver<576 brand=vpc,driver>=575,driver<576 brand=vcs,driver>=575,driver<576 brand=vws,driver>=575,driver<576 brand=cloudgaming,driver>=575,driver<576
TORCH_CUDA_ARCH_LIST=7.5 8.0 8.6 8.9 9.0 10.0 12.0+PTX
NVIDIA_GDRCOPY=enabled
NVIDIA_DRIVER_CAPABILITIES=compute,utility
NVIDIA_MOFED=enabled
VLLM_USAGE_SOURCE=production-docker-image
CUDA_VERSION=13.0.2
VLLM_ENABLE_CUDA_COMPATIBILITY=0
LD_LIBRARY_PATH=/usr/local/nvidia/lib64:/usr/local/cuda/lib64:/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/usr/local/cuda/lib64
NVIDIA_CTK_LIBCUDA_DIR=/usr/lib/x86_64-linux-gnu
VLLM_LOGGING_LEVEL=INFO
NVIDIA_GDS=enabled
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_root

🐛 Describe the bug

When serving google/gemma-4-31b-it with --tool-call-parser gemma4 and --reasoning-parser gemma4 under a tool-calling scenario, the vLLM OpenAI-compatible server completely fails to intercept and parse Gemma 4's native tool-calling sequences (e.g., <|tool_call|>call:...<tool_call|>).

Instead of converting these sequences into standard OpenAI tool_calls JSON payloads, the server passes the raw, unparsed tool-calling text blocks directly to the client as a regular assistant text response. Furthermore, it leaks internal Gemma-4 structural string delimiters like <|" and "|> directly into tool parameter fields (such as command, url, attachments, explanation).

Observed Behavior (Raw Client Output received):

<|tool_call>call:Bash{command:<|"|>/Users/john_doe/.workbuddy/binaries/python/versions/3.13.12/bin/python3 generate_dashboards_v2.py<|"|>,description:<|"|>Execute the updated dashboard generation script.<|"|>}<tool_call|>

Or for bash/array commands:

<|tool_call>call:deliver_attachments{attachments:<|"|>["/Users/john_doe/WorkBuddy/2026-06-02-14-11-40/Monitoring_Dashboard_v2.html"]<|"|>,explanation:<|"|>Delivering the fixed v2 monitoring dashboard.<|"|>}<tool_call|>

Expected Behavior:

The vLLM server should intercept these tokens using gemma4_tool_parser and output standard OpenAI Tool Call structures:

{
  "tool_calls": [
    {
      "id": "call_12345",
      "type": "function",
      "function": {
        "name": "preview_url",
        "arguments": "{\"explanation\": \"Display the finalized v3 monitoring dashboard...\", \"url\": \"/Users/john_doe/workspace/project/...\"}"
      }
    }
  ]
}

Steps to Reproduce / Server Command:

We are launching the vLLM docker container with the following arguments:

vllm serve /data/model/gemma-4-31b-it \
  --served-model-name gemma-4-31b-it \
  --tensor-parallel-size 4 \
  --trust-remote-code \
  --gpu-memory-utilization 0.90 \
  --dtype auto \
  --kv-cache-dtype fp8 \
  --max-model-len 262144 \
  --max-num-batched-tokens 65536 \
  --max-num-seqs 128 \
  --enable-chunked-prefill \
  --enable-auto-tool-choice \
  --reasoning-parser gemma4 \
  --tool-call-parser gemma4 \
  --async-scheduling \
  --enable-prefix-caching \
  --chat-template examples/tool_chat_template_gemma4.jinja \
  --default-chat-template-kwargs '{"enable_thinking": true}' \
  --mm-processor-kwargs '{"max_soft_tokens": 1120}' \
  --limit-mm-per-prompt '{"image": 8, "audio": 2}' \
  --speculative-config '{"model": "/data/model/gemma-4-31B-it-assistant", "num_speculative_tokens": 4}' \
  --host 0.0.0.0 \
  --port 8000

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[z-priyanshu]

Root cause + fix

Thanks for the detailed report. I've traced the bug and opened a fix in #44532.

Root cause

The FastIncrementalDetokenizer in vLLM's V1 engine uses added_tokens_decoder[id].content (the HuggingFace fast-tokenizer's stored content string) to produce output.text. On some Gemma4 tokenizer builds, the vocabulary key returned by get_vocab() and the decoded content string for the same token ID differ:

	String
get_vocab()["<|tool_call>"] → 48	<|tool_call> (12 chars)
tokenizer.decode([48]) →	<|tool_call|> (13 chars, extra |>)

The streaming guard in Gemma4ToolParser.extract_tool_calls_streaming checked:

if self.tool_call_start_token not in current_text:   # always True!
    return DeltaMessage(content=delta_text)           # tool call leaks as content

self.tool_call_start_token was set to the vocabulary key <|tool_call> (12 chars), but current_text contained <|tool_call|> (13 chars) from the decoded output — so the substring check always failed and the entire raw tool-call block was emitted as plain content.

The same mismatch affects STRING_DELIM (<|"|> vs <|">), causing garbled argument values.

Fix (#44532)

At parser construction time, call tokenizer.decode([token_id], skip_special_tokens=False) to detect the actual decoded form for TOOL_CALL_START, TOOL_CALL_END, and STRING_DELIM. All text-matching paths (guard check, buffer logic, regex, argument parsing) now use the detected form instead of the module-level constant.

Three new regression tests in TestAltTokenStrings directly simulate the mismatch (mock tokenizer whose decode() returns <|tool_call|> and <|">) and verify no content leakage.

[elseyu]

Root cause + fix

Thanks for the detailed report. I've traced the bug and opened a fix in #44532.

Root cause

The FastIncrementalDetokenizer in vLLM's V1 engine uses added_tokens_decoder[id].content (the HuggingFace fast-tokenizer's stored content string) to produce output.text. On some Gemma4 tokenizer builds, the vocabulary key returned by get_vocab() and the decoded content string for the same token ID differ:

String
get_vocab()["<|tool_call>"] → 48 <|tool_call> (12 chars)
tokenizer.decode([48]) → <|tool_call|> (13 chars, extra |>)
The streaming guard in Gemma4ToolParser.extract_tool_calls_streaming checked:

if self.tool_call_start_token not in current_text: # always True!
return DeltaMessage(content=delta_text) # tool call leaks as content
self.tool_call_start_token was set to the vocabulary key <|tool_call> (12 chars), but current_text contained <|tool_call|> (13 chars) from the decoded output — so the substring check always failed and the entire raw tool-call block was emitted as plain content.

The same mismatch affects STRING_DELIM (<|"|> vs <|">), causing garbled argument values.

Fix (#44532)

At parser construction time, call tokenizer.decode([token_id], skip_special_tokens=False) to detect the actual decoded form for TOOL_CALL_START, TOOL_CALL_END, and STRING_DELIM. All text-matching paths (guard check, buffer logic, regex, argument parsing) now use the detected form instead of the module-level constant.

Three new regression tests in TestAltTokenStrings directly simulate the mismatch (mock tokenizer whose decode() returns <|tool_call|> and <|">) and verify no content leakage.

@pens-u
Thanks for the quick fix! Apologies for the confusion—I realized I made a mistake in my previous description regarding the exact tokens. The actual raw output from the client is a bit different from what I initially stated.

Instead of <|tool_call|>, the engine actually outputs <|tool_call> at the start, and <tool_call|> at the end.

Here is the exact behavior we observed:

Observed Behavior (Raw Client Output received):

<|tool_call>call:Bash{command:<|"|>/Users/john_doe/.workbuddy/binaries/python/versions/3.13.12/bin/python3 generate_dashboards_v2.py<|"|>,description:<|"|>Execute the updated dashboard generation script.<|"|>}<tool_call|>

Or for bash/array commands:

<|tool_call>call:deliver_attachments{attachments:<|"|>["/Users/john_doe/WorkBuddy/2026-06-02-14-11-40/Monitoring_Dashboard_v2.html"]<|"|>,explanation:<|"|>Delivering the fixed v2 monitoring dashboard.<|"|>}<tool_call|>

Could you double-check if your PR #44532 covers this specific mismatch?

[z-priyanshu]

I will check

[gurix]

I was hitting the exact same symptom with pi (a concurrent agentic coding client similar to Claude Code) serving google/gemma-4-26B-A4B-it via vLLM. On vLLM 0.22.0 the raw <|tool_call> wire format leaked into content ~50–56% of the time once --reasoning-parser gemma4 was also enabled. I can confirm this is now fixed upstream in the main branch via the Unify-Parser refactor — posting my verification trace here for anyone tracking this.

Setup

Model:            google/gemma-4-26B-A4B-it  (fp8, 65k context)
vLLM image:       vllm/vllm-openai:nightly-9c7f7741d45154ca11455caa9875beb94e2e66b7
vLLM version:     v0.22.1rc1.dev245+g9c7f7741d
GPU:              RTX PRO 5000 Blackwell (48 GB), TP=1, max-num-seqs=4
Flags:            --tool-call-parser gemma4
                  --reasoning-parser gemma4       ← RE-ENABLED (was the cause on 0.22.0)
                  --chat-template tool_chat_template_gemma4.jinja
                  --default-chat-template-kwargs '{"enable_thinking":true}'
                  --enable-log-requests --enable-log-outputs
Client:           pi (pi.dev), 4 concurrent agents per round, multi-turn agentic load
                  (read → grep → edit → respond chains, 24-file synthetic corpus)
Downstream:       LiteLLM proxy — with the previous strip_gemma_channel.py callback
                  REMOVED from the config (no longer needed)

Result: 40 sessions, 0 leaks across both categories

Metric	vLLM 0.22.0 (old workaround)	vLLM nightly (Unify-Parser)
Tool-call leak (raw <|tool_call> in content)	0/12 ✅ (reasoning-parser OFF)	0/24 ✅ (reasoning-parser ON!)
Channel leak (<|channel> in content)	0/12 ✅ (LiteLLM strip)	0/24 ✅ (no strip needed)
Reasoning separated into reasoning_content	❌ (strip only)	✅ native
enable_thinking	false (forced off)	true

Raw vLLM log grep counts for this run:

• 223 raw <|channel> markers emitted by the model
• 207 raw <|tool_call> markers emitted by the model
• All intercepted by the Unify-Parser — zero reached the client.

Representative trace (chatcmpl-817e2ba49f4fec30)

Streaming delta logs show reasoning and tool calls separated during streaming, not post-hoc:

# reasoning tokens routed to reasoning_content field
(streaming delta): output: '[reasoning: Okay]',         token_id=[19058]
(streaming delta): output: '[reasoning: ,]',            token_id=[236764]
(streaming delta): output: '[reasoning:  I]',           token_id=[564]
(streaming delta): output: '[reasoning:  will]',        token_id=[795]
...
# tool call tokens routed to tool_calls field
(streaming delta): output: '[tool_calls: {"command": ]', token_id=[52]
(streaming delta): output: '[tool_calls: "ls]',          token_id=[5629]
(streaming delta): output: '[tool_calls:  -]',           token_id=[753]
(streaming delta): output: '[tool_calls: F]',            token_id=[236811]

streaming complete (raw generated text, before the Parser's output-side transformation — the <|channel> and <|tool_call> markers are still present here, but the Parser already routed each token to the correct field while streaming):

(streaming complete): output: '<|channel>thought
Okay, I will start by reading the `README.md` file to understand the project
structure and the task.<channel|><|tool_call>call:bash{command:<|"|>ls -F<|"|>}<tool_call|>'
finish_reason: streaming_complete

Client-side OpenAI-compatible response (reconstructed via LiteLLM passthrough, no transformation applied):

finish_reason: "tool_calls"
content: None                          ← clean, no raw wire tokens
reasoning_content: "Okay, I will start by reading the…"   ← natively populated
tool_calls: [{
  id: "chatcmpl-tool-...",
  type: "function",
  function: {
    name: "bash",
    arguments: "{\"command\": \"ls -F\"}"
  }
}]

Root cause on 0.22.0 (already diagnosed in this thread)

On 0.22.0 the gemma4 reasoning parser and tool parser were two separate code paths that stepped on each other's token IDs under streaming/concurrency. The reasoning parser consumed <|channel> markers but left the tool parser in an inconsistent state for the immediately-following <|tool_call>, causing it to fall through and leak the raw text into content. My earlier A/B with 16 sessions per arm on 0.22.0 showed reasoning-ON → 56% tool-call leak, reasoning-OFF → 100% <|channel> channel-leak (neither acceptable).

Fix

The Unify-Parser refactor (commit 68dafcc, merged 2026-06-02, part of RFC #32713) collapses reasoning + tool-call parsing into a single Parser.parse() dispatch. Both parsers now observe the full token stream with shared state, so the <|channel>→<|tool_call> boundary is handled atomically.

Recommendation

Anyone hitting the exact symptom here (raw <|tool_call> / <|"|> leaking into content specifically when --reasoning-parser gemma4 is also on) can fix it by switching to a vLLM build that includes 68dafcc — currently the nightly-* tags on Docker Hub, presumably the upcoming stable release. No downstream strip workaround is required with that build.

[bornmw]

#45553 applied on top of 0.23.0 works for me

[lucianommartins]

Hi @elseyu - I think it was fixed by #45588

Could you pls double check and close it (if really fixed)? Thanks!

[hexsm]

The latest nightly docker image fixed the issue for me too!

[elseyu]

Hi @elseyu - I think it was fixed by #45588

Could you pls double check and close it (if really fixed)? Thanks!

Thanks a lot，it works!