[Bug]: `--reasoning-parser gemma4` silently disables structured output (xgrammar) when `enable_thinking=false`

[leuski]

Your current environment

• vLLM version: v0.19.0 (vllm/vllm-openai:v0.19.0-x86_64-cu130-ubuntu2404 with transformers>=5.5.0)
• GPU: NVIDIA RTX PRO 6000 Blackwell Server Edition (96GB GDDR7, SM 12.0)
• OS: Linux 5.15.0-171-generic
• CUDA: 13.0
• Python: 3.12

Model

google/gemma-4-E4B-it (dense 4B model, FP8 quantization)

Also applies to any Gemma 4 variant (google/gemma-4-26B-A4B-it, google/gemma-4-31B-it) — and likely any model whose reasoning parser uses channel-style delimiters not present in the prompt.

Command

vllm serve google/gemma-4-E4B-it \
  --quantization fp8 \
  --max-model-len 25600 \
  --max-num-seqs 32 \
  --kv-cache-dtype fp8 \
  --gpu-memory-utilization 0.92 \
  --enable-auto-tool-choice \
  --tool-call-parser gemma4 \
  --reasoning-parser gemma4 \
  --default-chat-template-kwargs '{"enable_thinking": false}' \
  --structured-outputs-config '{"backend":"xgrammar"}'

🐛 Describe the bug

When --reasoning-parser gemma4 is specified together with --default-chat-template-kwargs '{"enable_thinking": false}', the xgrammar structured output engine is completely bypassed for every request. Grammar constraints (JSON schema, BNF, etc.) are never enforced. The model generates unconstrained text that happens to look valid because it's well-trained, but the grammar FSM never runs.

This manifests as a dramatic performance difference that led to discovering the bug:

Benchmark data (single GPU, google/gemma-4-E4B-it FP8)

max_parallel	TPS (with parser — grammar bypassed)	TPS (without parser — grammar enforced)
1	95.8	65.8
2	92.1	51.5
4	87.8	23.1
8	78.7	17.1
16	65.4	14.6

max_parallel	TTFT (with parser)	TTFT (without parser)
1	0.232s	0.284s
4	0.237s	0.388s
16	0.302s	0.532s

The "with parser" numbers are faster because xgrammar bitmask computation and FSM advancement are silently skipped on every decode step.

Root cause

The bug is in vllm/v1/structured_output/__init__.py, in the interaction between should_fill_bitmask() / should_advance() and the reasoning parser.

Step 1: is_reasoning_end() returns False for prompts without thinking tokens

Gemma4ReasoningParser inherits from BaseThinkingReasoningParser, which scans the prompt backward for <|channel> (start, token 100) or <channel|> (end, token 101):

# vllm/reasoning/basic_parsers.py
def is_reasoning_end(self, input_ids):
    for i in range(len(input_ids) - 1, -1, -1):
        if input_ids[i] == start_token_id:
            return False
        if input_ids[i] == end_token_id:
            return True
    return False  # ← neither found → "reasoning has NOT ended"

With enable_thinking: false, the prompt contains neither <|channel> nor <channel|>. The method returns False — meaning "reasoning has not ended yet."

Step 2: Structured output engine skips grammar enforcement

# vllm/v1/structured_output/__init__.py
def should_fill_bitmask(self, request):
    if self.reasoner is not None:
        if self.enable_in_reasoning:  # default: False
            return True
        return request.structured_output_request.reasoning_ended  # ← False!
    return True  # ← no parser: always fill

def should_advance(self, request):
    if self.reasoner is None:
        return True  # ← no parser: always advance
    if self.enable_in_reasoning:
        return True
    if structured_req.reasoning_ended:  # ← False!
        return True
    # ... checks is_reasoning_end_streaming() for <channel|> in delta ...
    return False  # ← never advances

Step 3: Model never generates <channel|>, so reasoning_ended stays False forever

Since thinking is disabled, the model never outputs the <channel|> end token. The is_reasoning_end_streaming() check on each decode step never finds it. Grammar enforcement is permanently disabled for the entire generation.

Summary

Config	Bitmask filled?	FSM advances?	Grammar enforced?
--reasoning-parser gemma4 + enable_thinking: false	NO	NO	NO — silently bypassed
No --reasoning-parser	YES	YES	YES — works correctly

Suggested fix

is_reasoning_end() should return True (not False) when no reasoning tokens are found in the input. If thinking was never started, there is no reasoning to wait for — the model is already in "content" mode.

# Fix in basic_parsers.py
def is_reasoning_end(self, input_ids):
    for i in range(len(input_ids) - 1, -1, -1):
        if input_ids[i] == start_token_id:
            return False
        if input_ids[i] == end_token_id:
            return True
    return True  # ← no thinking tokens found → treat as "not in reasoning"

Alternatively, the structured output engine could handle the "no reasoning tokens in prompt" case explicitly, treating it as reasoning_ended = True.

Related issues

• Guidance backend structured output doesn't work with openai_gptoss reasoning parser (offline LLM.generate) #37359 — Same root cause for GPT-OSS models with guidance backend (offline path)
• Guidance structured output blocked during thinking with nemotron_v3 reasoning parser (offline LLM.generate) #37362 — Same root cause for Nemotron V3 reasoning parser
• Bug: Speculative Decoding (MTP) Causes </think> Detection Failure in Structured Output + Reasoning Mode #34650 — Speculative decoding causes </think> detection failure in structured output
• think=false breaks format (structured output) for gemma4 — format constraint silently ignored ollama/ollama#15260 — Identical bug for Gemma 4 in Ollama (think=false + format silently ignored)
• format is ignored when think is disabled for qwen3.5 series ollama/ollama#14645 — Same bug for Qwen 3.5 in Ollama

Impact

Silent correctness issue: Users who configure --reasoning-parser gemma4 with enable_thinking: false (a common production setup to get reasoning parsing without thinking overhead) get zero grammar enforcement on structured output. The output appears correct because the model is well-trained, but the safety guarantee of grammar-constrained decoding is completely lost. There are no warnings or errors logged.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[bbrowning]

One thing to note - the chat template for the larger Gemma 4 models insert empty reasoning blocks when enable_thinking=false automatically, which I believe would keep this from being an issue? See https://huggingface.co/google/gemma-4-26B-A4B-it/blob/47b6801/chat_template.jinja#L263-L265

The smaller model chat templates do not. That would explain why you're seeing this with the E4B variant I believe. There are some other bugs with our reasoning end detection for these models that I'm tracking down as I use them in various coding assistants.

However, in streaming cases, the input_ids value here is only the delta token ids and not the full generated text. So, we can't simply assume that because no reasoning tokens are present in that delta that reasoning has ended, as we could be in the middle of reasoning. The Gemma 4 models may need to track if reasoning has started versus only if it's ended?

Or perhaps the generation prompt for them should be changed to add an open thinking tag when thinking is enabled and add the completed empty thinking tags when thinking is not enabled, even for the small models. I believe that would also fix this type of issue, as we'd always have a reasoning tag in every generation prompt to key off of.

[leuski]

One thing to note - the chat template for the larger Gemma 4 models insert empty reasoning blocks when enable_thinking=false automatically, which I believe would keep this from being an issue?

I believe you are right, I stand corrected. I do not see the same issue with the 26b version. Only with e4b.

[FredericOdermatt]

Agree with the findings. Additionally this applies to both xgrammar and llguidance backends.

Gemma Release Page mentions:

Disabled Thinking Behavior: For all models except for the E2B and E4B variants,
if thinking is disabled, the model will still generate the tags but with an empty thought block:
<|channel>thought\n<channel|>[Final answer]

I'm assuming the change comes from different post training.

Current work around: vllm maintains the model agnostic reasoning_effort in ["none", "low", "medium", "high"]

Working solution to run without reasoning and constrained decoding:

extra_body.update({"reasoning_effort": "none"})

[danielwit-lb]

How does it work with enabled reasoning? Is model is 'thinking' and once its done the xgrammar backend is applied for the final answer? Is it possible to get reasoning output and the structured output from the same request similar to OpenAI API?