[Bug] thinking_budget not enforced for Qwen3.6 — reasoning consumes all max_tokens

[HH1162]

---

title: "[Bug] thinking_budget not enforced for Qwen3.6 — reasoning consumes all max_tokens"
labels: ["bug", "reasoning", "qwen3"]

Summary

The thinking_budget parameter does not limit reasoning token count for Qwen3.6 models in SGLang. Despite setting thinking_budget: 200, the model generates ~1400 tokens of reasoning content and leaves zero tokens for the final answer (content: null).

This was tested on SGLang v0.5.11 → v0.5.12 with Qwen3.6-27B-FP8. Note that this behavior persists regardless of whether speculative decoding is enabled or disabled.

Environment

• SGLang version: 0.5.11 → upgraded to 0.5.12
• Model: Qwen3.6-27B-FP8 (Qwen3_5ForConditionalGeneration)
• GPU: NVIDIA RTX PRO 6000 Blackwell 96GB
• Quantization: FP8
• OS: Linux 6.17.0-29-generic

Server Launch Command

sglang serve \
  --model-path /path/to/qwen3.6-fp8 \
  --trust-remote-code \
  --attention-backend flashinfer \
  --context-length 260000 \
  --dtype auto \
  --quantization fp8 \
  --enable-cache-report \
  --max-running-requests 16 \
  --max-queued-requests 128 \
  --enable-dynamic-chunking \
  --chunked-prefill-size 8192 \
  --radix-eviction-policy lfu \
  --kv-cache-dtype fp8_e4m3 \
  --enable-strict-thinking \
  --tp-size 1 \
  --host 127.0.0.1 \
  --mamba-scheduler-strategy extra_buffer \
  --preferred-sampling-params '{"custom_params": {"thinking_budget": 200}}' \
  --chat-template /path/to/chat_template.jinja \
  --reasoning-parser qwen3-thinking \
  --tool-call-parser qwen3_coder \
  --enable-prefill-delayer \
  --enable-breakable-cuda-graph \
  --port 1234 \
  --max-prefill-tokens 65535 \
  --mem-fraction-static 0.8

How thinking_budget Was Passed

We tried two different approaches to enforce the budget, neither of which worked.

Phase 1: Custom Logit Processor

Initially, we attempted to use the built-in custom logit processor mechanism:

Server flag: --enable-custom-logit-processor

API Payload:

{
  "custom_logit_processor": "{\"callable\": \"<serialized_dill_hex>\"}",
  "custom_params": {"thinking_budget": 200}
}

Result: Failed. We discovered that Qwen3ThinkingBudgetLogitProcessor hardcodes token IDs (151667/151668) that don't match our model (248068/248069). After manually patching the source to fix the IDs, the processor still did not enforce the budget — reasoning output reached ~6215 characters unconstrained. The logit processor appeared to generate the thinking end token (EOS for reasoning), but the model continued generating reasoning content as if it hadn't seen it.

Phase 2: Strict Thinking

After the logit processor approach failed, we switched to using --enable-strict-thinking with preferred-sampling-params:

Server flags:

--enable-strict-thinking \
--preferred-sampling-params '{"custom_params": {"thinking_budget": 200}}' \
--reasoning-parser qwen3-thinking

Per-request payload:

{
  "model": "Qwen3-27B",
  "messages": [{"role": "user", "content": "请详细推导鸡兔同笼问题（35只头94只脚）的至少3种解法"}],
  "max_tokens": 1024,
  "extra_body": {"thinking_budget": 200}
}

Result: Also failed. The reasoning content was not limited (~1400 tokens generated vs 200 requested), and content was always null. This behavior persisted regardless of whether speculative decoding was enabled or disabled.

Results

Test	reasoning_content	content	total_tokens
Run 1	2702 chars (~1351 tokens)	null	1058
Run 2	2738 chars (~1369 tokens)	null	1058
Run 3	2860 chars (~1430 tokens)	null	1058

Expected: reasoning ~200 tokens, content ~800+ tokens.
Actual: All tokens consumed by reasoning, no answer produced. Budget exceeded by 7x.

Issues Found

1. Token ID mismatch in Qwen3ThinkingBudgetLogitProcessor

File: sglang/srt/sampling/custom_logit_processor.py

The processor hardcodes thinking token IDs that don't match the deployed model:

Token	Hardcoded in SGLang	Actual in Model Tokenizer
THINKING_START_TOKEN_ID	151667	248068
THINKING_END_TOKEN_ID	151668	248069

After manually patching the source to use the correct IDs (248068/248069), the logit processor still did not enforce the budget — reasoning output reached ~6215 characters unconstrained. The logit processor appeared to generate the thinking end token (EOS for reasoning), but the model continued generating reasoning content as if it hadn't seen it, suggesting the logit processor's token suppression is not being applied during generation.

2. --enable-strict-thinking does not enforce thinking_budget

Even with --enable-strict-thinking enabled and thinking_budget passed via both global params and per-request payload, the reasoning content was not limited. The flag appears to accept the parameter but does not implement hard token counting during generation.

We also tested with speculative decoding enabled (--speculative-algorithm NEXTN) and disabled — the budget constraint failed in both cases, confirming the issue lies within the strict thinking/logit processor implementation itself.

Expected Behavior

When thinking_budget: 200 is set, reasoning should be limited to ~200 tokens regardless of whether custom logit processor or strict thinking is used. Remaining max_tokens should be available for the final answer.

Reproduction Steps

1. Start server using the launch command above.
2. Send a chat completion request with max_tokens: 1024 and extra_body.thinking_budget: 200.
3. Observe that reasoning_content far exceeds 200 tokens and content is null.

[cs-cat]

The issue appears to be AI-generated? The custom logit processor's THINKING_START_TOKEN_ID, THINKING_END_TOKEN_ID, and NEW_LINE_TOKEN_ID are hard-coded into the bytecode of the class you pass in. These constants need to be adjusted according to the model's tokenizer configuration. The hard-coded templates in SGLang are not for the Qwen 3.5+ series. The enable-strict-thinking parameter may not be related to your required reasoning budget.

[HH1162]

The issue appears to be AI-generated? The custom logit processor's THINKING_START_TOKEN_ID, THINKING_END_TOKEN_ID, and NEW_LINE_TOKEN_ID are hard-coded into the bytecode of the class you pass in. These constants need to be adjusted according to the model's tokenizer configuration. The hard-coded templates in SGLang are not for the Qwen 3.5+ series. The enable-strict-thinking parameter may not be related to your required reasoning budget.

Thank you for looking into this issue. I did use an agent to help draft the original issue, and following your post I updated the THINKING_START_TOKEN_ID, THINKING_END_TOKEN_ID, and NEW_LINE_TOKEN_ID for Qwen3.6. However, the thinking budget still did not work. After further investigation, I discovered several problems:

Step 1: Custom Logit Processor not injected
Found that custom logit processor was not being injected at all. Modified tokenizer_manager.py to enable the injection. After this change, injection started working, but budget still did not take effect.

Step 2: Budget parameter passed incorrectly
After the logit processor was injected, I found the budget value was being overwritten during parameter passing — the auto_inject_thinking_budget=True logic re-detects and overwrites the user-provided value on each request. If certain conditions are detected (e.g., multi-turn conversation), it forces the budget to None, causing the logit processor to skip execution entirely. Had to disable auto-inject in tokenizer_manager.py.

Step 3: Long input causes定位 failure
Hermes agent sends a very long input prompt (system prompts, tool definitions, conversation history, etc.), causing the logit processor to fail when locating the think start token — when concatenating origin_input_ids + output_ids and using .index(), it may find a token from somewhere in the prompt rather than the  tag actually generated in the current turn, offsetting the entire counting logic.

Step 4: Multi-turn conversation bug
Further analysis revealed: when judging whether to end thinking, the code concatenates all historical messages' input_ids + output_ids, so it always finds the ending tag from a previous turn (always true). Meanwhile, .index() may find the start token from a previous turn, making the token count completely wrong. Fixed this logic.

Step 5: Speculative decoding bypasses constraints
Even after fixing token IDs, handling long input issues, fixing multi-turn bugs, and disabling auto-inject, the budget still did not work. Finally traced it to SGLang having speculative decoding enabled (NEXTN mode): the draft model generates thinking tokens first, and the main model only verifies them. The custom logit processor intervenes at the main model stage — by then, thinking tokens have already been batch-generated by the draft model, so the interception is too late to enforce the budget.

Step 6: Reverted all changes
Confirmed that modifications introduced new bugs and the problem was complex, so reverted all files to their original state.

Ultimately, I failed to make the thinking-budget feature work. Since Qwen3.6 reasoning outputs are too long, I wanted to set a limit, but I'm indeed stuck...

[OriginallyJoe]

The issue appears to be AI-generated? The custom logit processor's THINKING_START_TOKEN_ID, THINKING_END_TOKEN_ID, and NEW_LINE_TOKEN_ID are hard-coded into the bytecode of the class you pass in. These constants need to be adjusted according to the model's tokenizer configuration. The hard-coded templates in SGLang are not for the Qwen 3.5+ series. The enable-strict-thinking parameter may not be related to your required reasoning budget.

Thank you for looking into this issue. I did use an agent to help draft the original issue, and following your post I updated the THINKING_START_TOKEN_ID, THINKING_END_TOKEN_ID, and NEW_LINE_TOKEN_ID for Qwen3.6. However, the thinking budget still did not work. After further investigation, I discovered several problems:

Step 1: Custom Logit Processor not injected
Found that custom logit processor was not being injected at all. Modified tokenizer_manager.py to enable the injection. After this change, injection started working, but budget still did not take effect.

Step 2: Budget parameter passed incorrectly
After the logit processor was injected, I found the budget value was being overwritten during parameter passing — the auto_inject_thinking_budget=True logic re-detects and overwrites the user-provided value on each request. If certain conditions are detected (e.g., multi-turn conversation), it forces the budget to None, causing the logit processor to skip execution entirely. Had to disable auto-inject in tokenizer_manager.py.

Step 3: Long input causes定位 failure
Hermes agent sends a very long input prompt (system prompts, tool definitions, conversation history, etc.), causing the logit processor to fail when locating the think start token — when concatenating origin_input_ids + output_ids and using .index(), it may find a token from somewhere in the prompt rather than the  tag actually generated in the current turn, offsetting the entire counting logic.

Step 4: Multi-turn conversation bug
Further analysis revealed: when judging whether to end thinking, the code concatenates all historical messages' input_ids + output_ids, so it always finds the ending tag from a previous turn (always true). Meanwhile, .index() may find the start token from a previous turn, making the token count completely wrong. Fixed this logic.

Step 5: Speculative decoding bypasses constraints
Even after fixing token IDs, handling long input issues, fixing multi-turn bugs, and disabling auto-inject, the budget still did not work. Finally traced it to SGLang having speculative decoding enabled (NEXTN mode): the draft model generates thinking tokens first, and the main model only verifies them. The custom logit processor intervenes at the main model stage — by then, thinking tokens have already been batch-generated by the draft model, so the interception is too late to enforce the budget.

Step 6: Reverted all changes
Confirmed that modifications introduced new bugs and the problem was complex, so reverted all files to their original state.

Ultimately, I failed to make the thinking-budget feature work. Since Qwen3.6 reasoning outputs are too long, I wanted to set a limit, but I'm indeed stuck...

Hello, I'm also experiencing this issue. I've set max_tokens to 1000, but thinking_budget never seems to work—reasoning_content often consumes all available tokens, leaving the content as null. I noticed this issue has been closed; have you found a solution?

I currently can only consider switching to vllm for model deployment and use thinking_token_budget to achieve budget control.

[HH1162]

Hi @OriginallyJoe,

We hit the exact same problem and got it working. Sharing our solution here.

Why your original approach didn't work: The built-in Qwen3ThinkingBudgetLogitProcessor hardcodes token IDs 151667/151668, but Qwen3.5/3.6 uses 248068/248069. Even after patching those IDs, SGLang still doesn't invoke your custom processor unless you tell it to explicitly.

Our working method — custom logit processor:

Step 1: Add a new processor class in sglang/srt/sampling/custom_logit_processor.py:

class Qwen35ThinkingBudgetLogitProcessor(ThinkingBudgetLogitProcessor):
    """Controls thinking length for Qwen3.5/3.6 models."""
    THINKING_START_TOKEN_ID: int = 248068
    THINKING_END_TOKEN_ID: int = 248069
    NEW_LINE_TOKEN_ID: int = 198

Step 2: Start sglang serve with --enable-custom-logit-processor:

sglang serve \
  --model-path /path/to/qwen3.6-fp8 \
  --trust-remote-code \
  --enable-custom-logit-processor \
  --reasoning-parser qwen3 \
  --port 1234

Step 3: Serialize the processor (run once in your sglang venv):

import dill, json
from sglang.srt.sampling.custom_logit_processor import Qwen35ThinkingBudgetLogitProcessor

data = {"callable": dill.dumps(Qwen35ThinkingBudgetLogitProcessor).hex()}
print(json.dumps(data))

This gives you a hex string like "8004955500...".

⚠️ Most critical part — you MUST pass the callable in every API request. This is the link that tells SGLang which processor class to actually call. Without it, your Qwen35ThinkingBudgetLogitProcessor sits there unused. Just passing custom_params: {"thinking_budget": 200} alone does absolutely nothing — SGLang has no way of knowing WHICH processor should receive those params. The callable IS the bridge between your code and the runtime.

API request payload — both fields are required together:

{
  "model": "qwen3.6",
  "messages": [...],
  "custom_logit_processor": "{\"callable\": \"8004955500...\"}",
  "custom_params": {"thinking_budget": 1200}
}

• custom_logit_processor (with the callable hex string) → tells SGLang to deserialize and invoke YOUR Qwen35ThinkingBudgetLogitProcessor
• custom_params → passes the budget value INTO that processor

No callable = no invocation = thinking_budget is ignored. That's the whole missing piece.

Gotcha: No spaces in JSON keys. "custom_params " or "thinking_budget " will be silently dropped by SGLang — this caught us.

---

If you just want to disable thinking entirely (no budget needed):

Skip the logit processor completely. Just pass chat_template_kwargs:

{
  "model": "qwen3.6",
  "messages": [...],
  "chat_template_kwargs": {"enable_thinking": false}
}

This makes the model skip all reasoning tokens and output direct answers. We use it when thinking isn't needed — much simpler.

[OriginallyJoe]

Hi @HH1162,

Thank you so much for taking the time to give such a detailed explanation and elaboration. Your solution is very insightful, and after trying it out, I found it very effective — it successfully resolved the issue that had been troubling me. Thanks again for your generous help!

All the best.

[HH1162]

Hi @HH1162,

Thank you so much for taking the time to give such a detailed explanation and elaboration. Your solution is very insightful, and after trying it out, I found it very effective — it successfully resolved the issue that had been troubling me. Thanks again for your generous help!

All the best.

You are welcome! ;)