llama.cpp reasoning budget sampler causes silent hangs after April 10 builds

[antoinezambelli]

llama.cpp reasoning budget sampler causes silent hangs after April 10 builds

Problem

llama.cpp builds after April 10, 2026 (specifically commit d7ff074) register thinking_start_tag / thinking_end_tag for Gemma 4 models. Additionally, the auto-parser (--reasoning-format auto) detects thinking tags from templates at runtime for models like Qwen 3.5.

When thinking tags are registered, the reasoning budget sampler activates with a default budget of INT_MAX (2,147,483,647 tokens) — effectively unlimited. This causes models to enter unbounded thinking phases that:

• Hang indefinitely on some runs (non-deterministic)
• Fill KV cache until it spills to CPU RAM / swap
• Crash llama-server on memory-constrained setups

Affected models (confirmed):

• Gemma 4 (26B, 31B) — explicit thinking tags in chat.cpp
• Qwen 3.5 (27B, 35B) — auto-parser detects <think> tags when --reasoning-format auto is set
• Ministral Reasoning — explicit [THINK] tags

Previous behavior: The reasoning budget sampler did not exist / was not activated for these models. Updating llama.cpp without adding --reasoning-budget silently changes inference behavior.

Reproduction

# Start llama-server with a Gemma 4 or Qwen 3.5 model (llama.cpp post-April 10)
llama-server -m model.gguf -ngl 999 --jinja --port 8080

# Run any multi-turn tool-calling workflow — some runs will hang
# Server log shows: "reasoning-budget: activated, budget=2147483647 tokens"

Proposed fixes

Option A: Default --reasoning-budget 0 in ServerManager.start()

Add a default reasoning budget of 0 (thinking disabled) to the server start command in forge/server.py. Models that benefit from thinking can opt in via extra_flags.

Option B: Document the behavior

Add a note to MODEL_GUIDE.md and/or CHANGELOG that llama.cpp builds after April 10 require --reasoning-budget 0 (or a specific cap) to prevent hangs. List affected models and recommended budgets.

Context

Discovered during forge-code eval batch runs on dual 5070 Ti rig. Gemma 4 31B with unlimited budget spent 340s on a task that takes 100s with --reasoning-budget 0 and produced identical quality. Qwen 3.5 27B hung for 50+ minutes on a run that normally completes in 75s.

forge-code workaround: _get_server_flags() now defaults to --reasoning-budget 0 for all models.

[antoinezambelli]

Additional llama.cpp commits since then, will reevaluate.