Misc. bug: reasoning-budget stopped working

[wbste]

Name and Version

llama-cli --version
version: 9033 (70a830911)
built with Clang 19.1.5 for Windows x86_64

Operating systems

Windows

Which llama.cpp modules do you know to be affected?

llama-server

Command line

llama-server `
  --model "D:\AI\models\unsloth\Qwen3.5-2B-GGUF\Qwen3.5-4B-Q8_0.gguf" `
  --mmproj "D:\AI\models\unsloth\Qwen3.5-2B-GGUF\mmproj-F16.gguf" `
  --temp 0.6 `
  --top-k 20 `
  --top-p 0.95 `
  --min-p 0.0 `
  --repeat-penalty 1.0 `
  --presence-penalty 0.0 `
  --reasoning-budget 100 `
  --reasoning-budget-message "Wait, I am overthinking this. I should answer now." `
  --chat-template-kwargs '{"preserve_thinking":true}'

Problem description & steps to reproduce

Only tested on a few models so far. My workflow has been running fine until this morning. When reasoning turned on I'm not seeing the reasoning budget message any longer, and seems to be replaced with something like *'/'''''/''''/'. Therefore it blasts through the budget and tends to start to loop.

Demo code, where you'll see right at the reasoning-budget threshold the characters appear:

# /// script
# dependencies = ["httpx", "openai"]
# ///

import asyncio
import base64
import logging

import httpx
from openai import AsyncOpenAI

logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger(__name__)

OWNER, REPO = "ggml-org", "llama.cpp"
BASE_URL = "http://127.0.0.1:9292/v1/"
API_KEY = "local"
MODEL_NAME = "qwen3.5-2b"
USER_PROMPT = "Summarize the following in three sentences or less, no markdown. Don't preamble with you're making a summary; just do it."


async def fetch_readme(owner: str, repo: str) -> tuple[str, str]:
    headers = {
        "Accept": "application/vnd.github+json",
        "X-GitHub-Api-Version": "2022-11-28",
    }

    async with httpx.AsyncClient(timeout=30) as client:
        repo_data = (
            await client.get(
                f"https://api.github.com/repos/{owner}/{repo}", headers=headers
            )
        ).json()
        readme_resp = await client.get(
            f"https://api.github.com/repos/{owner}/{repo}/readme", headers=headers
        )

    title = repo_data.get("full_name", f"{owner}/{repo}")
    if desc := repo_data.get("description"):
        title = f"{title} — {desc}"

    markdown = ""
    if readme_resp.status_code == 200:
        markdown = base64.b64decode(readme_resp.json().get("content", "")).decode(
            "utf-8", errors="replace"
        )

    return title, markdown


async def stream_summary(content: str) -> None:
    client = AsyncOpenAI(base_url=BASE_URL, api_key=API_KEY)
    thinking_chars = 0
    in_thinking = True

    print("\n--- THINKING ---", flush=True)
    stream = await client.chat.completions.create(
        model=MODEL_NAME,
        messages=[{"role": "user", "content": f"{USER_PROMPT}\n\nContent:\n{content}"}],
        extra_body={"chat_template_kwargs": {"enable_thinking": True}},
        stream=True,
    )
    async for chunk in stream:
        if not chunk.choices:
            continue
        delta = chunk.choices[0].delta
        if reasoning := getattr(delta, "reasoning_content", None):
            thinking_chars += len(reasoning)
            print(reasoning, end="", flush=True)
        if answer := delta.content:
            if in_thinking:
                in_thinking = False
                print(
                    f"\n--- ANSWER (thinking ~{thinking_chars // 4} tokens) ---",
                    flush=True,
                )
            print(answer, end="", flush=True)

    print("\n--- END STREAM ---", flush=True)


async def main() -> None:
    log.info("Fetching %s/%s README...", OWNER, REPO)
    title, content = await fetch_readme(OWNER, REPO)
    log.info("Title: %s", title)
    try:
        await stream_summary(content)
    except Exception as e:
        log.error("Error: %s", e)


asyncio.run(main())

First Bad Commit

Around the b9018 release

Relevant log output

Logs

[aldehir]

Can you check if this was introduced in a4701c9 or was it before?

[wbste]

Can you check if this was introduced in a4701c9 or was it before?

Looking at it looks like b9019 does not contain that commit and b9020 does.

b9019: Behaves as expected and honors the budget.
b9020: Fails; where it should inject the message: The text lists Python, Go'/'''''/''''/''/''/''/''/''/'...

So that commit does indeed appear to be the root cause.

[tvall43]

did my usual weekly pull of llama.cpp this morning, later noticed that frigate wasnt getting event descriptions and litellm logs are full of output like ''/''''/'/'/'/'/'/'/'/'/'/'/'/'/'/'/'/'/'/'/'/'/'/'/'/'/'/'/'/'/'/'/'/'/'/'/'/'/'...". reverting a4701c9 seems to have fixed it.
qwen3.6-35b-a3b on a couple radeon pro v340s.

[aldehir]

See if #22740 fixes this for you.

[wbste]

See if #22740 fixes this for you.

Yep, seems to have fixed it!

QQ for ya, does adding reasoning-budget-message on top of reasoning-budget have any benefits?

[aldehir]

QQ for ya, does adding reasoning-budget-message on top of reasoning-budget have any benefits?

Yes. When using a reasoning budget, the model isn't aware that it has a limited window to think. It will think normally, and if you don't inject a message, it will forcefully output </think> and continue thinking in the content.

A message guides it out of thinking. This is really easy to test with a small reasoning budget and no message.

[kyano]

Even after applying #22740 , --reasoning-budget 0 does not disable thinking steps.
It's reproducible both on llama-server and llama-cli.

Reverting a4701c9 does resolve this.

1. --reasoning-budget 0 : not work
2. --reasoning-budget 0 --chat-template-kwargs "{ \"enable_thinking\": false }" : not work, too

I've tested with the official jinja template of Qwen3.6 35B A3B, from huggingface.

[aldehir]

Don't use --reasoning-budget 0 with enable_thinking = false. Use the template variable or --reasoning off.

[wbste]

Don't use --reasoning-budget 0 with enable_thinking = false. Use the template variable or --reasoning off.

Hm...llama-server works correctly with chat-template-kwargs set client-side, but llama-cli ignores it.

Works: llama-cli.exe -m .\Qwen3.5-2B-Q8_0.gguf --reasoning off
Doesn't work: lama-cli.exe -m .\Qwen3.5-2B-Q8_0.gguf --chat-template-kwargs '{"enable_thinking": false}'

Seems related: #20409

[kyano]

Don't use --reasoning-budget 0 with enable_thinking = false. Use the template variable or --reasoning off.

1. AFAIK, With --reasoning off, I can not enable thinking PER REQUEST (disabled globally). This is why I use --reasoning-budget 0 as a workaround.
2. Using the template variable shows a deprecation warning:

   llama.cpp/common/arg.cpp

   Lines 2949 to 2952 in 2496f9c

   	if (item.key() == "enable_thinking") {
   	LOG_WRN("Setting 'enable_thinking' via --chat-template-kwargs is deprecated. "
   	"Use --reasoning on / --reasoning off instead.\n");
   	}

Settings the default(nothing explicitly set) to non-thinking is not supported by design?

ADDED: And do you mean that the previous behavior (reverting a4701c9) is not intended, but just works by chance?

[aldehir]

I'll look into the template variable, it should remain supported. Thanks for the heads up.

ADDED: And do you mean that the previous behavior (reverting a4701c9) is not intended, but just works by chance?

Yes.

[kyano]

Oh, I got it. Then my issue is not related to the OP's. (and not a bug too)
Thanks a lot for the confirmation!

---

just for anyone who search this later:

The working one: --reasoning auto --chat-template-kwargs "{\"enable_thinking\": false}"
"reasoning" must be set to auto, not on. If its value is "on", --chat-template-kwargs "{"enable_thinking": false}" does not work.