Eval bug: unsloth/Qwen3.6-27B-GGUF `preserve_thinking` ignored?

[wbste]

Name and Version

ggml_cuda_init: found 1 CUDA devices (Total VRAM: 24575 MiB):
Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes, VRAM: 24575 MiB
load_backend: loaded CUDA backend from C:\llama\ggml-cuda.dll
load_backend: loaded RPC backend from C:\llama\ggml-rpc.dll
load_backend: loaded CPU backend from C:\llama\ggml-cpu-alderlake.dll
version: 8883 (134d6e54d)
built with Clang 19.1.5 for Windows x86_64

Operating systems

Windows

GGML backends

CUDA

Hardware

RTX 3090

Models

No response

Problem description & steps to reproduce

It's a bit weird. I've added chat-template-kwargs = {"preserve_thinking":true} in the past to unsloth/Qwen3.6-35B-A3B-GGUF and it works perfectly.

Confirm by asking:

1. Think of two numbers, tell me the first one.
2. After the reply...
3. what was the second number you thought of?

35b reliably can "see" its previous thoughts and grabs the number. 27b never does. I've confirmed embedded chat templates are identical.

First Bad Commit

No response

Relevant log output

Logs

[aldehir]

Is this through the WebUI?

[wbste]

Nevermind 😄
I needed to correctly capturing and re-inject the reasoning_content from the API response into the next request's messages. Quick test below.

# /// script
# dependencies = ["openai", "rich"]
# ///

import json
import re

from openai import OpenAI
from rich.columns import Columns
from rich.console import Console
from rich.live import Live
from rich.markdown import Markdown
from rich.panel import Panel
from rich.rule import Rule

console = Console()

DEBUG_JSON = True  # set True to dump raw request payload each turn

TurnLog = list[dict]  # each entry: {"prompt", "thinking", "reply"}
turn_log: TurnLog = []

BASE_URL = "http://127.0.0.1:9292/v1"
MODEL = "qwen3.6-27b"
# MODEL = "qwen3.6-35b-a3b"
client = OpenAI(base_url=BASE_URL, api_key="your-api-key-here")


def chat(messages: list[dict]) -> tuple[str | None, str | None]:
    thinking_parts: list[str] = []
    reply_parts: list[str] = []

    def renderable() -> Columns:
        panels = []
        if thinking_parts:
            panels.append(
                Panel(
                    Markdown("".join(thinking_parts)),
                    title="thinking",
                    border_style="dim",
                )
            )
        if reply_parts:
            panels.append(
                Panel(
                    Markdown("".join(reply_parts)), title="reply", border_style="cyan"
                )
            )
        return Columns(panels, equal=True, expand=True)

    stream = client.chat.completions.create(model=MODEL, messages=messages, stream=True)
    with Live(
        renderable(),
        console=console,
        refresh_per_second=15,
        vertical_overflow="visible",
    ) as live:
        for chunk in stream:
            delta = chunk.choices[0].delta
            piece = getattr(delta, "reasoning_content", None)
            if piece:
                thinking_parts.append(piece)
            if delta.content:
                reply_parts.append(delta.content)
            live.update(renderable())

    return "".join(thinking_parts) or None, "".join(reply_parts) or None


def turn(messages: list[dict], prompt: str) -> tuple[str | None, str | None]:
    messages.append({"role": "user", "content": prompt})
    thinking, reply = chat(messages)
    msg: dict = {"role": "assistant", "content": reply}
    if thinking:
        msg["reasoning_content"] = thinking
    messages.append(msg)
    turn_log.append({"prompt": prompt, "thinking": thinking, "reply": reply})
    return thinking, reply


history: list[dict] = []
t1_thinking, _ = turn(
    history, "Think of two secret numbers between 100 and 999. Tell me the first only."
)
_, r2 = turn(history, "Now tell me the second number.")

found = [
    n
    for n in re.findall(r"\b\d+\b", r2 or "")
    if t1_thinking and re.search(rf"\b{n}\b", t1_thinking)
]
if found:
    console.print(
        Panel(
            Markdown(f"{r2}\n\n**Matched in turn-1 thinking:** {found}"),
            title="✓ confirmed in turn-1 thinking",
            border_style="green",
        )
    )
else:
    console.print(
        Panel(
            Markdown(r2 or ""),
            title="⚠ not found in turn-1 thinking",
            border_style="yellow",
        )
    )

# ── full session summary ──────────────────────────────────────────────────────
console.print(Rule("session summary", style="bold"))
for i, entry in enumerate(turn_log, 1):
    console.print(Rule(f"turn {i}", style="dim"))
    console.print(Panel(Markdown(entry["prompt"]), title="user", border_style="blue"))
    if entry["thinking"]:
        console.print(
            Panel(Markdown(entry["thinking"]), title="thinking", border_style="dim")
        )
    console.print(
        Panel(Markdown(entry["reply"] or ""), title="reply", border_style="cyan")
    )

if DEBUG_JSON:
    console.print(
        Panel(
            json.dumps(history, indent=2),
            title="full conversation json",
            border_style="magenta",
        )
    )

[ggerganov]

The "preserve_thinking" does not always work. Here is a repro:

make -j && ./bin/llama-server -hf ggml-org/Qwen3.6-27B-GGUF:Q8_0 --chat-template-kwargs "{\"preserve_thinking\": true}" --webui-mcp-proxy --port 8013

Using the WebUI with MCP tool enabled causes the thinking to be stripped:

curl -s http://127.0.0.1:8013/slots | jq .[3].prompt
"<|im_start|>system\n# Tools\n\nYou have access to the following functions:\n\n<tools>\n{\"type\": \"function\", \"function\": {\"name\": \"web_search_exa\", \"description\": \"Search the web for any topic and get clean, ready-to-use content.\\n\\n      Best for: Finding current information, news, facts, people, companies, or answering questions about any topic.\\n      Returns: Clean text content from top search results.\\n\\n      Query tips: \\n      describe the ideal page, not keywords. \\\"blog post comparing React and Vue performance\\\" not \\\"React vs Vue\\\".\\n      Use category:people / category:company to search through Linkedin profiles / companies respectively.\\n      If highlights are insufficient, follow up with web_fetch_exa on the best URLs.\", \"parameters\": {\"type\": \"object\", \"properties\": {\"query\": {\"type\": \"string\", \"description\": \"Natural language search query. Should be a semantically rich description of the ideal page, not just keywords. Optionally include category:<type> (company, people) to focus results — e.g. 'category:people John Doe software engineer'.\"}, \"numResults\": {\"type\": \"number\", \"minimum\": 1, \"maximum\": 100, \"description\": \"Number of search results to return (must be a number, default: 10).\"}}, \"required\": [\"query\"], \"additionalProperties\": false, \"$schema\": \"http://json-schema.org/draft-07/schema#\"}}}\n{\"type\": \"function\", \"function\": {\"name\": \"web_fetch_exa\", \"description\": \"Read a webpage's full content as clean markdown. Use after web_search_exa when highlights are insufficient or to read any URL.\\n\\nBest for: Extracting full content from known URLs. Batch multiple URLs in one call.\\nReturns: Clean text content and metadata from the page(s).\", \"parameters\": {\"type\": \"object\", \"properties\": {\"urls\": {\"type\": \"array\", \"items\": {\"type\": \"string\"}, \"description\": \"URLs to read. Batch multiple URLs in one call.\"}, \"maxCharacters\": {\"type\": \"number\", \"minimum\": 1, \"description\": \"Maximum characters to extract per page (must be a positive number, default: 3000)\"}}, \"required\": [\"urls\"], \"additionalProperties\": false, \"$schema\": \"http://json-schema.org/draft-07/schema#\"}}}\n</tools>\n\nIf you choose to call a function ONLY reply in the following format with NO suffix:\n\n<tool_call>\n<function=example_function_name>\n<parameter=example_parameter_1>\nvalue_1\n</parameter>\n<parameter=example_parameter_2>\nThis is the value for the second parameter\nthat can span\nmultiple lines\n</parameter>\n</function>\n</tool_call>\n\n<IMPORTANT>\nReminder:\n- Function calls MUST follow the specified format: an inner <function=...></function> block must be nested within <tool_call></tool_call> XML tags\n- Required parameters MUST be specified\n- You may provide optional reasoning for your function call in natural language BEFORE the function call, but NOT after\n- If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls\n</IMPORTANT><|im_end|>\n<|im_start|>user\nHi<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\nHello! How can I help you today?<|im_end|>\n<|im_start|>user\nSup<|im_end|>\n<|im_start|>assistant\n<think>\n"

However, if I remove the MCP tool, it works:

curl -s http://127.0.0.1:8013/slots | jq .[2].prompt
"<|im_start|>user\nHi<|im_end|>\n<|im_start|>assistant\n<think>\nHere's a thinking process:\n\n1.  **Analyze User Input:**\n   - User said: \"Hi\"\n   - This is a simple greeting.\n\n2.  **Identify Intent:**\n   - The user is initiating a conversation.\n   - No specific question or task is provided yet.\n\n3.  **Determine Response Strategy:**\n   - Acknowledge the greeting warmly.\n   - Keep it open-ended to encourage the user to share what they need.\n   - Maintain a friendly, helpful tone.\n   - Keep it concise.\n\n4.  **Draft Response (Mental):**\n   \"Hello! How can I assist you today?\" or \"Hi there! What can I help you with?\"\n\n5.  **Refine Response:**\n   - \"Hi! How can I help you today?\" is simple, friendly, and open-ended.\n   - Matches standard AI assistant greeting protocols.\n\n6.  **Final Output Generation:** (matches the refined response)\n   \"Hi! How can I help you today?\"✅\n</think>\n\nHi! How can I help you today? 😊<|im_end|>\n<|im_start|>user\nSup<|im_end|>\n<|im_start|>assistant\n<think>\n"

Currently, not sure if this is a bug in the WebUI or in the chat template.

cc @aldehir @allozaur

[aldehir]

@ggerganov it's a bug in the WebUI. I noticed this the other day: the webui will strip reasoning from prior tool calling "loops", for lack of a better word. It only retains it for the currently running loop. This isn't a problem for most models, since their templates do this too, but it does break preserve_thinking = true.

[allozaur]

Pushed a fix for this in 3b7f326