Skip to content

v0.13.0: thinking off, decided by the model instead of guessed

Choose a tag to compare

@Helldez Helldez released this 19 Jul 19:24

Turning Thinking off used to set one template variable and hope. That variable is only a request
to the model's chat template, and plenty of templates never read it — so on those models the setting
was dropped in silence and the model reasoned anyway.

The engine now renders the template at load, works out what that model will actually honour, and
says so. Nothing about this is keyed on model names.

Three outcomes, decided from the model itself

  • template — the chat template reads the flag. Nothing else needed (Qwen3, Qwen3.6).
  • prefill — it does not, but reasoning is a structural section of the format, so the turn starts
    past it. Not something the model can decline (gpt-oss / harmony).
  • none — the model declares its own reasoning span and simply cannot be asked to skip it. Reported
    rather than papered over: the app disables the Thinking switch and explains why (LFM2.5).

Where the turn does start past the reasoning, the span is written by llama.cpp's own handler for
that template, so no family's markers appear in this engine at all. That retires the last
model-specific constants in the decode path: priming gpt-oss used to be a literal <|start|>assistant
suffix test plus a literal <|channel|>final<|message|> in session.cpp. An upstream bump that
changes those markers now needs no engine change.

Measured, not assumed. Handing a model that owns its <think> span a pre-closed empty one is a
suggestion it can ignore — and LFM2.5 does, reasoning untagged into the answer, worse than leaving
the setting alone. So the engine does not do it. Verified on-device across the catalog: LFM2.5
reports none and keeps its reasoning tagged and collapsible; gpt-oss-120b and Qwen3.6-35B-A3B
answer with no reasoning at all.

Protocol: BMOE_READY gains think_ctl (template | prefill | none) — see docs/telemetry.md.

Streaming, benchmarks and published numbers are unchanged.