Skip to content

v0.12.6 — router-mode token counting + Ollama reasoning visibility

Choose a tag to compare

@ahwurm ahwurm released this 19 Aug 14:02
· 241 commits to main since this release

Two community-surfaced fixes, shipped same-day.

  • llama.cpp router mode works now (#141, reported by @bgtmanuel with a complete repro — thank you): one llama-server hosting several models rejects unnamed /tokenize requests, so exact token counting refused to start. Every counting hop — /tokenize, /apply-template, and doctor's probes — now names the session's model. Single-model servers ignore the field (live-verified, including with mismatched names). Bonus: a tokenizer endpoint that answers with an error is now reported as "rejected the request (with the server's own message)" instead of "unreachable". Honest limit: router mode itself is unit-mocked against the reported 400 shape — the maintainer box serves a single model, so the end-to-end router path wasn't live-tested here.
  • Ollama reasoning deltas are read now (#142, surfaced by an external fork's commit trail — credit Ruivalim): Ollama spells the thinking field reasoning, not reasoning_content, so on reasoning models the decode-speed window opened only when the answer began (a 21-token generation with 1.5 s of thinking recorded 40 tok/s where the honest figure is 10) and reasoning text never surfaced. Both spellings are read in both streaming and non-streaming paths, an empty-text thinking delta counts as the start of generation, and the streaming path now exposes reasoning text on the assembled message at all — it previously did for no provider. Honest limit: no live Ollama on the maintainer box — the wire shape is mocked from the documented form; the SDK-object mechanics were verified against the installed client library.