Skip to content

v0.12.0 — unparsed tool calls are no longer silent

Choose a tag to compare

@ahwurm ahwurm released this 06 Aug 01:51
· 270 commits to main since this release

A tool call the serving runtime did not parse used to reach you as an ordinary
assistant message. A model that had called its tools correctly for an entire
session could therefore look like it was narrating work it never did — no file
written, no error, no warning, just prose.

What changed

Unparsed tool calls now surface. The agent loop has always published a
ParseFailed event when a reply looks like a tool call but nothing parsed.
Nothing in any channel subscribed to it, so the one signal that explained the
failure went only to the benchmark metrics sink. The terminal and Discord
channels now render it with the retry count and the runtime flag worth
checking. Loop behavior is unchanged: it still nudges the model with the taught
format and retries up to three times, and a run is never killed over a single
unparsed turn.

Detection of runtime-native tool syntax arriving as chat text — DeepSeek's
DSML (V4-Flash) and the V3 tool_calls_begin envelope.

This is detection, not parsing. LocalHarness deliberately does not add
per-model tool-call parsers. Converting a model's native syntax into
structured tool_calls is the serving runtime's job — llama.cpp --jinja
reads the template embedded in the GGUF, vLLM needs --tool-call-parser
and it already works when that flag is set. What this release changes is that
the failure is named instead of silently rendered as prose.

The capability probe no longer condemns a session on one hiccup. Tool-call
mode for a whole session was decided by a single generation, and any failure —
a busy slot, a slow first token, a connection blip — silently downgraded every
later turn to XML mode. For a model whose native syntax is not <tool_call>,
that meant every tool call in the session was rendered as chat and nothing ran.
An inconclusive probe is now retried. A definitive answer still resolves on the
first attempt, so the healthy path still costs exactly one request. Exhausting
the retries still ends in XML mode with the error recorded — the retry widens
the window for a transient blip, it never assumes a capability the server did
not demonstrate.

Fixed

  • #101doctor told healthy multi-slot llama.cpp servers to relaunch. The
    advisory fired on slot count alone, but modern llama.cpp shares one KV cache
    across slots (kv_unified) and does not divide --ctx-size; it now compares
    the per-request window from /props against the configured budget.
  • #102 — an explicit provider_type hard-failed start when it disagreed
    with the running server, which made setting it strictly worse than leaving it
    blank. It now orders the /tokenize probe instead of narrowing it, so a
    drifted config recovers with a warning. The hard error when neither shape
    answers is preserved.

Caveats

  • Detection covers the DeepSeek syntaxes observed in practice. A model family
    emitting some other unparsed format will still render as prose until its
    pattern is added — the warning exists because that class of failure cannot be
    fully enumerated in advance.
  • The fix for an unparsed tool call is still a runtime flag, not a harness
    setting. LocalHarness can now tell you that, but it cannot convert the syntax
    for you.

Pre-1.0: interfaces and config schema may change without notice.