v0.4.2
The no-silent-defaults release. Two review passes went through the
three chat surfaces asking one question of each field: what happens
when a request is wrong? Too often the answer was "it is repaired and
answered 200" - a missing field defaulted, an unrenderable turn
dropped, an invalid ordering rewritten, a control validated and then
never read. A prompt that does not say what the caller submitted is
the failure mode nothing downstream can detect, because the response
looks like a success. More than twenty of those are now HTTP 400s that
name the field. Qwen2.5 and Qwen3 also gain native tool calling here,
and the first external hardware report (a DGX Spark) brought NVFP4
support on the CPU and honest unified-memory reporting.
Qwen speaks its own tool protocol. Declarations render into the
family's trained # Tools / <tools> block, a call comes back as its
<tool_call>{"name":...,"arguments":...}</tool_call> turn, results
replay as grouped <tool_response> blocks, and tool names and
argument schemas are constrained directly in that native grammar
instead of runner's generic JSON envelope. Buffered and streaming
output map back to the OpenAI shape, and the same conversation renders
byte-identically on /v1/chat/completions, /v1/responses and
/v1/messages. Qwen3 history also keeps the reference template's
empty <think> block before a trailing historical assistant answer,
which runner did not render at all.
Histories are validated before anything is rendered. Every chat
turn must be an object with an explicit role and usable content: a
turn with no role used to be rendered as user, and a turn whose
content was neither string nor array was dropped from the prompt
outright. Replayed tool_calls must carry a non-empty id, a
function type, a non-empty name, and arguments that parse as a JSON
object. Role sequences no template can represent (a system turn after
history has started on llama2/gemma/mistral/apertus/ornith, or broken
user/assistant alternation) are refused by one shared check on all
three surfaces rather than rendered into something the reference
template would never produce. Responses input items and Anthropic
tool_use/tool_result blocks get the same treatment, including a
required function_call_output.output: an absent member and an empty
string used to collapse to the same empty tool turn, so broken agent
history looked accepted while the model was handed an event that never
happened. A tool result that names no call is now refused on every
family, not just Harmony, instead of being rendered under the
template's 'unknown' fallback or named after its tool_call_id - a
function name invented from an identifier and declared nowhere.
Runner is text-only, and now says so. Image, file and other
non-text content parts are refused on Chat Completions, Responses and
Anthropic tool_result content, rather than being removed while the
adjacent text is answered successfully.
Eight more fields were accepted and then ignored. seed: 0 asked
for a reproducible run and did not get one, because the sampler's
xorshift64 has a fixed point at state 0 and only a seed above zero was
adopted. Anthropic thinking.type was validated and never read, so
{"type":"disabled"} rendered exactly like a request that said
nothing. The request timeout was handed only to the decode loop, so a
long enough prompt overran its own bound by the whole prefill and then
reported "context overflow" instead of a timeout; it is polled at
every prefill chunk now and expiry there answers 408, naming the
prompt. Replayed Anthropic reasoning was dropped unconditionally while
chat replayed Harmony reasoning from the same conversation, so one
model described two ways gave two different prompts. Wrong-typed
tools[].type was normalized to "function" on all three surfaces. A
tool_result after text in the same Anthropic message was silently
reordered, answering 200 for a conversation that had been rewritten.
And usage.output_tokens_details.reasoning_tokens was hard-coded to
zero; it is counted per token now, which is also what makes an
unenforced thinking budget visible to the caller.
Six latent defects, none reachable on this tree. Each is a hole
the next change falls into: the scheduler not paying back its wait
count on the stop path; Metal's two hand-kept pipeline lists asserting
11 of the 16 types the loader admits, while the encoders index those
tables with no nil check (the check now runs off the admission
predicates themselves); a NUL byte in a request header hiding the real
terminator from every parse below it; outside_reason adding a
would-have-written count to a length; a failed strdup publishing
instance records with NULL model names; and json_escape writing its
terminator under cap 0.
The accept thread no longer drains requests. /health,
/v1/models, /v1/capabilities and /unload are answered from the
accept loop, which used to consume the request there under a 0.5 s
budget so that closing would not reset the connection and discard its
own reply. It now hands any request it cannot prove bodyless to a
slot, whose bounded reader consumes the whole request before
replying, so the only thread calling accept() never waits on a
client. Allocation failures inside the tool envelope and the Harmony
prompt renderer are also 500s now rather than being reported to the
caller as a 400, and /health can no longer identify a resident model
by a truncated name that /v1/models spells in full.
The Python consumer boundary. ManagedRunner.start() accepts only
a /v1/capabilities whose new pid field matches the child it
spawned, so a pre-existing Runner already on the port cannot be
mistaken for a successful startup. Startup leases treat unreaped
zombie owners as stale claims. And cancel_event now interrupts a
silent blocking stream read instead of being observed only between SSE
events.
First external hardware report: DGX Spark GB10. The arm64 plus
CUDA combination shipped untested and works. Two things it found:
NVFP4 (ggml type 40, NVIDIA's block-scaled FP4) refused with
"unsupported type 40 (?)" and now dequantizes and serves on the CPU
path, gated against an independent double-precision reference decode,
with CUDA and Metal declining it by name so --caps advertises it
under quants and not gpu_quants. And --caps reported
unified_memory: false on a machine that is nothing but, because the
field was a compile-time constant; it is queried from the device now,
and on an integrated device the GPU offload budget is capped at
OS-available RAM, because "VRAM free" and "RAM free" are two views of
one pool and treating them as two budgets over-promises.
What that last item has and has not been verified against. The
NVFP4 decode is gated by an automated test and runs on every platform.
The CUDA-side changes are covered by no automated test, because CI has
no GPU. The integrated-device probe and the unified-pool budget cap in
particular have not executed on integrated hardware at all: the field
report that prompted them was made against v0.4.1, so the fix has not
been back-confirmed on the machine that found the bug. Reasoned and
reviewed, not yet run. If you are on a Spark, a Jetson or any other
unified-memory device, this release wants your report rather than
claiming it already has it.
Full details in the CHANGELOG.