Repository navigation
A Qwen3.5-family request can now resume from a saved recurrent state. Every
request writes one prefill event.
Most other changes are fixes. A stop string now ends the decode on every path.
A generation that reaches the context ceiling stops there and keeps its
output. A speculative request that asks for logprobs gets them. A drafter
whose width does not match its verifier is refused.
Four changes to read before upgrading: prefill_ms moved off decode_profile
and the speculative done line to the new generate: prefill event
(Added); a speculative drafter whose hidden size, layer count or
vocabulary size does not match its verifier is refused, and an EAGLE-3
config.json must declare hidden_size and vocab_size (Changed); the
request time cap ends a reply that has started to stream (Changed); the SSD
index schema version is 9, so the SSD cache of an older version is wiped at the
first open (Fixed).
Added
-
Every request writes one
generate: prefillevent. It carries the
prompt tokens, the tokens a prompt cache supplied, the tokens the prefill ran
the model over and the prefill time, on every architecture and every
speculative drafter. It is the one producer of the prefill time:decode_profile
and the speculativedoneline no longer carryprefill_ms. An exact cache hit reports its whole prompt as reused and
a prefill time of zero. Before, only some paths wrote a prefill time, in
decode_profile, and no event said how many tokens the prefill ran over. Seedocs/PROFILING.md§ "The prefill event of a request". -
The published-protocol harness reports three more columns. The fixed
prompt's input speed now has two definitions, the prompt over the time to
first token and the engine's own prefill rate (prefill_engine_tps, off the
event above); the engine rate is refused on a prompt that was not cold. A
speculative run can be judged against a plain run at temperature 0 with
--greedy-passand--greedy-against, through the divergence oracle, and
the table counts identical, near-tie, refused and unjudgeable answers by
dataset.--longctx 32kmeasures a checked-in long prompt on the plain arm.
None of these runs by default.ingest/published_ingest.pyrecords the
long-context row and the per-dataset greedy counts (newgreedy_*registry
metrics), and the re-query after it checks them. -
A Qwen3.5-family request resumes from a saved recurrent state. A
prompt-cache entry of Qwen3.8 or Ternary-Bonsai-2 served an exact replay
only: the recurrent state cannot be cut back, so a longer or diverging prompt
prefilled from token 0. Two shapes now resume. A request that extends the
whole stored prompt resumes at the stored length. An entry also keeps one
saved recurrent state at the last full 256-token block that does not reach
the end of its prompt, and a request whose first differing token is at or
after that block resumes there, with the K/V cut to it. An entry whose stores
refuse the cut does not offer it. The saved state counts in the entry's
kv_bytes: at 4,096 tokens on Qwen3.8-27B-4bit the entry grows by 36%
(153,944,064 B). A cold prefill pays one clone of the state, 0.01% of the
time to first token at 4,096 tokens and 0.09% at 32,768. At 32,768 tokens
the entry is above the default 2 GiB RAM cap of the prompt cache, so it is
not stored and the clone buys nothing. The engine now judges every slot that
shares a block, best match first, and the longest reuse any slot allows
serves. The SSD tier spills the anchor of such an entry, so the tier serves
these prompts across restarts; before, an entry whose prompt ended inside a
block was withheld. Seedocs/PROMPT_CACHE.mdanddocs/SSD_TIER.md. -
rmlx eval spec-answer --plain F --spec Fjudges a speculative answer
against a plain one. It reads two non-streamed chat-completion bodies that
carrylogprobs, applies the divergence-confidence oracle of
docs/SPEC_ANSWER_EQUIVALENCE.mdand prints the verdict, the position, the
margin, the rank and the reason. Exit 0 is a near tie, 1 a divergence the
oracle refuses, 2 an input it cannot judge. It loads no model, takes no
claim and opens no metrics recorder. The oracle moved from the equivalence
test tormlx_models::answer_equivalence, so the test and the command share
one rule. -
The published-protocol harness reports the Metal peak, a perplexity section
and labelled campaigns.scripts/spec_bench_published.shreports the
Metal allocator peak beside the physical footprint of the fixed-length
prompt.make published-tablerenders a perplexity section from an
rmlx eval pplrecord of the same checkpoint and corpus. A run needs a
--labeland files its raw material in its own campaign directory under
<RMLX_HOME>/bench/published/<label>/. Every row it ingests carries the
label in its notes.--recordends by re-querying the rows it wrote, and it
refuses a result that they do not reproduce. The old fixed result directory
and the fixed staging path of the ingest are gone.
Changed
-
scripts/spec_bench.shfiles a near-tie flip and still stops on a real
divergence. The script stopped the run when a speculative arm's answer
differed from the plain arm's in any byte. Greedy verification emits the
verifier's own argmax, but a near tie in bf16 can flip between two equally
right streams, so every Qwen3.8-27B 4-bit arm that met one was refused. After
its warmup each phase now sends one unstreamed request withlogprobsand
top_logprobs2, binds the answer to its arm's digest, and pauses 15 s. A
difference with no sampler is judged byrmlx eval spec-answer. A near tie
is filed asanswer_check=near_tie@<token>, withdivergence_rankand
divergence_marginin the notes. A divergence the oracle refuses, an arm
that ran past or stopped far short of the reference, measured speculative
runs that differ from each other, a pass that does not bind to its digest,
an unanswered pass and a judge whose exit status does not match its verdict
stop the run with no row. When its digest differs, EAGLE-3 is refused unless
its logprob pass binds. -
Every speculative drafter is tied to its verifier by one pairing check.
The MTP sidecar was matched by the hidden size alone, DFlash and DFlash 2
never read the layer count of the base they were built for, and EAGLE-3 and
the Gemma4 assistant each kept a check of their own, so a drafter built for
another base loaded without a refusal. The hidden size, the layer count and
the vocabulary size that the drafter's config declares are now compared with
the verifier's in one place for all five drafters. A mismatch is
Error::SpeculativePairingand names the key and both values. A config that
declares none of the keys loads with a warning that names the drafter
directory. Behaviour change: an EAGLE-3config.jsonmust declare
hidden_sizeandvocab_size; the fallbacks 2048 and 248320 are gone. The
Gemma4 assistant is checked by the widths ofpre_projection.weightand
post_projection.weight, so one withoutbackbone_hidden_sizeis still
refused on a verifier of another width. No config key names the base model,
so two bases with the same trunk shape cannot be told apart
(docs/SPECULATIVE.md). -
The test targets are gemma-4-e4b, Qwen3.8-27B-4bit and
Ternary-Bonsai-2-27B.scripts/perf_canary.shandscripts/perf_ab.sh
default to them, as domake model-check-full,perf-iterand the
autoresearch defaults. The first-generation Bonsai, Qwen3.6 and medgemma
cells stay in the tree and run undermake gpu-test. The codec-matrix
runner reads each command's own exit status (it read the status of
|| true, so a held Metal claim failed a row as an output mismatch), ends
INCOMPLETEwith exit 3, and exits 1 for a failed row or a run that ran
nothing.docs/TESTING.mdlists the targets and the features no target
covers. -
A moved round-stream cell is a failure only on the pinned MLX pair. Each
of the six pinned speculative equivalence tests checks its own per-round
cells againstMANIFEST.sha256after it has judged the answers. On another
MLX pair it printsWAIVED <test>: <why>, andmake gpu-testends
INCOMPLETEinstead ofok. The manifest is re-pinned on MLX 0.32.3 and
mlx-c 0.7.0, and the steps for moving the pin re-pin it.make spec-round-stream-selftestis part ofmake ci. -
The request time cap ends a reply that has started to stream. The cap
(--max-timeout-secs,X-Request-Timeout-Seconds) bounded only the time to
the head of the reply. A chat, messages or transcription stream then ran on
until its generation ended, with no clock. The cap now counts from the arrival
of the request and ends the stream at the deadline, with no terminal item, and
the server logs onewarnand counts onetimeouterror. A client that
keeps its connection open and reads nothing no longer holds the generation,
the GPU permit and the generation lock past the cap. The same end now serves
the shutdown: a reply that streams to a client that reads nothing ended
only when the client read or the process exited, and now ends at the reply
bound.--max-timeout-secs 0still sets no cap. A reply of a known length is
not changed. A client that reads a stream to its end must look for the
terminal item ([DONE],finish_reason,message_stopor
transcript.text.done) to know that it has the whole answer. -
The speculative equivalence gate excuses a short arm that parted at a near
tie. The gate refused every speculative arm shorter than the answer floor
while the reference ran on. That rule was written for an arm cut short with
the reference's own text. A verifier can part from the reference at a near
tie and then end sooner on its own stop id, and the gate refused that run.
Such an arm is now unjudgeable, the mirror of the rule for a short
reference. A cell may take this excuse on at most one prompt. No threshold
moves.docs/SPEC_ANSWER_EQUIVALENCE.mdnames the class of defect that this
gives up, and the broken-engine counts there are re-read under the new rule. -
The long-context round streams of the recurrent-verifier pairs are
re-pinned. The single prefill plan below moves the state that enters round
0 by bf16 noise on prompts long enough to reach an anchor. Thelongctx-4k
streams of the adaptive, block, recurrent, restricted-vocabulary and
two-model cells moved; the other 31 pinned cells are unchanged. The
prefill-boundary tests of EAGLE-3 and MTP derive their counts from the
engine's split plan.
Fixed
-
A speculative request that asks for
logprobsgets them. The response
carried nologprobskey. Every drafter path now reports one record per
emitted token: the verifier's raw log-softmax at the position that produced
the token, the same quantity the plain path reports. At temperature 0 the top
entry is the emitted token. A request that asks for none keeps no row and
reads nothing back. A temperature-0 EAGLE-3 request that asks for logprobs
scores the full vocabulary, and it can emit a token that the same request
without logprobs would not (docs/SPECULATIVE.md). -
A repeated prompt that asks for
logprobsgets one record per token on
Qwen3.5, Gemma4, Gemma3 and BitNet. An exact prompt-cache hit replays the
cached first token without recomputing its logits, and only Qwen3 stored that
token's record, so the other four returned one record fewer than tokens.
Each now stores the record in the entry and replays it cut to the request's
top_logprobs. Laguna, Qwen2 and Qwen3-VL-MoE emit no logprobs on any path,
a miss included. An entry that comes back from the SSD tier is never an exact
hit; its repeat prefills again. Qwen3, Gemma3, Gemma4 and BitNet no longer
clone and evaluate the whole KV snapshot on a miss when the cache cannot
store the entry (no cache slots, or paged KV). -
An assistant message's
reasoning_contentreaches the chat template.
/v1/chat/completionsdropped the field, so a template that reads it
(Qwen3.8, Ternary-Bonsai-2) rendered every earlier assistant turn with an
empty think block, and the prompt of turn N+1 never extended the prompt of
turn N. The field now travels as the client sent it, on any role. Absent and
nullstay absent, and an empty string is defined and empty. It counts
toward the input bound likecontent./v1/messageshas no such field. -
A reused or exact-hit prompt-cache entry is bound to the context ceiling of
the new request. The entry kept the ceiling of the request that stored it.
Gemma4, Qwen3.5, Qwen3 and Qwen3-VL-MoE now resolve the context before the
cache lookup and rebind the cache to the new ceiling; a prompt above it is
the typedKvCeilingExceeded. -
An SSD-tier hit on a Qwen3.5-family model no longer feeds tokens twice.
The spill wrote an entry's caches at the prompt length L, and the hydrate
restored them at L but labelled the entry with the block-aligned prefix, so
the tokens between that prefix and L entered the K/V and the recurrent state
twice, with no error. A hydrated block is now brought to exactly the prefix
its digest names: the K/V past the claim is cut, together with the bf16
decode seed, and a recurrent state is reused only when it sits at the claim.
A block that records no position is a miss. An unusable block is deleted by
its row only if the row still has the size andlast_usedit was read with,
so a spill of the same key that landed since keeps its block.
SCHEMA_VERSIONis 9: the SSD cache of an older version is wiped on first
open and starts empty after the upgrade. -
Two first opens of one SSD index both succeed.
SsdKvIndex::open_at
failed for the loser of two concurrent first opens of a namespace: SQLite
does not call the busy handler for the switch to WAL, and the choice between
a new and an existing file readPath::exists(), so the loser saw the
winner's file and refused it as schema version 1. A spiller thread that lost
the race logged one warning and then dropped every spill job for the life of
the process. The pragmas now retry onSQLITE_BUSYup to one busy-wait
bound, and the schema is created or checked in oneBEGIN IMMEDIATE
transaction, decided by which tables are present. A 0-byte index file left by
a crash opens as new. A failed index open in the spiller thread, or a failed
hydrator open at start-up, is anerror!event that names the namespace. -
rmlx eval pplandrmlx baseline --recordrecord any model namespace and
refuse a bad identity before the work.rmlx eval pplscored a whole
corpus for 18 minutes and then refused to record, because the namespace of
the snapshot was not on a fixed list. The namespace is a free-form label
likekv_quant, refused only when empty. The command now resolves the
namespace, the model and the checked weight format before any work, so a
refusal costs seconds. The ingest scripts take the snapshot directory name
only, never a parent directory, and file a row as<namespace>/<model>.
Rows that an earlier release filed aslocal/<name>for an unlisted
namespace stay as they are, in a separatebestscell. -
A speculative round that does not advance the verifier by what it fed is an
error. The loop read the position of the KV stack before and after the
verify forward as the largest offset over the layers and never compared the
two reads. The stack position now comes from one producer that refuses a
stack whose advancing layers disagree and names two of them. After every
verify forward a round whose new position is not the old one plus the rows
fed is refused. Both are plain errors that hold underrelease-perf, read
host-side integers only, and add no evaluation per round. A two-model pair
asked only the verifier whether a KV codec is legal, so a draft model that
refuses the codec was built at it: the server reachedreadywith a Qwen3.5
MoE draft model atk_iso4. Both models are now asked at start-up, per
request and before the draft model's cache is built. -
Bench scripts attribute a shared run log by pid and file a row under the
commit of the run identity. The run id has a granularity of one second, so
anrmlxprocess that started in the same second as the previous one
appended to its log.bench_server.shkept only log files that were new
since its snapshot, did not find the server's log, and lost the run after
its first arm had been filed. A log that is new or has grown is now a
candidate, the one with anrmlx startevent from the server's pid wins, and
a phase whose bytes carry another pid's start is refused.
scripts/spec_bench.shoverwrote thegit_shaof the identity with the
checkout's HEAD, so a binary measured from a checkout at another commit was
filed under the wrong commit. The commit is now the identity's, shortened
when it resolves in the checkout, else the checkout's HEAD, elseunknown,
in the six scripts that file a row. -
A stop-string match ends the generation. The engine did not know the
stopandstop_sequencesstrings. After a match it ran tomax_tokensor
the end-of-turn token, counted every step inusage, and both streams sent
their finish events only when it stopped. The route now sets
end_requestedat a match and the decode ends on it as at an end-of-turn
token, on every decode loop, plain and speculative. The route reads on to the
terminal item,usagecounts the steps that ran (at temperature 0, at most
10 past the match), and a failure after the match starts no replay. -
One stop rule on all four chat routes. The routes disagreed about tool
calls: the non-streamed routes did not truncate a reply that held a tool
call, and the OpenAI stream stopped matching after one. Matching now runs on
the content channel from the first token, after the tool parser, whether or
not a tool call came before it. It does not match reasoning or the arguments
of a tool call. After a match the reply keeps no text, reasoning or tool call
that follows. A kept tool call givestool_calls(OpenAI) ortool_use
(Anthropic,stop_sequencenull); otherwise a match givesstopor
stop_sequencewith the string. -
/v1/messagesrecovers a tool call cut off bymax_tokens. The route
never called the tool parser'sfinalize, so the end-of-generation recovery
that/v1/chat/completionsruns did not run. The tail that the parser held
back as the start of a marker that never completed (<to) was lost on both
routes; it is now text. -
/v1/messagesrenders the earlierthinking,tool_useand
tool_resultblocks. The route kept thetextblocks of a message and
dropped the rest. A chat template saw an earlier turn with an empty think
block, and no tool call and no tool result of a multi-turn tool loop. The
route now gives the template the thinking text, each tool call and each tool
result as the client sent them, as the OpenAI route does. The input-size
bound counts them.input_audioblocks of ausermessage, which the route
size-checked and then dropped, reach the audio tower.signature,
redacted_thinking,is_errorand the image blocks of a tool result have no
template field and are not rendered. -
The input bounds count every string the template renders. The size bound
on both chat routes counted the text only. It now counts also thesystem
prompt of/v1/messages, the call id, the name, the tool-call arguments and
thetoolsarray. On/v1/messages, the message cap counts the rendered
messages: eachtool_resultblock is one. -
A chat template that fails to render is 400 or 500, not 503. A template
that failed on a request gave a retryable 503service_unavailable. An error
while it renders the request's values (for exampletool_callsarguments
that are not an object) is now 400invalid_request_errorwith the error, on
both chat routes. A failure that does not depend on the request (a missing
compiled template, or a filter, test or function the engine does not have) is
500internal_errorwith a fixed message, and the log holds the error. -
A generation that reaches the context ceiling stops there and keeps its
output. The route compared only the prompt with the ceiling. When the
prompt and the generated tokens reached it, the next decode step failed. A
blocking request lost every token. A streamed request got an error event
and no finish reason. The engine now runs at most as many decode steps as
the ceiling leaves after the prompt, for every architecture, for the
speculative round loops and for the image paths, and the reply ends with
finish_reason: "length"(stop_reason: "max_tokens"on/v1/messages).
A prompt that leaves no room is still a 400context_length_exceeded.
BitNet, Qwen2, Gemma3 and Laguna had no ceiling on a generation; they now
stop at the resolved--max-ctx. -
A cold prefill of a recurrent architecture takes one split plan. The
engine cut the prefill at the 256-token anchor, where it saves the recurrent
state, only when the request could store a cache entry. So the output of a
cold request at temperature 0 depended on--prompt-cache-slots,
--paged-kvand on whether the request took a speculative path, and the
plain and speculative arms of an equivalence pair prefilled in different
pieces.prefill_chunk::prefill_piecesis now the one producer of the plan.
The plain path, the resume tail, every speculative prefill and the capturing
verifier prefill of DFlash 2 and EAGLE-3 cut at the anchor whether or not
anything is stored. Storing only decides whether the state is copied.
Architectures without recurrent state are unchanged, and the perplexity
scorer stays outside the plan so that its rows stay comparable.
Performance
- The padded prefill mask is built once per forward. A prefill-sized
attention call whose query rows the row rule pads copied the mask to the
padded rows on every layer: about 469 MB per padded layer on Qwen3.8-27B at
130,790 tokens. The Qwen3.5-family prefill mask is now built once, at the
rows of the attention node and in the query dtype, andsdpa_undertakes a
mask that is already at the padded rows as it is. On a Qwen3.8-shaped layer
at 1,766 query rows the copying call needed 57,720,832 B more peak than the
pre-padded call at 16,102 keys and 116,441,088 B more at 32,486. On
Qwen3.8-27B-4bit the token ids at 7,910, 8,192 and 32,486 tokens are
identical, the generation allocation is equal, and the prefill rate is
within 1%.
Documentation
-
docs/SPEC_READING_A_RUN.mdstates what a round costs. Each extra fed row
costs about 0.095 of a plain step on Qwen3.8-27B-4bit and 0.113 on
gemma-4-e4b-it-mxfp8, from the quantised matmul itself, with a step between 5
and 6 rows. A speed-up is the tokens per round over the round cost in plain
steps, and the round cost differs by weight format. The doc also holds what
SPECULATIVE.md§ "Reading a run" held, split out for size. -
docs/SERVER_ANTHROPIC.mdholds the Anthropic routes. The/v1/messages
fields, message blocks, input bounds, response and stream shape, and the
stop_reasonmapping moved out ofdocs/SERVER.md, which was over the size
cap.SERVER.mdkeeps a pointer. -
Four claims about kernels and devices are corrected. The dense checkpoint
is not exact across forward lengths: a several-row verify forward and a
one-row decode step can take different MLX kernels
(docs/SPEC_ANSWER_EQUIVALENCE.md,docs/PROMPT_CACHE.md). The MoE
broadcast and sorted routes agree in exact arithmetic, not in bf16.
--device cpuis a correctness device with no speed promise, because the CPU
quantized matmul of MLX uses one thread per stream and has no SIMD path for a
bf16 activation (docs/CLI.md). The refusal of a 1-bit affine model now says
that it needs an MLX with 1-bit affine kernels, MLX 0.32.4 or later, and no
longer calls the kernel unreleased.
Dependencies
webpki-root-certs1.0.7 -> 1.0.10 inCargo.lock, for RUSTSEC-2026-0383:
the bundled root certificates allowed trust in distrusted emSign
intermediates.
Retrospective
- #718: ci(debt-report): the file-pair scan pairs files inside one directory only, so three per-arch generate.rs files at 57% to 73% never reach the report
- #719: test(refactor): fourteen test files implement tracing::Subscriber to capture events; 0.4.4 added eight and none is shared
- #720: server(refactor): the OpenAI and Anthropic routes each carry their own engine-error map and token count; 0.4.4 made the same edit in both
- #721: docs: cut the five docs that 0.4.4 grew or added to current truth (AUDIO, AUDIO_TTS, AUDIO_WAV, SERVER_SHUTDOWN, WEIGHT_QUANTS)
- #722: docs(releasing): the retrospective sets PREV from the nearest tag of any name, so a local tag gives the wrong range