Skip to content

rMLX 0.4.5

Latest

Choose a tag to compare

@Pushkinist Pushkinist released this 23 Oct 19:33
· 3 commits to main since this release

A Qwen3.5-family request can now resume from a saved recurrent state. Every
request writes one prefill event.

Most other changes are fixes. A stop string now ends the decode on every path.
A generation that reaches the context ceiling stops there and keeps its
output. A speculative request that asks for logprobs gets them. A drafter
whose width does not match its verifier is refused.

Four changes to read before upgrading: prefill_ms moved off decode_profile
and the speculative done line to the new generate: prefill event
(Added); a speculative drafter whose hidden size, layer count or
vocabulary size does not match its verifier is refused, and an EAGLE-3
config.json must declare hidden_size and vocab_size (Changed); the
request time cap ends a reply that has started to stream (Changed); the SSD
index schema version is 9, so the SSD cache of an older version is wiped at the
first open (Fixed).

Added

  • Every request writes one generate: prefill event. It carries the
    prompt tokens, the tokens a prompt cache supplied, the tokens the prefill ran
    the model over and the prefill time, on every architecture and every
    speculative drafter. It is the one producer of the prefill time: decode_profile
    and the speculative done line no longer carry prefill_ms. An exact cache hit reports its whole prompt as reused and
    a prefill time of zero. Before, only some paths wrote a prefill time, in
    decode_profile, and no event said how many tokens the prefill ran over. See docs/PROFILING.md § "The prefill event of a request".

  • The published-protocol harness reports three more columns. The fixed
    prompt's input speed now has two definitions, the prompt over the time to
    first token and the engine's own prefill rate (prefill_engine_tps, off the
    event above); the engine rate is refused on a prompt that was not cold. A
    speculative run can be judged against a plain run at temperature 0 with
    --greedy-pass and --greedy-against, through the divergence oracle, and
    the table counts identical, near-tie, refused and unjudgeable answers by
    dataset. --longctx 32k measures a checked-in long prompt on the plain arm.
    None of these runs by default. ingest/published_ingest.py records the
    long-context row and the per-dataset greedy counts (new greedy_* registry
    metrics), and the re-query after it checks them.

  • A Qwen3.5-family request resumes from a saved recurrent state. A
    prompt-cache entry of Qwen3.8 or Ternary-Bonsai-2 served an exact replay
    only: the recurrent state cannot be cut back, so a longer or diverging prompt
    prefilled from token 0. Two shapes now resume. A request that extends the
    whole stored prompt resumes at the stored length. An entry also keeps one
    saved recurrent state at the last full 256-token block that does not reach
    the end of its prompt, and a request whose first differing token is at or
    after that block resumes there, with the K/V cut to it. An entry whose stores
    refuse the cut does not offer it. The saved state counts in the entry's
    kv_bytes: at 4,096 tokens on Qwen3.8-27B-4bit the entry grows by 36%
    (153,944,064 B). A cold prefill pays one clone of the state, 0.01% of the
    time to first token at 4,096 tokens and 0.09% at 32,768. At 32,768 tokens
    the entry is above the default 2 GiB RAM cap of the prompt cache, so it is
    not stored and the clone buys nothing. The engine now judges every slot that
    shares a block, best match first, and the longest reuse any slot allows
    serves. The SSD tier spills the anchor of such an entry, so the tier serves
    these prompts across restarts; before, an entry whose prompt ended inside a
    block was withheld. See docs/PROMPT_CACHE.md and docs/SSD_TIER.md.

  • rmlx eval spec-answer --plain F --spec F judges a speculative answer
    against a plain one.
    It reads two non-streamed chat-completion bodies that
    carry logprobs, applies the divergence-confidence oracle of
    docs/SPEC_ANSWER_EQUIVALENCE.md and prints the verdict, the position, the
    margin, the rank and the reason. Exit 0 is a near tie, 1 a divergence the
    oracle refuses, 2 an input it cannot judge. It loads no model, takes no
    claim and opens no metrics recorder. The oracle moved from the equivalence
    test to rmlx_models::answer_equivalence, so the test and the command share
    one rule.

  • The published-protocol harness reports the Metal peak, a perplexity section
    and labelled campaigns.
    scripts/spec_bench_published.sh reports the
    Metal allocator peak beside the physical footprint of the fixed-length
    prompt. make published-table renders a perplexity section from an
    rmlx eval ppl record of the same checkpoint and corpus. A run needs a
    --label and files its raw material in its own campaign directory under
    <RMLX_HOME>/bench/published/<label>/. Every row it ingests carries the
    label in its notes. --record ends by re-querying the rows it wrote, and it
    refuses a result that they do not reproduce. The old fixed result directory
    and the fixed staging path of the ingest are gone.

Changed

  • scripts/spec_bench.sh files a near-tie flip and still stops on a real
    divergence.
    The script stopped the run when a speculative arm's answer
    differed from the plain arm's in any byte. Greedy verification emits the
    verifier's own argmax, but a near tie in bf16 can flip between two equally
    right streams, so every Qwen3.8-27B 4-bit arm that met one was refused. After
    its warmup each phase now sends one unstreamed request with logprobs and
    top_logprobs 2, binds the answer to its arm's digest, and pauses 15 s. A
    difference with no sampler is judged by rmlx eval spec-answer. A near tie
    is filed as answer_check=near_tie@<token>, with divergence_rank and
    divergence_margin in the notes. A divergence the oracle refuses, an arm
    that ran past or stopped far short of the reference, measured speculative
    runs that differ from each other, a pass that does not bind to its digest,
    an unanswered pass and a judge whose exit status does not match its verdict
    stop the run with no row. When its digest differs, EAGLE-3 is refused unless
    its logprob pass binds.

  • Every speculative drafter is tied to its verifier by one pairing check.
    The MTP sidecar was matched by the hidden size alone, DFlash and DFlash 2
    never read the layer count of the base they were built for, and EAGLE-3 and
    the Gemma4 assistant each kept a check of their own, so a drafter built for
    another base loaded without a refusal. The hidden size, the layer count and
    the vocabulary size that the drafter's config declares are now compared with
    the verifier's in one place for all five drafters. A mismatch is
    Error::SpeculativePairing and names the key and both values. A config that
    declares none of the keys loads with a warning that names the drafter
    directory. Behaviour change: an EAGLE-3 config.json must declare
    hidden_size and vocab_size; the fallbacks 2048 and 248320 are gone. The
    Gemma4 assistant is checked by the widths of pre_projection.weight and
    post_projection.weight, so one without backbone_hidden_size is still
    refused on a verifier of another width. No config key names the base model,
    so two bases with the same trunk shape cannot be told apart
    (docs/SPECULATIVE.md).

  • The test targets are gemma-4-e4b, Qwen3.8-27B-4bit and
    Ternary-Bonsai-2-27B.
    scripts/perf_canary.sh and scripts/perf_ab.sh
    default to them, as do make model-check-full, perf-iter and the
    autoresearch defaults. The first-generation Bonsai, Qwen3.6 and medgemma
    cells stay in the tree and run under make gpu-test. The codec-matrix
    runner reads each command's own exit status (it read the status of
    || true, so a held Metal claim failed a row as an output mismatch), ends
    INCOMPLETE with exit 3, and exits 1 for a failed row or a run that ran
    nothing. docs/TESTING.md lists the targets and the features no target
    covers.

  • A moved round-stream cell is a failure only on the pinned MLX pair. Each
    of the six pinned speculative equivalence tests checks its own per-round
    cells against MANIFEST.sha256 after it has judged the answers. On another
    MLX pair it prints WAIVED <test>: <why>, and make gpu-test ends
    INCOMPLETE instead of ok. The manifest is re-pinned on MLX 0.32.3 and
    mlx-c 0.7.0, and the steps for moving the pin re-pin it. make spec-round-stream-selftest is part of make ci.

  • The request time cap ends a reply that has started to stream. The cap
    (--max-timeout-secs, X-Request-Timeout-Seconds) bounded only the time to
    the head of the reply. A chat, messages or transcription stream then ran on
    until its generation ended, with no clock. The cap now counts from the arrival
    of the request and ends the stream at the deadline, with no terminal item, and
    the server logs one warn and counts one timeout error. A client that
    keeps its connection open and reads nothing no longer holds the generation,
    the GPU permit and the generation lock past the cap. The same end now serves
    the shutdown: a reply that streams to a client that reads nothing ended
    only when the client read or the process exited, and now ends at the reply
    bound. --max-timeout-secs 0 still sets no cap. A reply of a known length is
    not changed. A client that reads a stream to its end must look for the
    terminal item ([DONE], finish_reason, message_stop or
    transcript.text.done) to know that it has the whole answer.

  • The speculative equivalence gate excuses a short arm that parted at a near
    tie.
    The gate refused every speculative arm shorter than the answer floor
    while the reference ran on. That rule was written for an arm cut short with
    the reference's own text. A verifier can part from the reference at a near
    tie and then end sooner on its own stop id, and the gate refused that run.
    Such an arm is now unjudgeable, the mirror of the rule for a short
    reference. A cell may take this excuse on at most one prompt. No threshold
    moves. docs/SPEC_ANSWER_EQUIVALENCE.md names the class of defect that this
    gives up, and the broken-engine counts there are re-read under the new rule.

  • The long-context round streams of the recurrent-verifier pairs are
    re-pinned.
    The single prefill plan below moves the state that enters round
    0 by bf16 noise on prompts long enough to reach an anchor. The longctx-4k
    streams of the adaptive, block, recurrent, restricted-vocabulary and
    two-model cells moved; the other 31 pinned cells are unchanged. The
    prefill-boundary tests of EAGLE-3 and MTP derive their counts from the
    engine's split plan.

Fixed

  • A speculative request that asks for logprobs gets them. The response
    carried no logprobs key. Every drafter path now reports one record per
    emitted token: the verifier's raw log-softmax at the position that produced
    the token, the same quantity the plain path reports. At temperature 0 the top
    entry is the emitted token. A request that asks for none keeps no row and
    reads nothing back. A temperature-0 EAGLE-3 request that asks for logprobs
    scores the full vocabulary, and it can emit a token that the same request
    without logprobs would not (docs/SPECULATIVE.md).

  • A repeated prompt that asks for logprobs gets one record per token on
    Qwen3.5, Gemma4, Gemma3 and BitNet.
    An exact prompt-cache hit replays the
    cached first token without recomputing its logits, and only Qwen3 stored that
    token's record, so the other four returned one record fewer than tokens.
    Each now stores the record in the entry and replays it cut to the request's
    top_logprobs. Laguna, Qwen2 and Qwen3-VL-MoE emit no logprobs on any path,
    a miss included. An entry that comes back from the SSD tier is never an exact
    hit; its repeat prefills again. Qwen3, Gemma3, Gemma4 and BitNet no longer
    clone and evaluate the whole KV snapshot on a miss when the cache cannot
    store the entry (no cache slots, or paged KV).

  • An assistant message's reasoning_content reaches the chat template.
    /v1/chat/completions dropped the field, so a template that reads it
    (Qwen3.8, Ternary-Bonsai-2) rendered every earlier assistant turn with an
    empty think block, and the prompt of turn N+1 never extended the prompt of
    turn N. The field now travels as the client sent it, on any role. Absent and
    null stay absent, and an empty string is defined and empty. It counts
    toward the input bound like content. /v1/messages has no such field.

  • A reused or exact-hit prompt-cache entry is bound to the context ceiling of
    the new request.
    The entry kept the ceiling of the request that stored it.
    Gemma4, Qwen3.5, Qwen3 and Qwen3-VL-MoE now resolve the context before the
    cache lookup and rebind the cache to the new ceiling; a prompt above it is
    the typed KvCeilingExceeded.

  • An SSD-tier hit on a Qwen3.5-family model no longer feeds tokens twice.
    The spill wrote an entry's caches at the prompt length L, and the hydrate
    restored them at L but labelled the entry with the block-aligned prefix, so
    the tokens between that prefix and L entered the K/V and the recurrent state
    twice, with no error. A hydrated block is now brought to exactly the prefix
    its digest names: the K/V past the claim is cut, together with the bf16
    decode seed, and a recurrent state is reused only when it sits at the claim.
    A block that records no position is a miss. An unusable block is deleted by
    its row only if the row still has the size and last_used it was read with,
    so a spill of the same key that landed since keeps its block.
    SCHEMA_VERSION is 9: the SSD cache of an older version is wiped on first
    open and starts empty after the upgrade.

  • Two first opens of one SSD index both succeed. SsdKvIndex::open_at
    failed for the loser of two concurrent first opens of a namespace: SQLite
    does not call the busy handler for the switch to WAL, and the choice between
    a new and an existing file read Path::exists(), so the loser saw the
    winner's file and refused it as schema version 1. A spiller thread that lost
    the race logged one warning and then dropped every spill job for the life of
    the process. The pragmas now retry on SQLITE_BUSY up to one busy-wait
    bound, and the schema is created or checked in one BEGIN IMMEDIATE
    transaction, decided by which tables are present. A 0-byte index file left by
    a crash opens as new. A failed index open in the spiller thread, or a failed
    hydrator open at start-up, is an error! event that names the namespace.

  • rmlx eval ppl and rmlx baseline --record record any model namespace and
    refuse a bad identity before the work.
    rmlx eval ppl scored a whole
    corpus for 18 minutes and then refused to record, because the namespace of
    the snapshot was not on a fixed list. The namespace is a free-form label
    like kv_quant, refused only when empty. The command now resolves the
    namespace, the model and the checked weight format before any work, so a
    refusal costs seconds. The ingest scripts take the snapshot directory name
    only, never a parent directory, and file a row as <namespace>/<model>.
    Rows that an earlier release filed as local/<name> for an unlisted
    namespace stay as they are, in a separate bests cell.

  • A speculative round that does not advance the verifier by what it fed is an
    error.
    The loop read the position of the KV stack before and after the
    verify forward as the largest offset over the layers and never compared the
    two reads. The stack position now comes from one producer that refuses a
    stack whose advancing layers disagree and names two of them. After every
    verify forward a round whose new position is not the old one plus the rows
    fed is refused. Both are plain errors that hold under release-perf, read
    host-side integers only, and add no evaluation per round. A two-model pair
    asked only the verifier whether a KV codec is legal, so a draft model that
    refuses the codec was built at it: the server reached ready with a Qwen3.5
    MoE draft model at k_iso4. Both models are now asked at start-up, per
    request and before the draft model's cache is built.

  • Bench scripts attribute a shared run log by pid and file a row under the
    commit of the run identity.
    The run id has a granularity of one second, so
    an rmlx process that started in the same second as the previous one
    appended to its log. bench_server.sh kept only log files that were new
    since its snapshot, did not find the server's log, and lost the run after
    its first arm had been filed. A log that is new or has grown is now a
    candidate, the one with an rmlx start event from the server's pid wins, and
    a phase whose bytes carry another pid's start is refused.
    scripts/spec_bench.sh overwrote the git_sha of the identity with the
    checkout's HEAD, so a binary measured from a checkout at another commit was
    filed under the wrong commit. The commit is now the identity's, shortened
    when it resolves in the checkout, else the checkout's HEAD, else unknown,
    in the six scripts that file a row.

  • A stop-string match ends the generation. The engine did not know the
    stop and stop_sequences strings. After a match it ran to max_tokens or
    the end-of-turn token, counted every step in usage, and both streams sent
    their finish events only when it stopped. The route now sets
    end_requested at a match and the decode ends on it as at an end-of-turn
    token, on every decode loop, plain and speculative. The route reads on to the
    terminal item, usage counts the steps that ran (at temperature 0, at most
    10 past the match), and a failure after the match starts no replay.

  • One stop rule on all four chat routes. The routes disagreed about tool
    calls: the non-streamed routes did not truncate a reply that held a tool
    call, and the OpenAI stream stopped matching after one. Matching now runs on
    the content channel from the first token, after the tool parser, whether or
    not a tool call came before it. It does not match reasoning or the arguments
    of a tool call. After a match the reply keeps no text, reasoning or tool call
    that follows. A kept tool call gives tool_calls (OpenAI) or tool_use
    (Anthropic, stop_sequence null); otherwise a match gives stop or
    stop_sequence with the string.

  • /v1/messages recovers a tool call cut off by max_tokens. The route
    never called the tool parser's finalize, so the end-of-generation recovery
    that /v1/chat/completions runs did not run. The tail that the parser held
    back as the start of a marker that never completed (<to) was lost on both
    routes; it is now text.

  • /v1/messages renders the earlier thinking, tool_use and
    tool_result blocks.
    The route kept the text blocks of a message and
    dropped the rest. A chat template saw an earlier turn with an empty think
    block, and no tool call and no tool result of a multi-turn tool loop. The
    route now gives the template the thinking text, each tool call and each tool
    result as the client sent them, as the OpenAI route does. The input-size
    bound counts them. input_audio blocks of a user message, which the route
    size-checked and then dropped, reach the audio tower. signature,
    redacted_thinking, is_error and the image blocks of a tool result have no
    template field and are not rendered.

  • The input bounds count every string the template renders. The size bound
    on both chat routes counted the text only. It now counts also the system
    prompt of /v1/messages, the call id, the name, the tool-call arguments and
    the tools array. On /v1/messages, the message cap counts the rendered
    messages: each tool_result block is one.

  • A chat template that fails to render is 400 or 500, not 503. A template
    that failed on a request gave a retryable 503 service_unavailable. An error
    while it renders the request's values (for example tool_calls arguments
    that are not an object) is now 400 invalid_request_error with the error, on
    both chat routes. A failure that does not depend on the request (a missing
    compiled template, or a filter, test or function the engine does not have) is
    500 internal_error with a fixed message, and the log holds the error.

  • A generation that reaches the context ceiling stops there and keeps its
    output.
    The route compared only the prompt with the ceiling. When the
    prompt and the generated tokens reached it, the next decode step failed. A
    blocking request lost every token. A streamed request got an error event
    and no finish reason. The engine now runs at most as many decode steps as
    the ceiling leaves after the prompt, for every architecture, for the
    speculative round loops and for the image paths, and the reply ends with
    finish_reason: "length" (stop_reason: "max_tokens" on /v1/messages).
    A prompt that leaves no room is still a 400 context_length_exceeded.
    BitNet, Qwen2, Gemma3 and Laguna had no ceiling on a generation; they now
    stop at the resolved --max-ctx.

  • A cold prefill of a recurrent architecture takes one split plan. The
    engine cut the prefill at the 256-token anchor, where it saves the recurrent
    state, only when the request could store a cache entry. So the output of a
    cold request at temperature 0 depended on --prompt-cache-slots,
    --paged-kv and on whether the request took a speculative path, and the
    plain and speculative arms of an equivalence pair prefilled in different
    pieces. prefill_chunk::prefill_pieces is now the one producer of the plan.
    The plain path, the resume tail, every speculative prefill and the capturing
    verifier prefill of DFlash 2 and EAGLE-3 cut at the anchor whether or not
    anything is stored. Storing only decides whether the state is copied.
    Architectures without recurrent state are unchanged, and the perplexity
    scorer stays outside the plan so that its rows stay comparable.

Performance

  • The padded prefill mask is built once per forward. A prefill-sized
    attention call whose query rows the row rule pads copied the mask to the
    padded rows on every layer: about 469 MB per padded layer on Qwen3.8-27B at
    130,790 tokens. The Qwen3.5-family prefill mask is now built once, at the
    rows of the attention node and in the query dtype, and sdpa_under takes a
    mask that is already at the padded rows as it is. On a Qwen3.8-shaped layer
    at 1,766 query rows the copying call needed 57,720,832 B more peak than the
    pre-padded call at 16,102 keys and 116,441,088 B more at 32,486. On
    Qwen3.8-27B-4bit the token ids at 7,910, 8,192 and 32,486 tokens are
    identical, the generation allocation is equal, and the prefill rate is
    within 1%.

Documentation

  • docs/SPEC_READING_A_RUN.md states what a round costs. Each extra fed row
    costs about 0.095 of a plain step on Qwen3.8-27B-4bit and 0.113 on
    gemma-4-e4b-it-mxfp8, from the quantised matmul itself, with a step between 5
    and 6 rows. A speed-up is the tokens per round over the round cost in plain
    steps, and the round cost differs by weight format. The doc also holds what
    SPECULATIVE.md § "Reading a run" held, split out for size.

  • docs/SERVER_ANTHROPIC.md holds the Anthropic routes. The /v1/messages
    fields, message blocks, input bounds, response and stream shape, and the
    stop_reason mapping moved out of docs/SERVER.md, which was over the size
    cap. SERVER.md keeps a pointer.

  • Four claims about kernels and devices are corrected. The dense checkpoint
    is not exact across forward lengths: a several-row verify forward and a
    one-row decode step can take different MLX kernels
    (docs/SPEC_ANSWER_EQUIVALENCE.md, docs/PROMPT_CACHE.md). The MoE
    broadcast and sorted routes agree in exact arithmetic, not in bf16.
    --device cpu is a correctness device with no speed promise, because the CPU
    quantized matmul of MLX uses one thread per stream and has no SIMD path for a
    bf16 activation (docs/CLI.md). The refusal of a 1-bit affine model now says
    that it needs an MLX with 1-bit affine kernels, MLX 0.32.4 or later, and no
    longer calls the kernel unreleased.

Dependencies

  • webpki-root-certs 1.0.7 -> 1.0.10 in Cargo.lock, for RUSTSEC-2026-0383:
    the bundled root certificates allowed trust in distrusted emSign
    intermediates.

Retrospective

  • #718: ci(debt-report): the file-pair scan pairs files inside one directory only, so three per-arch generate.rs files at 57% to 73% never reach the report
  • #719: test(refactor): fourteen test files implement tracing::Subscriber to capture events; 0.4.4 added eight and none is shared
  • #720: server(refactor): the OpenAI and Anthropic routes each carry their own engine-error map and token count; 0.4.4 made the same edit in both
  • #721: docs: cut the five docs that 0.4.4 grew or added to current truth (AUDIO, AUDIO_TTS, AUDIO_WAV, SERVER_SHUTDOWN, WEIGHT_QUANTS)
  • #722: docs(releasing): the retrospective sets PREV from the nearest tag of any name, so a local tag gives the wrong range