Skip to content

BaseRT engine 0.4.4

Choose a tag to compare

@prabod prabod released this 03 Nov 15:38
23bddc5

BaseRT 0.4.4

New

  • Speculative decoding for Qwen3.8-27B and Qwen3.6-35B-A3B on Apple Silicon
    with the DFlash 2 block drafters from Inco AI, converted and paired in the
    catalog. Pull the target and its drafter, then serve with the drafter named
    or with auto, which takes it once it is installed:

    basert pull basecompute/Qwen3.8-27B
    basert pull basecompute/Qwen3.8-27B-DFlash2
    basert serve basecompute/Qwen3.8-27B --speculate dflash:basecompute/Qwen3.8-27B-DFlash2

    The same for basecompute/Qwen3.6-35B-A3B and its
    Qwen3.6-35B-A3B-DFlash2. On an M5 Pro the 27B at Q4 decodes 2.4 times
    faster than plain at one request with the defaults, and 1.9 times at four;
    with the tiled verify switched on (BASERT_TILED_VERIFY=2) 2.8 times at one
    request, and with speculative-sampling acceptance for sampled requests
    (BASERTD_SPEC_EXACT_ACCEPT=1, exact in distribution) 3.1 times. The
    35B-A3B on a 32 GB M6 mini decodes 1.5 / 1.3 / 1.3 times faster at one, two
    and four requests. Greedy requests reproduce plain decoding's text. The
    tiled verify and the exact rule stay opt-in; the switches and what each one
    costs are in OPERATIONS.md under "Speculative decoding policy".

    When running superfluid directly rather than through basert serve, name
    the drafter by its .base file: --speculate dflash:<path to the drafter's model.base>.

Changed

  • Gemma 4 prompts served from a paged KV cache now prefill at the speed
    basert bench measures on a contiguous one. On an M5 Pro a 13.6K-token
    prompt on E2B Q8 goes from 1,046 to 11,409 tokens per second through
    superfluid, E4B Q8 from 860 to 4,011, and 26B-A4B Q4 from 650 to 1,961.
    Served decode is unchanged.
  • The speculation policy under several concurrent requests. A request that a
    gate takes out of speculation (a prompt that repeats little, a price below
    plain decoding, a low acceptance) decodes plainly only while it is the only
    request, and drafts along with the others otherwise, since riding their
    rounds plainly costs more than drafting. A strategy's requests share one
    draft depth per round. A request count the throughput gate has not judged
    yet runs on a wider count's verdict only until it has evidence of its own.
    A request below the acceptance floor keeps speculating while its own rate
    still beats plain decoding at that request count.
  • Bundles converted from GGUF now declare tensor shapes in the same order as
    every other source. Bundles converted before this release keep working and
    give the same output; for a model whose key and value projections are
    narrower than its query projection, re-converting restores the fused
    query/key/value attention.

Fixed

  • On Apple Silicon, logits computed for every position of a 16,384-token
    prefill chunk, which Qwen3.6-35B-A3B takes for long prompts, came out
    zero past the 4,324th position of the chunk: the tensor-core matrix
    kernels addressed each output tile with a 32-bit offset, which wrapped
    once the chunk's logits passed 2 GiB. Perplexity and divergence scores
    over such chunks were wrong (one quantization run measured 18 times the
    expected divergence); the tiles are now addressed in 64-bit.
  • A served prompt of 512 to 4,096 tokens whose length was 1 to 32 past a
    multiple of 64 ran its first chunk without a causal mask on the full-
    attention layers of Qwen3.5 and Qwen3.6, so every position saw the tokens
    after it, and greedy output diverged from a shorter or longer prompt's. On
    a Qwen3.5 9B at 4,000 tokens the logits were 3.7 off; they are now within
    rounding. Present since 0.3.
  • Qwen3.5-2B at Q4 repeated its first token when decoding a single request:
    the fused recurrent kernel applied the convolution a second time after it
    had been folded into the projection.
  • Text containing vocabulary entries shaped like markers, such as <?> or
    [:] in Qwen's vocabulary, tokenized into pieces the model never saw,
    because the loader registered every such entry as a control marker. A
    bundle's markers are now the added tokens its tokenizer names, and only
    the ones flagged special are cut out of the text, so whitespace runs in
    Gemma 3's vocabulary stay ordinary text.
  • basert self update --dry-run said nothing about superfluid. It now names
    the superfluid the update would bring, or why the installed one cannot
    serve the engine, and still changes nothing.

superfluid 0.1.6 serves this engine as it did 0.4.3; the engine ABI is
unchanged.