Repository navigation
BaseRT engine 0.4.4
BaseRT 0.4.4
New
-
Speculative decoding for Qwen3.8-27B and Qwen3.6-35B-A3B on Apple Silicon
with the DFlash 2 block drafters from Inco AI, converted and paired in the
catalog. Pull the target and its drafter, then serve with the drafter named
or withauto, which takes it once it is installed:basert pull basecompute/Qwen3.8-27B basert pull basecompute/Qwen3.8-27B-DFlash2 basert serve basecompute/Qwen3.8-27B --speculate dflash:basecompute/Qwen3.8-27B-DFlash2
The same for
basecompute/Qwen3.6-35B-A3Band its
Qwen3.6-35B-A3B-DFlash2. On an M5 Pro the 27B at Q4 decodes 2.4 times
faster than plain at one request with the defaults, and 1.9 times at four;
with the tiled verify switched on (BASERT_TILED_VERIFY=2) 2.8 times at one
request, and with speculative-sampling acceptance for sampled requests
(BASERTD_SPEC_EXACT_ACCEPT=1, exact in distribution) 3.1 times. The
35B-A3B on a 32 GB M6 mini decodes 1.5 / 1.3 / 1.3 times faster at one, two
and four requests. Greedy requests reproduce plain decoding's text. The
tiled verify and the exact rule stay opt-in; the switches and what each one
costs are in OPERATIONS.md under "Speculative decoding policy".When running superfluid directly rather than through
basert serve, name
the drafter by its.basefile:--speculate dflash:<path to the drafter's model.base>.
Changed
- Gemma 4 prompts served from a paged KV cache now prefill at the speed
basert benchmeasures on a contiguous one. On an M5 Pro a 13.6K-token
prompt on E2B Q8 goes from 1,046 to 11,409 tokens per second through
superfluid, E4B Q8 from 860 to 4,011, and 26B-A4B Q4 from 650 to 1,961.
Served decode is unchanged. - The speculation policy under several concurrent requests. A request that a
gate takes out of speculation (a prompt that repeats little, a price below
plain decoding, a low acceptance) decodes plainly only while it is the only
request, and drafts along with the others otherwise, since riding their
rounds plainly costs more than drafting. A strategy's requests share one
draft depth per round. A request count the throughput gate has not judged
yet runs on a wider count's verdict only until it has evidence of its own.
A request below the acceptance floor keeps speculating while its own rate
still beats plain decoding at that request count. - Bundles converted from GGUF now declare tensor shapes in the same order as
every other source. Bundles converted before this release keep working and
give the same output; for a model whose key and value projections are
narrower than its query projection, re-converting restores the fused
query/key/value attention.
Fixed
- On Apple Silicon, logits computed for every position of a 16,384-token
prefill chunk, which Qwen3.6-35B-A3B takes for long prompts, came out
zero past the 4,324th position of the chunk: the tensor-core matrix
kernels addressed each output tile with a 32-bit offset, which wrapped
once the chunk's logits passed 2 GiB. Perplexity and divergence scores
over such chunks were wrong (one quantization run measured 18 times the
expected divergence); the tiles are now addressed in 64-bit. - A served prompt of 512 to 4,096 tokens whose length was 1 to 32 past a
multiple of 64 ran its first chunk without a causal mask on the full-
attention layers of Qwen3.5 and Qwen3.6, so every position saw the tokens
after it, and greedy output diverged from a shorter or longer prompt's. On
a Qwen3.5 9B at 4,000 tokens the logits were 3.7 off; they are now within
rounding. Present since 0.3. - Qwen3.5-2B at Q4 repeated its first token when decoding a single request:
the fused recurrent kernel applied the convolution a second time after it
had been folded into the projection. - Text containing vocabulary entries shaped like markers, such as
<?>or
[:]in Qwen's vocabulary, tokenized into pieces the model never saw,
because the loader registered every such entry as a control marker. A
bundle's markers are now the added tokens its tokenizer names, and only
the ones flagged special are cut out of the text, so whitespace runs in
Gemma 3's vocabulary stay ordinary text. basert self update --dry-runsaid nothing about superfluid. It now names
the superfluid the update would bring, or why the installed one cannot
serve the engine, and still changes nothing.
superfluid 0.1.6 serves this engine as it did 0.4.3; the engine ABI is
unchanged.