BaseRT engine 0.3.0

BaseRT 0.3.0

basert serve is a new server. It keeps the command line and the
OpenAI-compatible API it has always had, and runs them on
superfluid, our serving daemon:
requests are decoded together through one shared forward pass,
a prompt a conversation repeats is reused instead of read again, and hybrid
models (Qwen 3.5, 3.6 and 3.8) get context windows up to four times larger
from the same memory. This release also carries the fixes prepared for 0.2.7,
which was never published.

New

• Serving. Continuous batching and prefix reuse are always on; the server
  runs one request at a time unless --continuous-batching [N] or
  --max-batch-size N asks for more. A server is ready sooner (Qwen3-8B on an
  M5 Pro: 5.3 s to 1.0 s). A streamed tool call arrives as partial deltas —
  the function's name first, then its arguments as they are produced — and
  tool_choice naming a function, or required, is honoured. /v1/completions
  accepts suffix for fill-in-the-middle. /v1/models lists each model's
  capabilities, including whether its template takes a thinking switch and
  which reasoning-effort levels it defines.
• Larger windows for hybrid models. The paged KV cache charges only the
  attention layers, so on a 48 GB M5 Pro the automatically sized window grows
  from 72,704 to 262,144 tokens on Qwen3.6-35B-A3B and Qwen3.5-35B-A3B, from
  31,744 to 130,048 on Qwen3.6-27B, and from 27,648 to 111,616 on Qwen3.8-27B.
  Dense models keep the window they had.
• GLM 5.2 on Metal serves with continuous batching, and speculates with
  the multi-token-prediction head its bundle carries.
• Speculation. Block drafters with a reduced draft vocabulary (the DFlash
  heads published for Qwen3-30B-A3B and Nemotron-3-Super) convert and serve,
  token-for-token identical to plain decoding under greedy sampling. A sampled
  round verifies all of its rows on the GPU in one pass (an 8-lane round on
  Qwen3.8-27B: 109 ms to 93 ms). A strategy that loses to plain decoding is
  measured against a real plain run and retired, then rechecked later rather
  than written off. basert complete --spec turns speculation on for a bundle
  with its own head.
• Throughput under agent load. Sampled lanes no longer sort the whole
  vocabulary per token, served lanes sample on the GPU, and prompts are read
  prefill-first. Eight agents across six turns on Qwen3.6-35B-A3B (M5 Max):
  wall time 345 s to 136 s, first token at the median 46 s to 6 s.
• Automatic context sizing on Metal plans within the memory the engine can
  keep pinned, and counts the recurrent state a hybrid model allocates per
  lane, so a derived window no longer pushes the weights out during a long
  prompt.

Changed

• Flags the old server took but this one has no use for are accepted and
  noted on stderr: --paged-kv and --prefix-cache (always on),
  --prefix-cache-file and --prefix-cache-save-interval (the cache is not
  kept on disk), --files-dir and --media-dir (files live in the server's
  state directory; images and audio arrive as data: URIs or uploads), and
  the engine settings --metallib, --prefill-chunk, --paged-weights,
  --no-paged-weights-retry, --gpu-wait-timeout-ms, --decode-replay and
  --no-baked-decode. --request-timeout bounds fill-in-the-middle
  completions only. A flag value that is not a number is refused rather than
  read as 0. --model-dir alone loads its first model at startup. --verbose
  prints the load and GPU diagnostics without the per-kernel dispatch list.
• State — the request log and uploaded files — lives under $BASERT_STATE_DIR
  (default: serve/ beside the model cache), one directory per port, or
  wherever --sessions <dir> says; the request log starts fresh at each start.
• /props no longer reports trained_context; /v1/models reports the
  serving window as meta.n_ctx; replies say system_fingerprint: superfluid;
  /metrics series are named superfluid_*. Error messages are worded
  differently. Without --api-key on a non-loopback address, the server
  answers only requests addressed to an IP or to this machine's own name.
• Requests that omit sampling parameters keep the old server's defaults
  (temperature 0, repeat penalty 1.05, and top-p 0.9 / top-k 40 unless the
  model publishes its own truncation); an explicit value always wins.
  presence_penalty and frequency_penalty outside [-2, 2] are refused, as
  before.
• A model cache without hub.json is served under its repository name, and
  a second variant of the same model as name:variant.
• Text that looks like a chat marker — <|im_end|>, <|im_start|>,
  <think> — inside a message, a tool result or a document is content, never
  framing: a quoted marker can no longer end a turn, forge a system turn, or
  put the rest of a session into reasoning.
• A tokenizer-only handle opens the bundle's metadata only. On NVIDIA that is
  tens of gigabytes less resident for a server that keeps several.

Fixed

• Gemma 4 26B-A4B misread every image it was shown, on Mac and NVIDIA: a step
  of its vision encoder was skipped. On NVIDIA its per-head normalization also
  ran on a partial warp. Gemma 4 E2B and E4B were not affected.
• Hybrid-model prefix reuse now survives a replayed or branched conversation.
  A conversation replayed after the snapshot store had cycled, or a second
  turn branched off a shared first turn, read its whole prompt again, so
  identical loads ran two to three times slower than distinct ones.
• Hybrid models serve at one lane (--max-batch 1), as the Local app starts
  large ones; the recurrent state the single request needs is now reserved.
• The unfused attention path for 512-wide heads with Q8 KV on Metal misread
  part of its input on every odd-length prompt.
• GLM-DSA served a stale key row on every single-sequence paged step.
• Llama-family models produced garbage in batch-invariant mode
  (BASERT_BATCH_INVARIANT=1); Gemma 4 and Qwen 3.5 now use the same kernels
  in that mode whether a request runs alone or in a batch.
• A LoRA adapter on Qwen 3.5's shared-expert gate is applied during
  generation on NVIDIA, not only while reading the prompt.
• Two mixture-of-experts paths on NVIDIA no longer misread weights when a
  model mixes scale formats or keeps routing weights in full precision.
• The GPU sampler treated top_k: 0 as greedy and applied top-p and min-p
  without the requested temperature.
• A small GPU buffer is freed when a model is unloaded.