v0.8.2

Highlights

• Ternary Bonsai 2 27B support, with a draft head trained for it on its own output:
  on an M3 Ultra 114 tok/s on short prompts (136 on code), 2.6x llama.cpp and 2.7x
  Prism ML's own MLX runtime; on a DGX Spark 62 tok/s (80 on code), 2.1x llama.cpp.
  Replies are the model's own: every drafted token is checked by the model.
• Ternary Bonsai 2 27B on DGX Spark in 5.9 GB: the ternary weights are kept in a
  1.6-bit form in memory (8.6 GB on disk), prompts are read 1.6x as fast as llama.cpp's
  CUDA build (1603 vs 1001 tok/s on a 4k document), and decoding without the draft
  head is 12% faster (33 vs 29.5 tok/s).
• Linux + CUDA on NVIDIA DGX Spark (GB10): the main model families run on the GB10's
  unified memory, including the paged MoE models, with fused CUDA kernels for the
  Qwen3.5-family and ternary layers.
• Faster prompt processing on DGX Spark for every model: long prompts are read up to
  about 3x as fast on CUDA -- Ternary Bonsai 2 27B 551 -> 1603 tok/s on a 4k document,
  Qwen3.8-27B 669 -> 1175, Qwen3.6-35B-A3B 2617 -> 3362 -- with the ternary model's prompt
  processing on the int8 tensor cores.
• Faster draft-head decoding at long contexts: the draft depth now adapts to the
  context length, so speculative decoding stays ahead of plain decoding on long
  conversations (GLM-5.3 at a 32k context: 40 -> 53 tok/s).

Improvements

• Paged MoE models build their prompt caches faster and use memory more evenly on
  unified-memory machines.
• Faster prefill on CUDA for MoE and long-context attention.

Fixes

• Server: a client disconnecting mid-stream could leave its request holding the GPU,
  delaying the next request; streams are now closed as soon as the client leaves.
• Codex: /v1/responses accepts parallel_tool_calls: false, which some Codex
  versions send; such a turn returns the model's first tool call instead of an error.
• Stability fixes for long prompts on paged models.
• CUDA: a long conversation (tens of thousands of tokens) resumed from a session cache could stop
  responding on Qwen3.5-family models, Ternary Bonsai 2 included; it now completes.