Skip to content

Basalt 0.4.0

Choose a tag to compare

@jesdga95 jesdga95 released this 14 Nov 23:51
· 60 commits to main since this release

2026-11-14. Taken from main. Build from source: building, or take a release archive (platforms). The engine's first dense model joins the mixture-of-experts ones. The .basalt format stays version 2 (two new tensor types, see Upgrading) and the config stays version 2.

Highlights

  • Qwen3.8-27B, Basalt's first dense model, from QUASAR's quantization-aware NVFP4 checkpoint with an FP8 output head and the DFlash 2 block drafter. One card, chat, tools, thinking, images and video, up to 262,144 tokens of context. guide
  • Speed on one RTX 5090 at 400 W, --kv fp16, cold (basalt-bench): prompt reads 12,735 tok/s at 4k tokens and 8,751 tok/s at 32k; decode 90 tok/s plain and 262 tok/s drafted (conversation), 776 on struct.
  • Exact speculative sampling. A sampled 27B request without a seed draws the block drafter's drafts and keeps each with probability min(1, p / q), drawing from the residual where it does not: the tokens follow the model's distribution exactly, and more drafts are kept than the drafter's best guesses would give. A request with a seed keeps the plain rule, so its reply is the same at any window. precision
  • Drafting in windows of up to 16 rows, with prompt lookup (--suffix-draft) beside the drafter, and --parallel slots that share one pass through the weights each round.
  • Checkpoints of every model's conversation state go through one shared, page-locked transfer.
  • A thinking budget stops the engine on its last token, so the answer continues the engine's own state.

Qwen3.8-27B

  • Pack. convert.py --checkpoint quasar --drafter dflash2 builds it from QUASAR's checkpoint (QUASAR-QAT/Qwen3.8-27B-QUASAR-NVFP4, Apache-2.0) and z-lab's drafter, 19.7 GiB. The GDN, attention and MLP projections keep QUASAR's NVFP4 codes and scales bit for bit (NVFP4_PLANES, tensor type 45), the GDN a and b projections become BF16 of their values, and the BF16 output head becomes FP8 rows with an FP32 scale per row (F8_E4M3, type 44; --head nvfp4 rounds it to NVFP4 instead). NVIDIA's NVFP4 checkpoint converts with the same command, its FP8 GDN and attention projections kept as F8_E4M3. model file
  • Prompt reads run every NVFP4 projection W4A4 on the FP4 tensor cores at every chunk length (an FP8 projection W8A8 on the FP8 tensor cores), in chunks of 4,096 tokens, on TMA tile pipelines adapted from Infernix. The NVFP4 GDN and attention steps pass BF16 rows between kernels, with the norm and FP4 quantization fused (@giveen, #361). The residual stream and activations are BF16. Int8 KV prompt attention reads 8-bit P V.
  • Decode windows run their products on one kernel that reads each NVFP4 or FP8 matrix once for up to 16 rows, the FP8 head with a scale per row, and attention on the tensor cores. A row's arithmetic does not depend on the window, so greedy output with drafting is bitwise plain greedy, alone or beside other slots.
  • Drafting. The DFlash 2 block drafter's block of 8 rows widens after windows whose drafts were all kept, and the window is offered every draft of the block; the draft policy reads each draft's confidence to pick the window. --spec defaults to 16. Without the block drafter the pack's MTP layer drafts.
  • Memory. 16.0 GiB on the card with the block drafter, the token embedding in page-locked RAM. --kv fp16 (default) or int8. --parallel N gives each slot its own cache and recurrent state. --vision dynamic for pictures and video. guide
  • Web UI. A dense model shows a Weights card, a VRAM/RAM bar like the experts card; the engine reports its card's weight bytes and its vision encoder apart.
  • Quality against the BF16 model, 100 chunks, --kv fp16: PPL 6.8388 against 6.7411 for the base on the same rows, mean KLD 0.0262, top-1 92.82%; --kv int8 6.8260, 0.0259 and 92.89%. A pack from NVIDIA's checkpoint: 6.7843, 0.0239 and 92.52%. precision

Shared engine

  • A dense stack (src/dense) runs hybrid GDN and gated attention layers with a dense FFN from a pack's widths; a model of that family needs only its pack read in. adding a model
  • GDN kernels are shared by Flash-Next and the dense stack, and so are the Qwen pre-tokenizer, the tool-call form and the Qwen serve dialect (serve/models/qwen.py). Flash-Next keeps its 8-row verify window and its output.
  • Slots of a round run as one batch when the model batches its sessions, and start their drafts as one pass (draft_batch).
  • Checkpoints take their running state as device regions the model lists; the engine copies them through page-locked blocks it holds from start, and a save reaches the host while the request goes on. DeepSeek and the 27B use the same path as Flash-Next did.
  • Images and video take the encoder's grid through the session; a model that reads the shared preprocessor's patches (the 27B) gets video from the server.
  • Thinking budget. The first pass asks for the budget's tokens only, so the engine stops on its last one and the answer continues its state instead of reading the overshoot again through the prompt path.
  • Speculative sampling in the shared sampler: a row with a drawn draft keeps it by min(1, p / q), else draws the residual, on every sampler implementation (sampler_drawn_parity). Philox moved to a header of its own with a purpose key per stream. Greedy output does not change for any model.
  • CI builds qwen35 as a single-model build too, beside qwen4exp and deepseek41.

Fixes and platforms

  • DeepSeek: with experts left in the file, the RAM tier now caches each file expert a lookup routes to, read whole off the file cache, instead of the CPU computing it from the mapped pack a page fault at a time (#318, @mitalbert).
  • A --resident-budget-gib that holds every expert no GPU tier holds keeps the prompt path's lend region, so a prompt read beside decoding slots keeps its full chunks (#329, @gdevenyi).
  • Windows: the expert RAM is backed by a host VMM allocation where CUDA host registration fails under WDDM (#297), and positional reads run concurrently on unbuffered handles (#296), both by @LesterLyu.
  • Image and video encodes write their scratch files per attempt, and a scratch file the encoder still holds no longer fails the request (#302, @easonLiangWorldedtech).
  • The window products build with CUDA 13.0 (the toolkit of CI and the release archives), converting FP4 and FP8 through f16 where its direct bf16 conversion is missing.
  • Community rows from @changlv (#335, #340) and @huangwb68 (#336).

Upgrading

  • .basalt tensor types 44 (F8_E4M3) and 45 (NVFP4_PLANES) appear in Qwen3.8-27B packs; no version change. An engine from before refuses such a pack by tensor name. file changelog
  • Build a 27B pack with this release's converter. The engine refuses a 27B pack whose MLP is NVFP4 in ggml blocks and asks for a re-conversion.
  • The published 27B pack is Qwen3.8-27B-QUASAR-NVFP4.basalt (Hugging Face). A Qwen3.8-27B-NVFP4.basalt from NVIDIA's checkpoint, as first published, still runs.
  • -DBASALT_MODELS takes qwen35 as well; the default builds every model.
  • Conversation checkpoints take page-locked RAM from start (--prompt-cache plus one per slot): the start reserves them before it sizes the RAM tier. A conversation parked to disk by an older build whose running state had another size is read again, not mounted.
  • A thinking budget now stops the engine pass at the budget (all models).
  • No flag or config key is removed or renamed. The .basalt format stays version 2 and the config version stays 2.

Checked

At 400 W + 180 W, on the release commit against 0.3.3's main where a model's output could move:

  • Flash-Next and DeepSeek greedy output on frozen tiers: identical to 0.3.3's main (Flash-Next: conv, struct and prose, 300 tokens each, --kv int8; DeepSeek: 128 tokens each with the RAM tier pinned; each on two cards), and DeepSeek's start plan and tier placement unchanged. A sampled Flash-Next request with drafting replies normally; the same seeded Flash-Next request can give a different reply on a second run, as on 0.3.3.
  • Qwen3.8-27B: greedy output with drafting is plain greedy's, and two slots, alone or on different prompts, give each slot's solo text (conversation, prose, 8k and 16k prompts; 32 prompt-set texts in pairs), on the published pack and on a pack from NVIDIA's checkpoint. NVIDIA's pack gives the same tokens and logprobs as before the QUASAR, FP8 head and sampling changes.
  • Parity tests for every new kernel, against the CPU reference and the kernel it replaces: pass (dense_ops, gqa_attn, window_gemm, gdn_seg_split, fp8_gemm, fp4_dense_tma, e4m3_producers, bf16_chunk, silu, sampler_drawn), and Flash-Next's GDN kernels stay bitwise (dec_gdn, gdn_head_split, mid_chain).
  • Quality gate for the 27B, 100 chunks: see Qwen3.8-27B above.

Contributors of merged pull requests since 0.3.3: @mitalbert, @gdevenyi, @LesterLyu, @easonLiangWorldedtech, @changlv, @huangwb68. The 27B's prompt and decode kernels adapt Infernix's (Ian Ranson and the Infernix contributors); its drafter flow and speculative sampling follow NInfer (Neroued). The BF16 prompt paths of the NVFP4 GDN and attention steps come from @giveen's #361. The pack's weights are QUASAR-QAT's (Apache-2.0).