Skip to content

Basalt 0.5.5

Latest

Choose a tag to compare

@jesdga95 jesdga95 released this 21 Nov 13:34
· 93 commits to main since this release

2026-11-21. Taken from main. Build from source: building, or take a release archive (platforms). A fifth model joins: Thinking Machines Lab's Inkling 2 Mini, a looped mixture of experts that runs on one card. The .basalt format stays version 2 and the config stays version 2.

Highlights

  • Inkling 2 Mini (Apache-2.0): 24 layers visited 48 times a token (a core of 12 layers run three times), 128 routed experts per layer (top 8), attention residuals, sliding-window and full attention with a learned relative bias. One RTX 5090, with the least routed experts in RAM only when a long context needs their room. Chat, tools, thinking effort, images, up to 1,048,576 tokens of context. guide
  • Speed on an RTX 5090 alone at 400 W, basalt-bench -r 2, drafting on: conversation 338 tok/s, struct 613, prose 287; a 16K prompt reads at 12,240 tok/s. llama.cpp's IQ4_NL GGUF on the same card decodes the same fixtures at 92 to 95 tok/s.
  • Quality against the official release with its experts dequantized to BF16 (inkling2-bf16-base.kld, 100 chunks): mean KLD 0.00305, top-1 97.76%. That base is not a true BF16 model, so the number is not comparable with the other models'. precision

Inkling 2 Mini

  • Pack. convert.py --checkpoint DIR builds it from the official release: layers 0 to 19's NVFP4 experts bit for bit, layers 20 to 23's MXFP8 experts as Q8_0, the other projections Q8_0, the head BF16 (Q8_0 on the card), with the vision encoder and the DSpark drafter. 19.7 GB. guide
  • Decode runs each window of up to 16 rows as CUDA graphs per row count, the experts planned on the card, the small kernels of each layer visit chained with programmatic dependent launch.
  • Prompts of more than 16 tokens read in chunks of up to 4,096 rows on the tensor cores, the NVFP4 experts on the FP4 block-scaled tensor cores with each activation row as two FP4 terms (about 8 bits). precision
  • Drafting. The release's DSpark drafter (three dense layers reading the target's visits 5, 23 and 41, a recurrent Markov head over the target's head, screened by 2-bit copies of its rows) drafts up to seven tokens a window, on by default.
  • Images through --vision dynamic: the engine's own encoder reads a picture at its size, one token per 40 x 40 cell.
  • Attention caches fp16 by default, --kv fp8 halves them.
  • Experts in RAM when the context needs the room. The least routed experts (by the shipped data/profiles/inkling2-24x128.prof) go to page-locked RAM, just enough, and decode reads them there; the full visits' caches grow elastically and the RAM experts come back to the card as copies while the caches leave room. At --max-context 1048576 --kv fp8 on an RTX 5090: 264 tok/s on a short chat, 173 at 131K, 108 at 400K (plain decoding); needles at 131K and 400K found.

Shared engine

  • The shared vision encoder runs class-token towers (a projector with v.class_embd), its patch rows cut on the card from the picture's bytes, with a CPU reference beside it.
  • gqa_window_attn takes a block's shared window (block), the window a block drafter's rows attend; the per-row window stays the default.
  • The native expert kernels take a softcapped SwiGLU (NativeExpertLayout), and NVFP4 expert down products at 1,152 columns run eight lanes a row (native_down_sub8_kernel). Other models' shapes keep their paths.
  • MMQ's SwiGLU takes the softcaps and a per-expert scale tail (mmq::SwigluForm).
  • The router's logits and selection launch through launch_pdl; outside a PDL scope that is the plain launch, as before.
  • prefill::fp4 runs a layer's experts in one launch (Product::base, stride, blk_off from plan_blocks), with the softcapped SwiGLU, the down product scattered to its rows (dst_ids), and the activations as two FP4 terms (two). Flash-Next's launches keep their arithmetic.
  • bf16_rows_dot runs few outputs over a long row (64 over 4,608) as tiles of eight outputs a warp, the rows and weights staged through shared memory; the same sums in the same order.
  • mmq::read_past_bytes: the bytes MMQ reads past a matrix whose rows are not a multiple of 256 values, for buffers it reads in place.
  • Tool calls written as JSON objects (ToolForm::json_args) are constrained by the request's schema as the other tool forms are.

Fixes and platforms

  • Windows gathers a prefetch request's windows into one call (#384, @kaichri).
  • Hy4-flash: faster single GPU NVFP4 decode on hybrid CPUs (#466, @giveen), the prompt line splits the Engram's host step (#468, @giveen), and the docs say what --kv fp8 costs it (#472, @giveen).
  • A </think> mentioned inside the reasoning no longer ends the thinking (#448, @giveen).
  • "reasoning": {"enabled": false} is respected on the OpenAI route (#408, @azido-azide-azide).
  • Serve: --api-key-file keeps the API key out of the process list (#440), the vision work folder is removed at shutdown (#439), a request cut by a stop ends with the stopping error (#436), and a multi-pass request's prompt time is taken from its first pass (#459), all @woodsonl.
  • A request body that stops arriving gets a 408 instead of holding a handler thread (#425, @florian862-art).
  • A prompt read beside decoding slots reads alone when its loan is too short (#414, @cloudfinapps).
  • Responses release inference on cancellation and failure (#400, @cjxsn).
  • The grok build conversation headers count as session identity (#341, @elevenfund).
  • Qwen4exp: PROBE adapt puts a displaced expert only in a free slot its blob fits (#330, @gdevenyi).
  • convert.py reads FP4-Local checkpoints (#444, @woodsonl), names a Flash-Next pack after its GGUF's quant tag (#449, @sramteare), and plans the hc8 section in a dry run (#435, @woodsonl).
  • basalt-bench: a community row's plan probes ask about the cards the run uses (#474, @erkrali), and every --exp reaches the engine (#438, @woodsonl).

Upgrading

  • -DBASALT_MODELS takes inkling2; the default builds every model.
  • No flag or config key is removed or renamed. The .basalt format stays version 2 and the config version stays 2.

Checked

At 400 W, on the release branch against main (a365240):

  • Qwen3.8-27B greedy output identical to main (conversation prompt, 96 tokens, with and without drafting); Hy4-flash greedy output identical with a pinned, frozen tier (96 tokens, drafting on); Qwen3.8-Flash-Next greedy output identical with its tier pinned, after a short and a 16,671-token prompt (the FP4 prompt path).
  • Every model built, also with warnings as errors; CPU tests and serve tests pass; GPU tests pass (148, 5 skipped for want of their packs).
  • Inkling 2 Mini: its kernel, prompt-path and drafter parity tests; greedy drafted output equal to plain on prose, struct and conversation after short and 16K-token prompts; compute-sanitizer memcheck clean on the prompt path with the drafter; pictures from 57 x 23 to 7,260 x 5,052 pixels, a transparent PNG, two pictures in one message, a picture in a multi-turn chat and a picture with a tool call, through both the OpenAI and the Anthropic endpoints; needles at 10, 50 and 90 % of 32K and 128K-token prompts found.

Contributors of merged pull requests since 0.5.0: @kaichri, @giveen, @woodsonl, @azido-azide-azide, @florian862-art, @cloudfinapps, @cjxsn, @elevenfund, @gdevenyi, @sramteare, @erkrali. The pack's weights are Thinking Machines Lab's (Apache-2.0).