Skip to content

Basalt 0.3.2

Choose a tag to compare

@jesdga95 jesdga95 released this 08 Nov 19:56
· 376 commits to main since this release

2026-11-08. Taken from main. Build from source: building, or take a release archive (platforms). No flag changes; the .basalt format stays version 2.

Highlights

  • DeepSeek decodes faster. Cold, seed 42, RAM tier 316, 400 W + 180 W, basalt-bench ABBA, 0.3.1's main against this release: struct 154 to 178 tok/s, prose 88 to 96, conversation 81 to 94 (conversation's text changes with the promotion margin below, so its windows do not compare one to one).
  • A clean Release build takes 290 s instead of 933 s at -j32 (309 s at -j16): Flash-Next's dec_moe.cu is split so the peer kernel's four variants compile in parallel, and tools/check_cu_includes.sh keeps a .cu from including another. The GPU code is the same except those four variants, which ptxas schedules slightly differently: greedy output identical on both models, decode unchanged.
  • Release archives for Linux x64 and Windows x64 come with each release. They hold no CUDA libraries: the CUDA 13 toolkit's runtime and an R580 or newer driver are required. platforms
  • The website is basalt.jgulfo.com.
  • The binary finds its data, serve/ and .venv next to its own bin/ folder, not at the source path baked in at build time, so an install can be moved.
  • A RAM tier that leaves experts in the file keeps the OS file cache its floor again, so a machine the model does not fit no longer swaps itself unresponsive while loading (#285, @tddgit). memory floors
  • The second card's share of a layer reads its activations and plan from mapped memory, so it no longer waits behind a slot swap's copies on the card's copy engines. First expert kernel after the ids, with a slot copy in flight: 81.4 to 41.8 us.

DeepSeek decode

All bitwise the same output (greedy identical on frozen tiers) except the promotion margin.

  • The second card's exchanges go out in pieces in the bus's idle time.
  • Down rows of 72 calls take eight lanes, four rows a warp.
  • Gate/up pairs of 5120 values take two rows a warp, IQ1_S among them, and three entries a pass at three blocks an SM.
  • The exact CPU expert rows (IQ1_S, IQ2_XXS, IQ2_XS, IQ1_M) decode a block once for up to four tokens: 3 tokens, 1 thread, per row 0.54 to 0.29 us for IQ1_S, 0.51 to 0.21 for IQ2_XXS.
  • Short windows' sparse MQA takes 8 heads a block.
  • The drafter's kernels: Q6_K short rows stage their columns once per block, wo_a takes Q8_0 weights, and the experts take four rows a warp.
  • The expert pool: a down part waiting for its expert's intermediate prefetches its own rows into L2 (down phase 64.2 to 60.4 us in a trace), and a layer's last parts are a quarter of the others.
  • Promotion to the second card needs its victim to be 1,280 lookups (about 32 windows) older than the main card's, so more promotions take a main card slot. This one moves which experts the CPU computes, so the rounding moves. BF16 gate, 100 chunks: KLD 0.27971 to 0.27944 (paired -0.00027, 95% CI -0.00199 to +0.00144), top-1 83.89% to 83.77%: within noise. precision

Memory, start and serving

  • The file cache keeps its floor. 0.3.1 sized every RAM tier to the free RAM less --ram-reserve-mib, so where the model does not fit, the page-locked tier left the file reads and the rest of the system about 2 GiB and the load never finished (reported by @tddgit, #285). A tier that leaves experts in the file now also leaves the file cache what it needed before 0.3.1 (every expert of the largest layer, a prompt chunk's reads, at least 4 GiB), for DeepSeek and Flash-Next; a tier that holds every expert is sized as before, and a given --resident-budget-gib is not cut. The start line counts the floor.
  • The RAM top-up reads the container's limit. After the load it sized itself from /proc/meminfo alone and pinned experts past a cgroup limit; it reads the same container-aware figure as the plan and the copy.
  • The engine refuses to start without its expert profiles. A binary run away from its install folder found no data/profiles and fell back silently (DeepSeek's drafter took 7.47 GiB instead of 2.41 and 240 experts went to the file). The start now stops before loading weights, naming the missing profile and that data/ must sit next to bin/; --expert-profile still replaces the main profile.
  • A failed free-memory query says why. The call, the error and the card are in the message instead of an unrelated copy failure later (@easonLiangWorldedtech, #289).
  • DeepSeek's start checks every card after pinning the RAM tier. With a second card and a RAM tier the driver caps, every CUDA memory call can fail once the tier is page-locked. The start now asks each card for its free memory and a 1 MiB allocation, and fails with the call, the error, the card and a smaller --resident-budget-gib to try (@easonLiangWorldedtech, #289).
  • The Experts card shows an all-VRAM box as all VRAM. With every expert resident the card stayed at "-"; /metrics carries expert_total and the card draws a full VRAM bar, "All in VRAM" (@beastrobin, #287).

Release archives and install

  • basalt-vX.Y.Z-linux-x64.tar.gz and basalt-vX.Y.Z-windows-x64.zip are attached to each GitHub Release: bin/, serve/, data/, docs/ and what the server imports. tools/package_release.py packs a built tree.
  • bin/basalt serve uses the .venv at the archive's top directory when it is there. The build-time source path (BASALT_SOURCE_DIR) is gone: the install root is the directory above the executable's own.

Checked

At 400 W + 180 W, on the commits that changed the code:

  • Kernel and pool changes: greedy output identical to the build before (frozen tiers, struct and prose).
  • Promotion margin: the BF16 gate above.
  • The benchmark checker takes DeepSeek's fp8+fp4 cache, the format basalt-bench records for --kv fp8, and the community table has a DeepSeek-V4.1-Flash IQ1_M row on an RTX 5090 + RTX 5060 Ti. community

Contributors of merged pull requests since 0.3.1: none.