Repository navigation
2026-11-19. Taken from main. Build from source: building, or take a release archive (platforms). A fourth model joins: Tencent's Hy4-flash, a mixture of experts with a million-token context. The .basalt format stays version 2 and the config stays version 2.
Highlights
- Hy4-flash (
tencent/Hy4-flash): 512 routed experts in each of 41 layers (top 16, in a 1,536 latent), MLA attention with an indexer over every position, the Engram n-gram table, identity hyper-connections. One RTX 5090, optionally a second card for experts, and system RAM for the rest. Chat, tools, thinking, images, up to 1,048,576 tokens of context. guide - Speed on an RTX 5090 at 400 W with an RTX 5060 Ti at 180 W, serve
bench.sh(the second pass of a server): conversation 92.2 tok/s, struct 171.5, prose 83.8; prompt reads 3,454 tok/s. - 1M context: a needle at 50 % of a 1,015,207-token prompt is found, read in 486 s (2,090 tok/s) with
--kv fp8. KV cache - Quality against the BF16 model (
hy4flash-bf16-base.kld, 100 chunks): mean KLD 0.201, top-1 86.09%. quality - The expert tier gives each layer its own share of the main card's copies, lends a long prompt only the room it needs, and takes back what a prompt borrowed even when caches grew.
Hy4-flash
- Pack.
convert.py --gguf Hy4-flash-BF16.gguf --nvfp4builds it from Tencent's BF16 checkpoint converted to a BF16 GGUF, with the vision projector and the DFly2 drafter as side files: routed experts NVFP4, the other projections Q8_0, the Engram table as E4M3 codes, the head BF16. 158.7 GB. Published asHy4-flash-NVFP4.basalt(Hugging Face). model file - Attention caches. 16-bit rows are fp16 by default (
--kv fp16);--kv fp8stores vLLM's 656-byte FP8 row and nearly halves the cache at long context. The caches are elastic as Flash-Next's: a session backs its first 32,768 positions at start and grows into the expert tier's room as prompts need it. A prompt the caches cannot grow for is refused and the engine serves on. - Drafting. The pack's DFly2 block drafter (five layers reading the target's caches and index selections) drafts up to five tokens a window; the draft policy picks each window from the drafts' stated confidence. Its projections load as Q8_0, so the expert tier is sized after a 0.87 GiB drafter.
- Images through
--vision dynamic: the encoder is staged once in pinned RAM and a request's pictures borrow the main card's room while they are encoded. - Expert tier. The engine ships
data/profiles/hy4flash-42x512.prof(each layer's experts ranked by decode routing over 40 chats), so a fresh start places the right experts first. The early layers route more widely, so they get more of the main card's copies (about 1.4 times the mean, the quiet layers 0.73). A second card holds experts of its own; the rest run on the CPU from pinned RAM. - Decode runs each layer in CUDA graphs, with the CPU's and the second card's expert rows joined through a word the host writes, and each sublayer's add, norm and stream mix in one launch.
Shared engine
- Per-layer copies (
ExpertTierSpec::layer_weights): a model can give its layers different shares of the main card's copies, the same total spread deterministically. Empty for every other model. - Bounded lend (
ExpertTierSpec::lend_segment): a long prompt borrows only its chunk's buffers and the expert ring from the tier, in segments, instead of the whole arena. 0 for every other model. - Partial reclaim: a reclaim that cannot take back every slot a prompt borrowed opens as many as the card holds above the reserve, and the planned size again later.
Status::Refused: a prompt a model refuses for want of room leaves the session as it was; serve answers it with an error and serves the next request.- The decode MQA chunk stage and its merge with a sink are shared by DeepSeek and Hy4-flash (
basalt/kernels/mqa_chunk.cuh). - The loader rounds BF16 to Q8_0 on the way (
Conv::Bf16ToQ8_0, ggml's reference rounding). - The Lanczos image resize is gone; no model used it any more.
- CI builds
hy4flashas a single-model build too, besideqwen4exp,deepseek41andqwen35.
Fixes and platforms
- Expert-tier peers on Ada (sm_89) cards: a
-DPEER_ADA=ONbuild lets an RTX 40-series card be the second card that holds experts; the main card stays Blackwell (#343, @cobra91). - DeepSeek: a follow-up of 16 tokens or more reads in one batched chunk when the caches leave no prompt workspace (#327, @cfelicio).
- Engine crashes, Windows memory figures and runtime errors reach the log and the console (#377, @giveen).
- On x86 Linux a hybrid processor's core classes come from the hybrid PMUs' CPU lists (#402, @aamsellem).
convert.pybuilds the Flash-Next pack from NVIDIA's mixed-precision (QAD) checkpoint (#406, @giveen).- Community rows from @cobra91 (#390), @TiddlyWiddly (#386) and @HexhamAllstar (#394, #401).
Upgrading
-DBASALT_MODELStakeshy4flash; the default builds every model.- Hy4-flash reads images through
--vision dynamiconly. - No flag or config key is removed or renamed. The
.basaltformat stays version 2 and the config version stays 2.
Checked
At 400 W + 180 W, on the release commit against main (63cb68a):
- Flash-Next and DeepSeek greedy output on pinned tiers: identical to
main(conversation, struct and a 16.7K-token prompt, 128 tokens; Flash-Next on one and two cards with and without drafting, DeepSeek on two cards with and without drafting). After a 33K (Flash-Next) or 30.8K-token (DeepSeek) prompt with the default adaptive tiers, the tokens are identical and the hit rates equal. - WERROR builds of every model and of each model alone; CPU tests pass; GPU tests pass.
- Hy4-flash: parity tests for its kernels and the drafter, tokens identical with frozen tiers for every decode change, serve smoke.
Contributors of merged pull requests since 0.4.1: @cobra91, @cfelicio, @giveen, @aamsellem, @TiddlyWiddly, @HexhamAllstar. The pack's weights are Tencent's (Tencent Hy License 1.0).