Repository navigation
2026-11-16. Taken from main. Build from source: building, or take a release archive (platforms). The 27B gets a second card and a context past 262,144; Flash-Next's RAM plan leaves room for what the engine takes after it again. The .basalt format stays version 2 and the config stays version 2.
Highlights
- Qwen3.8-27B on two cards. A second Blackwell card reads the last layers of long prompts while the main card reads the next chunk (the
prompt line): with an RTX 5060 Ti beside an RTX 5090, prompt reads +13% at 8k, +19% at 16k, +22% at 32k and +23% at 64k tokens. A 27B that does not fit its main card splits its layers across both (thelayer stage), with output bitwise the main card's alone. two cards - The 27B past 262,144 tokens with YaRN (
--rope-scaling yarn --rope-scale F), and--kv-resident N, which keeps each attention layer's first N positions on the card and streams the rest from page-locked RAM, so a context or--parallelwhose cache does not fit the card still starts. memory - Flash-Next on 64 GB of RAM. Since 0.3.1 the RAM plan left exactly
--ram-reserve-mibfree, and the memory the engine takes after the plan (graph capture, page-locked buffers, the first request) came out of it: a start on a 64 GB box could end at the RAM guard or be stopped by the load watchdog. The plan now leaves 2 GiB beside the reserve, as 0.2.1 did (#339). - DeepSeek expert tiers on up to four second cards, each holding the experts of its own block of layers (#367, @stevenm76).
Qwen3.8-27B
- Prompt line. When the model, the context and the slots fit the main card, a second card reads the last layers of each long prompt chunk, its hand-off through page-locked RAM. Decoding stays on the main card. Off with
--kv-residentbelow the context, and for contexts of 16,384 or less. The start line names it:prompt line: layers 44-63 …. - Layer split. When they do not fit, the start plans a cut from the pack's bytes per layer and both cards' free memory: the main card keeps layers 0 to N-1 with the head, the drafter and the image encoder, the second card the rest with its own KV cache and recurrent state per slot. Each card runs its layers of a window as its own CUDA graph. About 0.37 ms per window per layer on an RTX 5060 Ti. Two cards only; later cards stay idle.
- YaRN reads the same flags as Flash-Next (
--rope-scaling,--rope-scale,--yarn-*), now shared code. It changes short prompts' output too, so leave it off below 262,144. precision --kv-resident N(default 0): decode reads every position past N over the bus, so it slows with each position in RAM; prompt reads stay close; output is bitwise the same.- Idle sleep works for the 27B: a dense model has no expert cache to give back, so SLEEP now succeeds and the driver suspend runs (one card 1.7 s back, the layer split 3.6 s). Before, the server logged that the model stays awake.
Serve
--parallelhands the prompt to the engine as text, as a single slot already did: the engine tokenizes it once and returns each slot's match, and one C++ matcher serves both modes. An over-context prompt is refused with a 400 before the stream starts, from the engine's count.- A reply sent back in other pieces keeps the slot. The model can write a text in pieces the tokenizer would cut differently; a client that sends the reply back unchanged got other ids for it, the prompt stopped matching inside the reply, and the turn was read again from a checkpoint (with
--parallel, in another slot, leaving a dead copy in the old one: 6–8 GiB each at 550K tokens). Where the held ids spell the same bytes as the prompt, the engine now keeps them (#378, @hiraghi). /slotsrows are the engine's slots: row N is engine slot N, a running request shows on the slot it runs in, and an idle row shows what its slot holds (#379, @hiraghi).POST /tokenizeand/detokenize, as llama.cpp's server has them (#363, @mitalbert).--reasoning-effort(or"reasoning-effort"in a model entry) sets the effort for requests that name none (#364, @mitalbert).- Responses API usage carries the engine's prompt count (#381, @JimJafar).
/v1/modelsentries carrycreatedandowned_by.
Fixes and platforms
- A
--resident-budget-gibbetween the experts and the experts plus the adaptive swaps' free places keeps those places, as in 0.3.3; since 0.4.0 decode on a 16 GB card halved there (#366, #369, @giveen). - In a container, the RAM readings follow the cgroup's limit, and the cgroup's shared memory (page-locked buffers among it) no longer counts as reclaimable cache: a 46 GiB container that was OOM-killed after the load now loads (#339 in containers).
- A
-march=nativebuild on an AVX-512 CPU no longer emits AVX10.2 encodings in the AVX-VNNI-INT8 NVFP4 rows (#323). - The shipped executables land in
bin/with every generator (#373); test targets build only withBUILD_TESTING(#370, @giveen); the DeepSeek TIER line says where file reads go through the file cache (#372); a Windows macro no longer breaks the dense build. - The 27B's per-card weights are one allocation, so the layer split's plan is what the load takes.
- Community rows from @edicristofaro (#380), @Suppressor72 (#362) and @hronoas (#371).
Upgrading
- New flags only: the 27B takes
--rope-scaling,--rope-scale,--yarn-*and--kv-resident. No flag or config key is removed or renamed. - Flash-Next's RAM copy is about 2 GiB smaller on a box where RAM bounds it (the plan leaves 2 GiB beside
--ram-reserve-mib).--ram-reserve-mibstill sets the guard. --parallelengine protocol: the server sends prompts as text withPROMPT/PMATCH/PFREE. A server and engine must come from the same release, as always.- The
.basaltformat stays version 2 and the config version stays 2.
Checked
At 400 W + 180 W, on the release commit against the main before the last seven changes (cgroup readings, the RAM plan, --parallel prompts in the engine, /slots, -march=native):
- Greedy output: identical for Flash-Next alone and with
--parallel 2(requests one at a time), the 27B on one card, and DeepSeek on two cards. Concurrent--parallelrequests and one long DeepSeek prompt differ between two runs of the same build, on both builds alike. - Speed (
basalt-bench, 16k prompt, conv/struct/prose, interleaved runs): Flash-Next with its peer card, the 27B and DeepSeek on two cards within run-to-run spread; prompt reads, conv and struct within 0.1–0.4%. - Flash-Next limited to about 56 GiB of free RAM on two cards: the RAM plan 2.2 GB smaller, the lowest free RAM 3.7–4.0 GiB against 2.0–2.6, every start loads and serves, and warm decode unchanged within its spread (198 against 196 tok/s).
--parallel 2with two conversations: each follow-up reuses its history,/slotsshows each slot's held tokens.- The 27B layer split: output bitwise one card's at every window width, with
--parallel, checkpoints and parked conversations. - CPU tests, serve tests, lint and the WERROR build: pass.