Repository navigation
Basalt 0.3.3
2026-11-12. Taken from main. Build from source: building, or take a release archive (platforms). One flag is gone and one default changes; see Upgrading. The .basalt format stays version 2 and the config stays version 2.
Highlights
- A client can end a running reply's thinking:
POST /v1/answer_now, by the response'sX-Request-Idor by session./propsnames Basalt and lists its capabilities. sampling - Token log probabilities on Chat Completions, Responses and Anthropic Messages, on both models, computed on the GPU:
logprobsandtop_logprobs, also for the thinking's tokens and a tool call's. sampling - Documents in messages: a PDF is read as its pages (a scanned page as its image), and Word, PowerPoint and Excel files as their text, on all three APIs. documents
- Video input on Qwen3.8-Flash-Next, decoded on the GPU's NVDEC. A minute of 4K reads, and a second question about the same video reads in a fraction of the first one's time. running
- Session-affinity headers pick the slot: a conversation stays where its K/V sits (#214, @gtrak).
- Flash-Next on one card with
--parallel 4reads long prompts faster: the next chunk's n-gram rows are read beside the current chunk. Cold, one RTX 5090 at 400 W, 40k / 100k / 250k tokens: 12% / 20% / 22% shorter. - DeepSeek decodes a little faster on two cards, and the drafter offers its whole block by default. DeepSeek packs can be built from NVIDIA's NVFP4 checkpoint.
- A container image for CUDA 13 on Blackwell (#293, @eisenbergrobin). platforms
Serving
- Answer now.
POST /v1/answer_nowwith{"request_id": ...}, or with a session (session_idor the session headers), ends the thinking of a running reply; the reply goes on to its answer in the same response. Chat Completions, Messages and Responses, streaming or not, on both models. Every chat response carriesX-Request-Id. A request still queued or reading its prompt accepts the call and closes at its first token; one that is answering, finished or has thinking off gets a 409. It closes with the thinking budget's own wrap, but stops the engine's window softly so the next pass extends what the engine holds instead of reading the thinking again. From the call to the first answer token, 400 W + 180 W, greedy: Qwen alone 15-28 ms, DeepSeek 54-88 ms (two busy batch slots on Qwen 35-102 ms). /propsnames Basalt. The llama.cpp fields are unchanged and abasaltkey carries theversionand acapabilitieslist:answer_now,logprobs,documents, andvideowhile video is on. A client detects Basalt by the key. Detecting Basalt- Logprobs. OpenAI's
logprobs: trueandtop_logprobs0 to 20 on Chat Completions,include: ["message.output_text.logprobs"]on Responses, and the same fields on Messages as an extension. Beyond OpenAI, the thinking's tokens (logprobs.reasoning_content) and each tool call's tokens (logprobs.tool_calls) carry entries, so the least likely token of a call shows where the model guessed. The values are the row's raw head logits, before temperature, penalties, bias, the sampling cut and the grammar mask, computed on the GPU once a window's picks are final; the reply is unchanged. Against a double-precision CPU reference, at both models' head widths: max abs error 1.9e-6, ids exact, and a row alone is bit-identical to its batch row. Decode costs about 1%, 10 alternated rounds, 400 W + 180 W, top 5: Flash-Next 173.7 to 171.7 tok/s (1.1%, real and small), DeepSeek 121.7 to 120.4 tok/s (1.1% +-0.9%, within noise); batch slots measured flat. Layer-split pipelined batch groups refuse it with a 400. - Documents. Anthropic
documentblocks (base64 PDF or text,text,contentandurlsources, also inside tool results, as Claude Code's Read tool sends a PDF), Chat Completionsfileparts and Responsesinput_file. Before, they were dropped without a word. The PDF's text layer is read page by page (pypdf), with[page K]lines; a page with no text, or little text and a large picture, also hands its main image to the vision encoder."pages": "40-55"reads only those pages. Word, PowerPoint and Excel files are read as text. A URL or path follows the image-source rules. A document that cannot be read is a 400 that names it (password, damage, over 600 pages or 64 MiB, unknown type,file_id); two PDF parses run at a time, and a parsed PDF is kept by hash, so a document resent every turn keeps the cached prefix. 400 W + 180 W, a 13-page paper takes 0.19 s to parse; resent in a new turn, Qwen found 10,766 of 10,773 tokens cached (3.4 s to 0.5 s) and DeepSeek 9,791 of 9,792 (8.2 s to 1.4 s). documents - Video input (Qwen3.8-Flash-Next; DeepSeek refuses a video with a 400). Video parts in every API, read and encoded with the images. FFmpeg 6, 7 or 8 is loaded at run time, and frames decode on NVDEC in the process's own CUDA context, converted on worker threads; a stream NVDEC refuses, or that fails part-way, decodes on the CPU. In
--vision dynamicthe expert cache lends NVDEC its surfaces between replies (a minute of 4K: 1013 MiB lent, shrink 9-21 ms, refill 32 ms); the cache is not shrunk while a slot decodes, so a video that arrives during a reply decodes on the CPU beside the batch windows. A video may be 1 GiB, and the guard caps what is held (sampled frames at their canvas size, 256 Mpx; one decoded frame, 64 Mpx), so a minute of 1080p or 4K reads. Cold server, NVFP4 pack,--vision dynamic, greedy, thinking off, 400 W + 180 W: time to first token 4.34-4.68 s for a minute of 1080p H.264 and 5.23-5.25 s for a minute of 4K HEVC (6.00-6.08 s on the CPU); a minute of 4K decodes in 2.4-2.7 s on NVDEC (CPU, 8 threads: 3.5-3.7 s). Decoded frames are bit-identical to PyAV's, on NVDEC and on the CPU. On a short clip NVDEC is no faster than the parallel CPU path (30 s of 720p: 2.9-3.2 s against 2.67 s). running - The media checkpoint. A prompt with pictures keeps a checkpoint where its last picture ends, so the same image or video asked about again in a new chat resumes there. Qwen, one 30 s clip (8,202 tokens), three questions in three new chats: the second and third took 1.98 s and 1.99 s, now 0.37 s and 0.38 s. DeepSeek, an image asked in new chats: 264 of 274 tokens cached, 0.24 / 0.19 s against 0.99 / 0.90 s.
- Answer room is
autoor a token count.--answer-room auto|N, and"answer-room": "auto"or a number in a config. Auto now limits the thinking down to a cap of 32 tokens (it began at 1,024), somax_tokens700 thinks at most 350 tokens and answers, where it returned an empty reply (#334). A number N reserves N tokens at any cap; 0 reserves nothing. sampling - Session-affinity headers pick the slot (#214, @gtrak).
x-llm-flow-id,x-claude-code-session-id,x-session-idandx-session-affinity, then the body'ssession_id, name a conversation. It keeps its own slot, so its K/V is already there; another conversation is displaced only when no unowned slot is free. Two conversations with a shared system prompt no longer take turns in each other's slot. - llama.cpp clients get each model's template capabilities.
/propsreportschat_template_capsfrom the model's dialect,reasoning_budgetis read on the chat endpoint (0 off, -1 no cap, N a budget at the level it names) andpreserve_reasoningmaps to each template's own switch. - The console shows the conversation cache, session totals and the request's phase, read from protocol facts. Its best figures are bounded and belong to the engine that made them, a request's prompt-read time is the sum of its runs (#299, @easonLiangWorldedtech), and the best prefill bound leaves room for fast rigs (#298).
- Every model file is checked at launch, one plain line per problem (missing, a directory, unreadable), naming the model and where the path was set, then exit status 2.
- Smaller:
/v1/modelsalso writes themodelskey newer Codex builds decode (#286, @beastrobin); the elastic KV's ceiling is in the startup line and/v1/status(#237, @f1zz1ec0ke); the big-target rate survives bursty, slow and re-planned read-out (#271, @lindworm-hermes2026); boolean JSON schemas are accepted at every subschema position (#276); an image file is closed before its reply (#75, @LesterLyu); the image resize follows the reference's geometry (#70, @jsmcconn); the stdin reader thread ends before the runtime it reads into (#304).
Flash-Next prompt reading
- The next chunk's n-gram rows are read beside the current one. With
--parallelon one card, a prompt longer than a chunk is read as one run per chunk, and a run's own prefetch of the next chunk's rows only worked inside that run: every chunk read its rows from the SSD at layer 1 with the GPU and the link idle. The loop now arms the row-cache warm-up for the tokens after the chunk, so the next run's gather finds them in the cache. The same bytes, the output unchanged. Cold, one RTX 5090 at 400 W, Qwen NVFP4,--parallel 4: a prompt of 40k / 100k / 250k tokens reads 12% / 20% / 22% faster, 100k at 8.3k tok/s. - A prompt read the slots wait for lends as on an idle engine (#303).
- Peer access between the cards a run uses (#211, @gtrak), asked of the driver before a card is touched and silent where no pair can. The prompt split runs with a control vector (#253, @Hotcooler). A reader's failed n-gram open no longer waits for reads it never issued (#283, @arytek).
DeepSeek decode
All with the same text and tier counters in the commits' own arms (greedy identical on frozen tiers); the numbers are the commits'.
- The drafter offers its whole block by default.
--specdefaults to 6 rows in place of 3: the draft policy takes the rows that pay, so structured output runs windows of 6 and prose keeps windows of 1 to 3. The policy also reads a drafter that states each draft's chance. - The second card's share is queued with the head and waits on the card for its plan; a half of a bus share as large as the CPU's crosses the second card's own link into scratch slots there (400 W + 180 W, CLI prose, seed 1: verify 19.07 to 18.82 ms a window with windows of 3, -1.1%).
- Copy engines: promotion pieces yield them to later copies and the homes go out on their own stream (5090 alone: 36.0-36.5 to 34.8 ms, -4.6%; two cards -0.45%).
- The window's start: the later Engram layers' reads go out behind layer 0's head, the next window's row 0 is read while the drafter runs, the drafter goes out before the compressor commits, and the token table goes back into the file cache once the expert tier has its RAM (-0.9%, 400 W + 180 W, CLI prose).
- Kernels: the indexer's top-k lists up to 12,288 rows from one block without a sort, and counts per warp past it (12,288 rows 21.4 to 15.1 us; 131,072 rows 34.1 to 18.6 us, RTX 5090 at 400 W); the drafter's MXFP4 experts take q8_1 activations and dp4a; the staged Q6_K kernel reads a warp's rows before their dots; the sampler's wide path sorts on a stream of its own.
- The parked pool worker yields its core once every 50 us, not on every check, so a CPU held at its package power limit keeps its clock for the cores that compute experts; the draft wait,
JobRunner::joinand the adapt go-spin yield after 50 us too (#246, @xyehya). - The landing stack of 2026-11-11, ABBA, 400 W + 180 W, two cards: harness short fixture +0.7% tok/s, deep +1.2%;
basalt-benchconversation +2.9%, struct +1.2%, prose -0.7%; verify ms a window -1.2% to -2.4%. Prose did not gain. - Memory: the compressed caches take the VRAM tier's slots when shrinking is not enough, where a step used to fail and the engine exit (#304, a patch from @KimDaeHwan99); the batched attention is a prompt workspace, counted and released with the others.
NVFP4 packs
- DeepSeek-V4.1 packs from NVIDIA's NVFP4 checkpoint (
--sections-fromwith--checkpoint): the routed experts' NVFP4 rows move into Basalt's blocks, the FP8 projections become Q8_0 of their exact values, and the MXFP4 Engram rows are kept; every tensor is checked against the base pack's. The pack decodes through NVFP4 experts on the prompt path and the decode tiers, MXFP4 Engram rows, and NVFP4 CPU gate/up rows that take the SwiGLU limit. A test pack, not yet scored against BF16. model file, DeepSeek - The BF16 KLD bases are published on Hugging Face. precision
Fixes and CPU kernels
- BF16 rows in the per-layer token embedding table (#185, @azido-azide-azide).
- Longer native multi-token row tasks on AVX2 (#255, @Hotcooler); IQ3_S, IQ3_XXS and IQ2_S CPU rows gather their grid rows where gathers are fast, on Intel cores only, as AVX-VNNI parts without core types measured slower with the gather (#281, @arytek). Every output bit is unchanged.
tools/migrations/config_v1_to_v2.pytakes a single-model config marked"version": 2; before, the server sent such a config to the tool and the tool refused it.- The container image is a two-stage Dockerfile laid out like a release archive; HOME is
/data, packs go on/models. Its path is untested by benchmarks. platforms
Bench and community
basalt-benchtakes the engine and fixtures of the install it sits in; a run with lossy flags shows the engine's warning and is marked LOSSY, and--communitynames the lossy flag when it refuses one (#309). The struct fixture no longer checks its token count against a fixed number.- A community row records each decode test, and a second run of the same setup updates the first. The GitHub Pages site is gone; the runs are at basalt.jgulfo.com/community/. New rows: RTX 5090 (#258, #266, #274, #275, #312), two RTX 5080 (#295, #300), RTX 5060 Ti (#316, #317), RTX 5070 Ti (#292, #257), RTX PRO 6000 (#290), DeepSeek-V4.1-Flash and Qwen3.8-Flash-Next on one RTX 5090 (#310). community
Upgrading
--no-answer-roomis gone, and so is"answer-room": falsein a config. Use--answer-room 0, or"answer-room": 0. A boolean in the key is refused at start, naming the new form.--answer-roomalso takesauto(the default) or a number of tokens.pip install -r requirements.txtin the venv: it addspypdf(7.0.1), which PDF documents need. Without it a PDF request gets a 400 that says so; everything else runs.- Video needs FFmpeg 6, 7 or 8 libraries at run time, and is optional: without them the start prints
video offand the reason, and a video is refused with it. platforms - DeepSeek's
--specdefault is 6 (was 3). Pass--spec 3for the old window. basalt-benchstarts thebasaltbeside it and reads the fixtures of its own install when--engineand--fixturesare absent; a built tree behaves as before.- No other flag or config key is removed or renamed. The
.basaltformat stays version 2 and the config version stays 2.
Checked
At 400 W + 180 W, on the commits that changed the code:
- Features that are not asked for leave the engine's inputs byte-identical to the build before (answer now, logprobs, documents); Qwen's greedy output on frozen tiers is identical. DeepSeek's greedy text differs between fresh starts even on 0.3.2's
main(adaptive tiers), so its proof is the engine's inputs. - Kernel and pool changes: greedy output identical to the build before (frozen tiers, struct and prose).
Contributors of merged pull requests since 0.3.2: @changlv, @eadra, @giveen, @Hotcooler, @gtrak. Patches landed from others: @murez, @glfenix, @eisenbergrobin, @LesterLyu, @jsmcconn, @azido-azide-azide, @xyehya, @arytek, @easonLiangWorldedtech, @f1zz1ec0ke, @lindworm-hermes2026, @beastrobin, @RogerYong, @wasiejen, @polygizer, @tselle, @gpu-node-inferno.