v.0.5.8-alpha

Install command, run in PowerShell. From an admin PowerShell it installs for the whole machine instead.

irm https://raw.githubusercontent.com/omgitsbase/llmash/main/install.ps1 | iex

On Linux, with sudo sh at the end to install for the whole machine:

curl -fsSL https://raw.githubusercontent.com/omgitsbase/llmash/main/install.sh | sh

Already installed: llmash update.

Changed

• Many requests at once. With LLMASH_PARALLEL above 1 the streams of one model share each step on the card, and that step is now built for it: every stream is sampled in one step on the GPU, a verify over several sequences runs FP4 and FP8 GEMMs instead of matvecs, attention reads each sequence's own cells, and an MTP drafter keeps a cache stream per sequence. The previous runtime stopped at load with a drafter and more than four slots. Tokens per second of all streams together, steady decode, on one RTX PRO 6000 Blackwell, every provider on the same NVFP4 checkpoint, thinking on, 8,192-token budget (the README's Batching section has the method and the charts):

streams at once	1	2	4	8	16	32
Gemma 4 E4B, llmash	400	920	1,741	3,632	6,738	11,397
Gemma 4 E4B, vLLM 0.32	335	702	1,338	2,690	5,446	9,207
Gemma 4 E4B, Ollama 0.40.11	211	299	382	449	467	473
Qwen3.6 35B-A3B, llmash	521	756	1,062	1,775	2,764	3,863
Qwen3.6 35B-A3B, vLLM 0.32	377	664	1,161	1,910	2,616	3,511
Qwen3.6 35B-A3B, Ollama 0.40.11	189	181	181	183	197	197

On the 35B vLLM is still ahead at 4 and 8 streams, by 9% and 7%. Ollama serves the 35B one request at a time whatever OLLAMA_NUM_PARALLEL says, so its 35B figures past one stream are tokens over the wave's whole time.

• The K/V cache follows the model. LLMASH_KV=auto, the new default, keeps the cache in FP8 when the checkpoint declares an FP8 cache (NVIDIA's and unsloth's NVFP4 checkpoints do), as vLLM does, and in f16 otherwise. f8_e4m3 can also be asked for by name. A model without cache scales runs as before.
• The recurrent state of Qwen3.5, 3.6, the Next models and the other delta-net and RWKV models is kept in BF16, as vLLM keeps it: half the state memory per slot. Wikitext-2 perplexity of Qwen3.6 35B-A3B: 5.8651 with it, 5.8716 with F32 state.

Runtime

The runtime changed, build 776 (fork commit b5985560d), so llmash update replaces the one you have.

• The batching above, and the fixes it needed on hybrid models: Qwen's FP8 projections under several streams ran on the CPU; a hybrid model's batch split one sequence at a time after slots went idle; a drafter with its own cache kept pad rows its target had dropped, which could stop the server at startup.
• An MTP drafter over several slots no longer builds a mask of every slot's cells on the host each round. At 32 streams deep in context that mask took about 20 ms of host time a round and grew with the context; each sequence's stream now holds its own cells, and the drafter's cache follows the model's cache scales.
• Cheaper copies and gathers: the convolution state's per-token snapshots in one pass, 16-bit rows read 16 bytes at a time (Gemma's and Qwen's token tables in pinned host memory), NVFP4 expert scales in one pass after a batched expert GEMM, a vocabulary head under eight rows or more through the FP4 GEMM.
• Identical replies need GGML_CUDA_GEMM_TUNE=0: the GEMM tuner picks among equally correct kernels by timing, and the picks round differently.

Checked before release:

• The client program and its tests on Windows and Linux.
• 16 different requests at once, each asked to repeat a sentence only it was given, three rounds, on Qwen3.6 35B-A3B and Gemma 4 E4B: every reply holds its own sentence and no other.
• The same greedy reply with and without each new copy and scale kernel (GGML_CUDA_GEMM_TUNE=0), and test-backend-ops for the changed operations.
• The perplexity above for the BF16 state.
• Not checked: the same greedy text as the previous runtime. This one changes the numbers on purpose (BF16 state, the FP8 cache where declared).