Repository navigation
Halobox v1.0.0
Build b1049, master 0c1021e1f. Ubuntu and Windows zips for ROCm (gfx1151) and Vulkan.
Summary
Qwen3.8 Flash-Next on ROCm (gfx1151). With UD-Q4_K_XL on a 128 GB Strix Halo: 2089 t/s pp2048 at depth 0, 1680 t/s at 32k and 1621 t/s at 100k, 26.5 t/s decode. With the MTP draft head, decode reaches 36-53 t/s depending on the prompt.
On a 64 GB box, run Gyro-M (TQ2_T, a format only halo-box loads) on Vulkan: 30 t/s decode, 35-61 t/s with the MTP draft, about 1110 t/s pp2048.
Also measured on this release: Qwen3.8 27B dense (22-38 t/s decode with its MTP head, Vulkan), Qwen3.6 35B-A3B MoE (63 t/s decode, 3100-3400 t/s prefill) and Ternary Bonsai 2 27B (PTQ1_0, 28 t/s decode).
halo-box also loads formats upstream llama.cpp cannot (Gyro TQ2_T, Bonsai PTQ1_0, ROCmFP4/FPx), and supports 1M context and multi-slot serving for Flash-Next.
Performance at a glance
b1049 release zips, Strix Halo (Ryzen AI Max+ 395, Radeon 8060S), llama-bench, flash attention on, 3 repetitions. Flash-Next with -lzm on-direct; each model at its best ubatch (2048 for all of them on this build).
| model | box | backend | pp2048 | pp2048 @32k | pp2048 @100k | tg128 | tg128 @32k | tg128 @100k |
|---|---|---|---|---|---|---|---|---|
| Flash-Next UD-Q4_K_XL | 128 GB | ROCm | 2089 | 1680 | 1621 | 26.5 | 24.5 | 23.5 |
| Flash-Next Gyro-M (TQ2_T) | 64 GB | Vulkan | 1112 | 957 | 30.7 | 29.1 | ||
| Flash-Next Gyro-M (TQ2_T) | 64 GB | ROCm | 1135 | 1007 | 27.3 | 25.5 | ||
| Flash-Next UD-Q4_K_XL | 128 GB | Vulkan | 1522 | 1268 | 1160 | 27.9 | 27.1 | 25.9 |
| Qwen3.6 35B-A3B UD-Q4_K_XL | 128 GB | Vulkan | 3096 | 1757 | 63.1 | 51.7 | ||
| Qwen3.6 35B-A3B UD-Q4_K_XL | 128 GB | ROCm | 3397 | 1529 | 57.3 | 48.8 | ||
| Qwen3.8 27B UD-Q4_K_XL | 64 GB | Vulkan | 520 | 387 | 12.7 | 11.2 | ||
| Qwen3.8 27B UD-Q4_K_XL | 64 GB | ROCm | 458 | 314 | 12.2 | 10.3 | ||
| Ternary Bonsai 2 27B (PTQ1_0) | 128 GB | Vulkan | 321 | 272 | 28.0 | 21.8 |
With a q8_0 KV cache (-ctk q8_0 -ctv q8_0) decode at 32k rises to 55.6 t/s on 35B-A3B Vulkan (+7.6%), 50.6 on 35B-A3B ROCm (+3.7%) and 11.8 on 27B Vulkan (+5.4%), with prefill unchanged.
Decode in llama-server, 256 tokens per prompt, with and without the MTP draft head (Q8_0 head, 3 draft tokens):
| prompt | UD-Q4_K_XL, ROCm | + MTP | Gyro-M, Vulkan | + MTP |
|---|---|---|---|---|
| code | 26.3 | 47.2 | 30.0 | 50.4 |
| prose | 26.3 | 35.8 | 30.2 | 34.9 |
| reasoning | 26.4 | 49.6 | 30.2 | 54.1 |
| edit (837-token prompt) | 26.6 | 53.3 | 30.6 | 60.9 |
| 28.6k-token prompt | 24.3 | 42.3 | 28.0 | 46.1 |
Qwen3.8 27B dense with its MTP head (MTP/mtp-Qwen3.8-27B-Q4_0.gguf), llama-server, q8_0 KV:
| prompt | Vulkan | + MTP | ROCm | + MTP |
|---|---|---|---|---|
| code | 12.5 | 31.9 | 12.1 | 27.9 |
| prose | 12.5 | 22.2 | 12.1 | 19.4 |
| reasoning | 12.5 | 33.5 | 12.1 | 29.8 |
| edit (837-token prompt) | 12.4 | 38.1 | 12.1 | 33.5 |
| 28.6k-token prompt | 11.6 | 24.8 | 10.5 | 20.7 |
Recipes
The b1049 Ubuntu zips store the binaries without the execute bit. After unzipping, run chmod +x llama-* in the folder.
128 GB Strix Halo: UD-Q4_K_XL on ROCm (highest prefill)
- Download the ROCm (Ubuntu) zip and unzip it. ROCm 10.1 is bundled.
- From
unsloth/Qwen3.8-Flash-Next-GGUF, download the 4UD-Q4_K_XLshards (about 111 GB) andMTP/mtp-Qwen3.8-Flash-Next-Q8_0.gguf. - Run:
llama-server -m Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf \ -lzm on-direct -c 40960 -b 4096 -ub 4096 -np 1 -md mtp-Qwen3.8-Flash-Next-Q8_0.gguf
Expected: 36-53 t/s decode with the draft and 1416 t/s prefill on a 28.6k-token prompt. Peak GPU memory is about 89 GB.
64 GB Strix Halo: Gyro-M on Vulkan
UD-Q4_K_XL does not fit in 64 GB. Gyro-M is a trellis-coded 2-bit quant of the same model. It is one 92 GB file, of which about 38 GB stays on the GPU (45 GB with the draft head).
- Download the Vulkan (Ubuntu) zip and unzip it.
- Download
Qwen3.8-Flash-Next-Gyro-M-TQ2_0.gguffromagentionai/Qwen3.8-Flash-Next-Gyro-GGUF, andMTP/mtp-Qwen3.8-Flash-Next-Q8_0.gguffromunsloth/Qwen3.8-Flash-Next-GGUF. - Run:
llama-server -m Qwen3.8-Flash-Next-Gyro-M-TQ2_0.gguf \ -lzm on-direct -c 40960 -ub 2048 -md mtp-Qwen3.8-Flash-Next-Q8_0.gguf
Expected: 35-61 t/s decode with the draft, and about 10 GB of RAM left free. The ROCm zip also runs Gyro-M with the same output quality (perplexity 2.2196 vs 2.2178 on Vulkan). It has 2-5% more prefill but about 12% less decode.
Qwen3.8 27B dense on Vulkan
- Download the Vulkan (Ubuntu) zip and unzip it.
- From
unsloth/Qwen3.8-27B-GGUF, downloadQwen3.8-27B-UD-Q4_K_XL.gguf(17.6 GB) andMTP/mtp-Qwen3.8-27B-Q4_0.gguf. - Run:
llama-server -m Qwen3.8-27B-UD-Q4_K_XL.gguf \ -c 40960 -ub 2048 -np 1 -ctk q8_0 -ctv q8_0 -md mtp-Qwen3.8-27B-Q4_0.gguf
Expected: 22-38 t/s decode with the draft (12.5 without), about 520 t/s pp2048. Fits a 64 GB box with about 33 GB of RAM left free. Vulkan is ahead of ROCm on this model in both prefill (+13%) and decode.
Qwen3.6 35B-A3B MoE
- Download the Vulkan (Ubuntu) zip and unzip it.
- From
unsloth/Qwen3.6-35B-A3B-GGUF, downloadQwen3.6-35B-A3B-UD-Q4_K_XL.gguf(22.4 GB). - Run:
llama-server -m Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf -c 40960 -ub 2048 -np 1 -ctk q8_0 -ctv q8_0
Expected: about 63 t/s decode and 3100 t/s pp2048 (llama-bench). The ROCm zip has about 10% more prefill on short prompts; Vulkan has more decode and more prefill at 32k.
What the flags do
The other defaults are already right. GPU offload and flash attention are automatic, the MTP draft type is read from the draft file, and 3 draft tokens is the default.
| flag | why |
|---|---|
-lzm on-direct |
Keeps the per-layer-embedding table (27 GB on UD-Q4_K_XL) on disk and reads only the rows each batch needs. The default copies the whole table into RAM on Strix Halo. |
-c 40960 |
Without -c, the server sizes the KV cache for the model's full trained context. |
-b 4096 -ub 4096 |
The defaults are 2048 / 512. With the draft head, 4096 gives about 7% more prefill on long prompts than 2048. |
-np 1 |
One slot for a single user, about 5% more prefill than the default 4 slots. Leave it off when serving several clients. |
-ctk q8_0 -ctv q8_0 |
q8_0 KV cache: 4-8% more decode at depth on the dense and MoE models, at no prefill cost. Not used for Flash-Next. |
-md |
The MTP draft head speeds up decode but costs prefill on long prompts (1766 vs 1416 t/s on 28.6k tokens). Drop it for long-document or RAG work. |
Models and formats
| TQ2_T (Gyro) | Trellis-coded 2.125 bpw weights, with kernels on CPU, Vulkan and HIP. Upstream llama.cpp cannot load Gyro files. |
| TQK6 / TQK7 / TQP5 / TQP6 / TQP7 | Bit-packed trellis siblings of TQ2_T, on CPU, Vulkan and HIP. |
| PTQ1_0 (Bonsai / PrismML ternary) | quantize, CPU and Vulkan. There are no HIP kernels yet. |
| ROCmFP4 / ROCmFPx | Q4_0_ROCMFP4 variants and Q2/Q3/Q6/Q8_0_ROCMFPX weight formats, on CPU, Vulkan and HIP. |
| Flash-Next long context | QSA sparse attention kernels on both Vulkan and HIP, 1M context (#215), multi-slot compute memory (#216), and the per-layer-embedding table read from disk (-lzm on-direct). |
New in this release:
- Flash-Next 1M context (#215) and multi-slot compute memory (#216).
- The QSA prefix is kept across images in multimodal prompts (#235).
- Convert writes the QSA compress ratios when newer transformers name the layers
indexed_attention(#251). GGUFs converted with transformers 5.22 or later before this fix have all ratios at 0 and run dense attention at long context. Re-convert them. - Routed IQ1_M / IQ2_S / IQ2_XXS / Q2_0 decode on gfx1151, for faster prefill on the low-bit Flash-Next quants (#260).
Speed work
Vulkan (RADV)
- MoE gate, up and SwiGLU in one launch, with streamed down projection (#223), including q5_1 (#224)
- Dense q8_0 x f16 GEMM, on by default (#245, #246)
- q8_0 / q6_K mat-vec without chunking (#241), split_k at small n (#242), XL tile margin (#244)
- QSA score / expand fusions (#228, #229)
ROCm / HIP on gfx1151
- Flash-Next prefill (#213)
- Routed low-bit decode tiers (#260)
- Packed MMVQ for small Q8_0 batches
- Also carried over from earlier builds: the MMB routed-expert path, hyper-connection kernels, the fused GDN chain, RDNA 3.5 flash-attention tiles, and a zero-SGPR-spill check in CI.
Loading: With an explicit -lm mmap on an integrated GPU, weight copies are staged through a host buffer (#214). Flash-Next UD-Q4_K_XL on ROCm used to not finish loading after 400 s; it now loads in 56 s. The default load mode is unchanged (48 s).
Speculative decoding
halo-box adds these on top of upstream's MTP and DFlash drafting:
| flag | what it does |
|---|---|
--spec-draft-adaptive |
Sizes each draft from measured acceptance instead of always drafting --spec-draft-n-max. On Flash-Next with the MTP head, the fixed default of 3 was as fast or faster. |
--spec-draft-mtp-pcum P |
Stops an MTP chain when the product of top-1 probabilities drops below P. |
--spec-draft-mtp-vocab N |
Computes draft logits over a vocabulary subset; verification is unchanged. |
--spec-draft-vocab-dynamic N |
Low-rank dynamic draft LM head (off by default). |
--spec-draft-max-slots N |
Stops drafting when many server slots are busy. |
--spec-n-rs-seq N |
Caps the recurrent-state rollback copies on linear-attention models. |
--spec-prefill |
A small draft model picks which prompt tokens to prefill, to cut time-to-first-token on long prompts. |
Server
- Multi-point reasoning budget:
--reasoning-budget-intro-message,-intro-mode,-soft-message,-soft-ratio,-grace-tokens,-enable. - Hidden model presets (
hiddenin the models.ini). - Server dashboard (
tools/server/dashboard). - Freed KV cells are zeroed, so output stays repeatable at depth.
Fixes
- MTP decode numerics for Flash-Next (#234)
- QSA: no abort when image cells share a tail slot (#261), graph reservation on an empty cache (#262), image recovery (#235)
- Vulkan: truncated MoE helper ends restored (#240)
- Tests: large quantized tensors filled from a block pool (#259)
Known issues
- The Ubuntu zips store the binaries without the execute bit; run
chmod +x llama-*after unzipping. - The Windows zips are built in CI and have not been tested on Strix Halo hardware.
- The
ubuntu-llvmpipeVulkan CI job is red on master:test-llama-context,test-draft-vocabandtest-qwen4exp-mtp-cache-onlyfail on the software rasterizer. The GPU paths in this release were tested on gfx1151 hardware.
Downloads
| Ubuntu x64 | Windows x64 | |
|---|---|---|
| ROCm (gfx1151, ROCm 10.1 bundled) | llama-b1049-ubuntu-rocm-gfx1151-x64.zip |
llama-b1049-windows-rocm-gfx1151-x64.zip |
| Vulkan | llama-b1049-ubuntu-vulkan-x64.zip |
llama-b1049-windows-vulkan-x64.zip |
For Vulkan we test on Mesa 26.2.4.
Contributors
gaetan-puleo, pwilkin, rcfa, LaurentZuijdwijk, Nathanw1014