Skip to content

Halobox v1.0.0

Latest

Choose a tag to compare

@Nathanw1014 Nathanw1014 released this 11 Nov 14:52
0c1021e

Halobox v1.0.0

Build b1049, master 0c1021e1f. Ubuntu and Windows zips for ROCm (gfx1151) and Vulkan.

Summary

Qwen3.8 Flash-Next on ROCm (gfx1151). With UD-Q4_K_XL on a 128 GB Strix Halo: 2089 t/s pp2048 at depth 0, 1680 t/s at 32k and 1621 t/s at 100k, 26.5 t/s decode. With the MTP draft head, decode reaches 36-53 t/s depending on the prompt.

On a 64 GB box, run Gyro-M (TQ2_T, a format only halo-box loads) on Vulkan: 30 t/s decode, 35-61 t/s with the MTP draft, about 1110 t/s pp2048.

Also measured on this release: Qwen3.8 27B dense (22-38 t/s decode with its MTP head, Vulkan), Qwen3.6 35B-A3B MoE (63 t/s decode, 3100-3400 t/s prefill) and Ternary Bonsai 2 27B (PTQ1_0, 28 t/s decode).

halo-box also loads formats upstream llama.cpp cannot (Gyro TQ2_T, Bonsai PTQ1_0, ROCmFP4/FPx), and supports 1M context and multi-slot serving for Flash-Next.

Performance at a glance

b1049 release zips, Strix Halo (Ryzen AI Max+ 395, Radeon 8060S), llama-bench, flash attention on, 3 repetitions. Flash-Next with -lzm on-direct; each model at its best ubatch (2048 for all of them on this build).

model box backend pp2048 pp2048 @32k pp2048 @100k tg128 tg128 @32k tg128 @100k
Flash-Next UD-Q4_K_XL 128 GB ROCm 2089 1680 1621 26.5 24.5 23.5
Flash-Next Gyro-M (TQ2_T) 64 GB Vulkan 1112 957 30.7 29.1
Flash-Next Gyro-M (TQ2_T) 64 GB ROCm 1135 1007 27.3 25.5
Flash-Next UD-Q4_K_XL 128 GB Vulkan 1522 1268 1160 27.9 27.1 25.9
Qwen3.6 35B-A3B UD-Q4_K_XL 128 GB Vulkan 3096 1757 63.1 51.7
Qwen3.6 35B-A3B UD-Q4_K_XL 128 GB ROCm 3397 1529 57.3 48.8
Qwen3.8 27B UD-Q4_K_XL 64 GB Vulkan 520 387 12.7 11.2
Qwen3.8 27B UD-Q4_K_XL 64 GB ROCm 458 314 12.2 10.3
Ternary Bonsai 2 27B (PTQ1_0) 128 GB Vulkan 321 272 28.0 21.8

With a q8_0 KV cache (-ctk q8_0 -ctv q8_0) decode at 32k rises to 55.6 t/s on 35B-A3B Vulkan (+7.6%), 50.6 on 35B-A3B ROCm (+3.7%) and 11.8 on 27B Vulkan (+5.4%), with prefill unchanged.

Decode in llama-server, 256 tokens per prompt, with and without the MTP draft head (Q8_0 head, 3 draft tokens):

prompt UD-Q4_K_XL, ROCm + MTP Gyro-M, Vulkan + MTP
code 26.3 47.2 30.0 50.4
prose 26.3 35.8 30.2 34.9
reasoning 26.4 49.6 30.2 54.1
edit (837-token prompt) 26.6 53.3 30.6 60.9
28.6k-token prompt 24.3 42.3 28.0 46.1

Qwen3.8 27B dense with its MTP head (MTP/mtp-Qwen3.8-27B-Q4_0.gguf), llama-server, q8_0 KV:

prompt Vulkan + MTP ROCm + MTP
code 12.5 31.9 12.1 27.9
prose 12.5 22.2 12.1 19.4
reasoning 12.5 33.5 12.1 29.8
edit (837-token prompt) 12.4 38.1 12.1 33.5
28.6k-token prompt 11.6 24.8 10.5 20.7

Recipes

The b1049 Ubuntu zips store the binaries without the execute bit. After unzipping, run chmod +x llama-* in the folder.

128 GB Strix Halo: UD-Q4_K_XL on ROCm (highest prefill)

  1. Download the ROCm (Ubuntu) zip and unzip it. ROCm 10.1 is bundled.
  2. From unsloth/Qwen3.8-Flash-Next-GGUF, download the 4 UD-Q4_K_XL shards (about 111 GB) and MTP/mtp-Qwen3.8-Flash-Next-Q8_0.gguf.
  3. Run:
    llama-server -m Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf \
      -lzm on-direct -c 40960 -b 4096 -ub 4096 -np 1 -md mtp-Qwen3.8-Flash-Next-Q8_0.gguf

Expected: 36-53 t/s decode with the draft and 1416 t/s prefill on a 28.6k-token prompt. Peak GPU memory is about 89 GB.

64 GB Strix Halo: Gyro-M on Vulkan

UD-Q4_K_XL does not fit in 64 GB. Gyro-M is a trellis-coded 2-bit quant of the same model. It is one 92 GB file, of which about 38 GB stays on the GPU (45 GB with the draft head).

  1. Download the Vulkan (Ubuntu) zip and unzip it.
  2. Download Qwen3.8-Flash-Next-Gyro-M-TQ2_0.gguf from agentionai/Qwen3.8-Flash-Next-Gyro-GGUF, and MTP/mtp-Qwen3.8-Flash-Next-Q8_0.gguf from unsloth/Qwen3.8-Flash-Next-GGUF.
  3. Run:
    llama-server -m Qwen3.8-Flash-Next-Gyro-M-TQ2_0.gguf \
      -lzm on-direct -c 40960 -ub 2048 -md mtp-Qwen3.8-Flash-Next-Q8_0.gguf

Expected: 35-61 t/s decode with the draft, and about 10 GB of RAM left free. The ROCm zip also runs Gyro-M with the same output quality (perplexity 2.2196 vs 2.2178 on Vulkan). It has 2-5% more prefill but about 12% less decode.

Qwen3.8 27B dense on Vulkan

  1. Download the Vulkan (Ubuntu) zip and unzip it.
  2. From unsloth/Qwen3.8-27B-GGUF, download Qwen3.8-27B-UD-Q4_K_XL.gguf (17.6 GB) and MTP/mtp-Qwen3.8-27B-Q4_0.gguf.
  3. Run:
    llama-server -m Qwen3.8-27B-UD-Q4_K_XL.gguf \
      -c 40960 -ub 2048 -np 1 -ctk q8_0 -ctv q8_0 -md mtp-Qwen3.8-27B-Q4_0.gguf

Expected: 22-38 t/s decode with the draft (12.5 without), about 520 t/s pp2048. Fits a 64 GB box with about 33 GB of RAM left free. Vulkan is ahead of ROCm on this model in both prefill (+13%) and decode.

Qwen3.6 35B-A3B MoE

  1. Download the Vulkan (Ubuntu) zip and unzip it.
  2. From unsloth/Qwen3.6-35B-A3B-GGUF, download Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf (22.4 GB).
  3. Run:
    llama-server -m Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf -c 40960 -ub 2048 -np 1 -ctk q8_0 -ctv q8_0

Expected: about 63 t/s decode and 3100 t/s pp2048 (llama-bench). The ROCm zip has about 10% more prefill on short prompts; Vulkan has more decode and more prefill at 32k.

What the flags do

The other defaults are already right. GPU offload and flash attention are automatic, the MTP draft type is read from the draft file, and 3 draft tokens is the default.

flag why
-lzm on-direct Keeps the per-layer-embedding table (27 GB on UD-Q4_K_XL) on disk and reads only the rows each batch needs. The default copies the whole table into RAM on Strix Halo.
-c 40960 Without -c, the server sizes the KV cache for the model's full trained context.
-b 4096 -ub 4096 The defaults are 2048 / 512. With the draft head, 4096 gives about 7% more prefill on long prompts than 2048.
-np 1 One slot for a single user, about 5% more prefill than the default 4 slots. Leave it off when serving several clients.
-ctk q8_0 -ctv q8_0 q8_0 KV cache: 4-8% more decode at depth on the dense and MoE models, at no prefill cost. Not used for Flash-Next.
-md The MTP draft head speeds up decode but costs prefill on long prompts (1766 vs 1416 t/s on 28.6k tokens). Drop it for long-document or RAG work.

Models and formats

TQ2_T (Gyro) Trellis-coded 2.125 bpw weights, with kernels on CPU, Vulkan and HIP. Upstream llama.cpp cannot load Gyro files.
TQK6 / TQK7 / TQP5 / TQP6 / TQP7 Bit-packed trellis siblings of TQ2_T, on CPU, Vulkan and HIP.
PTQ1_0 (Bonsai / PrismML ternary) quantize, CPU and Vulkan. There are no HIP kernels yet.
ROCmFP4 / ROCmFPx Q4_0_ROCMFP4 variants and Q2/Q3/Q6/Q8_0_ROCMFPX weight formats, on CPU, Vulkan and HIP.
Flash-Next long context QSA sparse attention kernels on both Vulkan and HIP, 1M context (#215), multi-slot compute memory (#216), and the per-layer-embedding table read from disk (-lzm on-direct).

New in this release:

  • Flash-Next 1M context (#215) and multi-slot compute memory (#216).
  • The QSA prefix is kept across images in multimodal prompts (#235).
  • Convert writes the QSA compress ratios when newer transformers name the layers indexed_attention (#251). GGUFs converted with transformers 5.22 or later before this fix have all ratios at 0 and run dense attention at long context. Re-convert them.
  • Routed IQ1_M / IQ2_S / IQ2_XXS / Q2_0 decode on gfx1151, for faster prefill on the low-bit Flash-Next quants (#260).

Speed work

Vulkan (RADV)

  • MoE gate, up and SwiGLU in one launch, with streamed down projection (#223), including q5_1 (#224)
  • Dense q8_0 x f16 GEMM, on by default (#245, #246)
  • q8_0 / q6_K mat-vec without chunking (#241), split_k at small n (#242), XL tile margin (#244)
  • QSA score / expand fusions (#228, #229)

ROCm / HIP on gfx1151

  • Flash-Next prefill (#213)
  • Routed low-bit decode tiers (#260)
  • Packed MMVQ for small Q8_0 batches
  • Also carried over from earlier builds: the MMB routed-expert path, hyper-connection kernels, the fused GDN chain, RDNA 3.5 flash-attention tiles, and a zero-SGPR-spill check in CI.

Loading: With an explicit -lm mmap on an integrated GPU, weight copies are staged through a host buffer (#214). Flash-Next UD-Q4_K_XL on ROCm used to not finish loading after 400 s; it now loads in 56 s. The default load mode is unchanged (48 s).

Speculative decoding

halo-box adds these on top of upstream's MTP and DFlash drafting:

flag what it does
--spec-draft-adaptive Sizes each draft from measured acceptance instead of always drafting --spec-draft-n-max. On Flash-Next with the MTP head, the fixed default of 3 was as fast or faster.
--spec-draft-mtp-pcum P Stops an MTP chain when the product of top-1 probabilities drops below P.
--spec-draft-mtp-vocab N Computes draft logits over a vocabulary subset; verification is unchanged.
--spec-draft-vocab-dynamic N Low-rank dynamic draft LM head (off by default).
--spec-draft-max-slots N Stops drafting when many server slots are busy.
--spec-n-rs-seq N Caps the recurrent-state rollback copies on linear-attention models.
--spec-prefill A small draft model picks which prompt tokens to prefill, to cut time-to-first-token on long prompts.

Server

  • Multi-point reasoning budget: --reasoning-budget-intro-message, -intro-mode, -soft-message, -soft-ratio, -grace-tokens, -enable.
  • Hidden model presets (hidden in the models .ini).
  • Server dashboard (tools/server/dashboard).
  • Freed KV cells are zeroed, so output stays repeatable at depth.

Fixes

  • MTP decode numerics for Flash-Next (#234)
  • QSA: no abort when image cells share a tail slot (#261), graph reservation on an empty cache (#262), image recovery (#235)
  • Vulkan: truncated MoE helper ends restored (#240)
  • Tests: large quantized tensors filled from a block pool (#259)

Known issues

  • The Ubuntu zips store the binaries without the execute bit; run chmod +x llama-* after unzipping.
  • The Windows zips are built in CI and have not been tested on Strix Halo hardware.
  • The ubuntu-llvmpipe Vulkan CI job is red on master: test-llama-context, test-draft-vocab and test-qwen4exp-mtp-cache-only fail on the software rasterizer. The GPU paths in this release were tested on gfx1151 hardware.

Downloads

Ubuntu x64 Windows x64
ROCm (gfx1151, ROCm 10.1 bundled) llama-b1049-ubuntu-rocm-gfx1151-x64.zip llama-b1049-windows-rocm-gfx1151-x64.zip
Vulkan llama-b1049-ubuntu-vulkan-x64.zip llama-b1049-windows-vulkan-x64.zip

For Vulkan we test on Mesa 26.2.4.

Contributors

gaetan-puleo, pwilkin, rcfa, LaurentZuijdwijk, Nathanw1014