Skip to content

hipfire 0.4.4

Latest
Latest

Choose a tag to compare

View all tags
@github-actions github-actions released this 27 Oct 07:16
· 378 commits to master since this release
v0.4.4
75c28fc

hipfire v0.4.4

Built from 943b5323da (origin/beta at the 07:00Z cutoff on 2026-10-27); the v0.4.4 tag is 75c28fc064, which only dates the CHANGELOG and sets CITATION to 0.4.4. The previous release is v0.4.3.1 (2026-10-24), which followed v0.4.3 (2026-10-24). The full change list is the v0.4.4 section of CHANGELOG.md.

This release leads with speculative-decode speed on the 7900 XTX (two community PRs by @Ben1078), generic tensor parallelism as the default dense TP route, vision on by default with a streamed tower, and batched prefill for every LLaMA-arch model. It also lands a large Strix Halo (gfx1151) wave from @fivetide and @HUSRCF, hardens multi-slot swap (@ghazni101), and ships the TLS and CI repairs that left beta green.

Provenance matters for several rows below. Where a row says contributor-measured, the merge gates ran the build, the CPU tests and a smoke on a different card, not the contributor's benchmark on the contributor's card; the card and fixture are named so the number can be reproduced.

Headline numbers

Change Hardware and fixture Result Quality
gfx1100 int8-activation small-batch verify GEMV, default on (#987, @Ben1078) One 7900 XTX (hipx), qwen3.8-27b.mq4-xts, hipfire bench --spec mtp, Q8 KV, 256 tokens, committed mtpfloor prompts, fresh-process ABBA, precompiled pack. Maintainer gate (pr987/REPORT.md) Decode code-1k 104.7 -> 127.4 tok/s (1.217x), prose-1k 106.7 -> 131.1 (1.229x), code-32k 81.8 -> 95.5 (1.168x), prose-32k 51.7 -> 58.9 (1.139x); prefill equal; tau equal except prose-32k 1.38 -> 1.32. Author's runs (9K 68.5 -> 85.6, 39K 51.7 -> 61.0 tok/s) agree in direction NUMERIC, KLD-gated, not byte-identical. Standard-scoring .kldseq byte-identical to the gfx1100 pins in both arms (the route is never taken at 1023-row prefill); verify-shaped (--scored-max-batch 4) ON minus OFF: WT2 +0.000073, code24 -0.000054; serve battery 5/5 stop, text read. HIPFIRE_GFX1100_Q8X_VERIFY=0 restores the WMMA route
gfx1100 WMMA split-K verify attention (1..=8 rows), default on (#989, @Ben1078) One 7900 XTX, same model and method. Standalone numbers are the author's (beta without #987); the stacked numbers are the maintainer gate (stacked on #987, fresh-process ABBA, MTP) Standalone: 9K 64.8 -> 69.6, 39K 51.8 -> 64.3, 101K 43.6 -> 68.5 tok/s. Stacked on #987: code 32K 95.7 -> 121.2 (1.27x), prose 32K 59.0 -> 73.3 (1.24x), 1K neutral. Kernel (4 rows, 9K/40K/100K): 194.5/777.0/2080.0 us -> 37.3/130.6/309.5 us NUMERIC, KLD-gated. Author: WT2 -0.000004, code24 +0.000225 (both inside the bar). Gate, verify-shaped ON-OFF at batch 4/8: WT2 -0.000051/+0.000593, code24 -0.000100/-0.000086, no paired CI. HIPFIRE_GFX1100_WMMA_VERIFY_ATTN=0 (or HIPFIRE_VERIFY_ATTN=0) restores the incumbent
Dense Qwen3.5/3.6/3.8 serve --tp N loads the generic mesh executor by default (#927, HIPFIRE_DENSE_TP_ROUTE=auto) 4x R9700 (hiptrx), qwen3.8-27b.mq4-xts, bench --matrix --spec off, 3 fresh processes per arm, median of the run medians, mesh vs hand route. Maintainer-measured. TP2 rows are on beta 3385df7ea5 (generic TP runs the LM head on rank 0 only; tp-followup-runs/bench-tp2/); TP4 rows are on the earlier build (generic-tp-hiptrx-runs/final-perf-tp4/, before the LM-head change) TP2 pp512 2,866.9 -> 3,293.4 (+14.9%), pp2048 2,746.1 -> 3,358.5 (+22.3%); TP2 decode ties the hand route: tg64@128 54.95 -> 54.89, tg64@4096 54.24 -> 54.19; TP4 pp512 1,919.4 -> 2,061.0 (+7.4%), tg64@128 43.96 -> 51.77 (+17.8%); short prompts are slower: pp16 496.2 -> 479.6 (-3.3%), pp32 880.7 -> 859.0 (-2.5%) Maintainer hwtest on 3385df7ea5: logits fnv64 equal to the hand route at TP2 and TP4 (tp-followup-runs/hw-new-tp{2,4}.log); serve battery and chain ran on TP2 and TP4 with the default route on the earlier build. HIPFIRE_DENSE_TP_ROUTE=hand rolls back
Image turns: batched prefill, WMMA vision tower, streamed tower, vision_mode default auto R9700, qwen3.8-27b.mq4-xts + qwen3.8-27b-vision.hfq, 1024x1024 image, HTTP TTFT, greedy TTFT 37.3 s -> 9.3 s (batched LLM prefill alone) -> 0.51 s (with the tower attention and GEMM changes). Streamed tower, 1024^2 TTFT: R9700 0.51 s, 7900 XTX 0.81 s (PCIe 4.0 x4 host), Strix Halo 1.84 s. Idle VRAM vs off: auto +127.6 MiB (R9700) / +142.2 MiB (7900 XTX), resident +958 MiB Tower GEMM bit-identical to the old kernel; tower attention rel-L2 1.5e-3..2.0e-3 vs an HF F32 reference (naive path 0.8e-3..2.0e-3). The streamed tower's embeddings are byte-identical to the resident path on gfx1201 for every committed benchmarks/vision/images image plus a 1024^2 fixture. Image-prefill batching is NUMERIC: greedy text can diverge from per-token after a few dozen tokens
LLaMA-arch models (LLaMA, Mistral, SmolLM2) prefill in batched chunks (#982) SmolLM2-360M mq4v1, gfx1201 (gate); Llama-3.2-1B HFQ4G256, gfx1100 serve (gate) SmolLM2-360M 148 -> 3,705 tok/s at 1,169 tokens (87 -> 3,127 in the second fixture); Llama-3.2-1B 463 -> 8,130 tok/s. Author, gfx1151 SmolLM2-360M PARO G64: 101.4 -> 1,753.5 (712 tokens), 41.8 -> 1,229.0 (3,445) NUMERIC, not byte-identical. SmolLM2 run 30/32 identical, serve battery diverges after a matching prefix (coherent); Llama-3.2-1B 8/9 identical. HIPFIRE_PREFILL_BATCHED=0 restores per-token
hd64 Q8 flash decode tile (#978, HIPF-124) SmolLM2-360M PARO, gfx1151 (author); gate: SmolLM2-360M q8 gfx1201, Llama-3.2-1B gfx1100 gfx1151 decode 72.1 -> 126.0 (712 ctx), 24.7 -> 121.9 (3,445), 11.5 -> 112.3 tok/s (8,196). Gate: gfx1201 26.6 -> 265.5 tok/s at 1.1K; gfx1100 Llama-3.2-1B 153 -> 508 tok/s at 1.1K Within 1e-6 of the old kernel and an f64 oracle, but a different f32 summation order: greedy text equal at 712/3,445/8,196 tokens, differs after 35 characters at 7,066; SmolLM2 q8 text identical on 8 of 10 prompts (two near-tie flips at 4K). Not byte-identical
Strix Halo Qwen3-8B (llama-family) DFlash and DSpark: AR's batched prefill and lowered decode (#958, @fivetide) Strix Halo, qwen3-8b.mq4, 7,741-token prompt, 512 greedy tokens. Contributor-measured DFlash prefill 538 s -> 13.1 s, decode 7.4 -> 20.8 tok/s; DSpark prefill 537 s -> 13.4 s, decode 5.1 -> 11.8 tok/s Text equals AR at 51/1,896/3,750/7,741-token prompts (gfx1151); gfx1201 one 1,970-token prompt byte-identical; gfx1100 still diverges from AR near a tie (margin not measured)
Strix Halo 9B DFlash (#961 + #962, @fivetide) Strix Halo, qwen3.5-9b.mq4 + qwen35-9b-dflash-mq4.hfq. Contributor-measured, shared box (indicative) 2K 45.7 -> 59.6 tok/s, 4K 36.0 -> 56.5; XLDS alone 2K 45.5 -> 57.9, 4K 35.8 -> 42.7 Byte-identical to the previous build (md5 at 2K and 4K); 9B DFlash differs from AR with or without these changes (pre-existing on gfx1151)
Flash-Next on the R9700: decode-share, default on (admitted loads only) One R9700 (card B), qwen3.8-flash-next-gptq3.mq4, PLE q4, 16,384-row chunk; 3 rounds, one fresh process per arm per round, arm order rotated, decode-share on vs off on the release code (125b2b5ca9); the shadows row is from a0e571597f MTP decode (11-prompt suite) 61.82 vs 51.14 tok/s with it off (+20.9%); pp16384 6,677.5 vs 6,686.7 (-0.1%); 16K TTFT 2,483.7 vs 2,469.9 ms; second token after a 16K prompt 78.7 ms vs 22.7 ms. Shadows follow-on: resident experts 7,590 -> 8,184, MTP 60.23 -> 61.75 (+2.5%) Byte-identical (12-prompt greedy ids, serve text, kldseq WT2/code24). Greedy MTP equals AR after prompts of 512 tokens or more: the first 0.4.4 RC failed this check because MTP verify and draft forwards kept running on the lent banks; fixed before release (125b2b5ca9). HIPFIRE_QWEN4_DECODE_SHARE=off disables it
Flash-Next shared-expert prefill on INT8 (default on, exact gfx1201) One R9700 (card B), pp16384 at the 16,384 chunk, 3 fresh processes per arm 6,459.1 -> 6,604.5 tok/s (+2.25%); MTP decode flat NUMERIC, KLD-gated. code24 +0.0050 [-0.0005, +0.0106], WT2 +0.0014 [-0.0051, +0.0080]. A ~11K-token prose prompt diverges from the first token (coherent). HIPFIRE_QWEN4_SHARED_INT8=0 restores F16

All other numbers in this document come from the named change-level measurements in the CHANGELOG entries. Numbers from different sessions are never chained.

Highlights

  • 7900 XTX (gfx1100): MTP speculative verify is 1.14x to 1.23x faster from #987 alone (gate, 1K to 32K contexts) and a further 1.24x to 1.27x at 32K from #989 stacked on it, from two PRs by @Ben1078 (table above): an int8-activation GEMV for 2..=5-row MQ4G256V2 verify (#987) and a WMMA split-K verify attention (#989). Both are default on for exact gfx1100 and NUMERIC, not byte-identical; both ship their kernels in the gfx1100 pack, so nothing JIT-compiles on the first request. #987 resolves #946; #989 is the first follow-up promised in #947. The reporter's own patched-build figure for the stack (27B mq4-xts MTP K=3, 67/59/43 -> 102/97/90 tok/s, #947) is a different comparison (v0.4.2.1 vs patched) and is not a release gate.
  • 7900 XTX (gfx1100): MTP draft over a 65,536-token vocabulary subset (#998, @Ben1078, 9746dbc0c5): on exact gfx1100 the MTP draft runs over a fixed 65,536-row subset sliced from the trunk lm_head, default on; HIPFIRE_GFX1100_DRAFT_VOCAB=0 restores the full head. It copies ~170 MiB of lm_head rows at load. NUMERIC, not byte-identical for greedy MTP text: target KLD pins are byte-identical, and greedy MTP text matches beta on 3 of 4 mtpfloor prompts (code-1k diverges at a near-tie). Stacked on #987 and #989, a fresh-process ABBA on the 27B mq4-xts gives 1.074x to 1.094x decode at 1K and 32K (mq044b-pr998/). Measured only on the 27B at up to 39K context.
  • Dense tensor parallelism now defaults to the generic device-mesh executor. serve --tp N on Qwen3.5/3.6/3.8 dense models takes the mesh route (every KV backend, VMM included); auto falls back to the hand route, with a logged reason, for speculation.mtp = on and uneven TP3/TP5 splits. TP2/TP4 prefill is 7% to 22% faster on 4x R9700 and TP2 decode ties the hand route; 16 to 32-token prompts prefill 2.5% to 3.3% slower than on the hand route. Speculation auto at --tp > 1 now drops an auto-paired DFlash draft with a log line instead of failing (explicit on is still refused), and a VMM-KV reset panic after growth is fixed. Muse Glimmer dense TP runs on the same executor (developer only: the loader still serves Glimmer on one device).
  • Penalties work at --tp N, and the A3B registry no longer injects a presence penalty. Repeat, presence and frequency penalties are applied to rank 0's logits on Qwen3.5/3.6/3.8 dense TP (mesh and hand routes) and Qwen3.5-MoE EP, so hipfire run qwen3.5:4b --tp 2 and serve requests carrying registry sampling are no longer refused (presence_penalty is not supported at tp>1 / HTTP 400) (2c1d4157fb). The Qwen3.5 and Qwen3.6-35B-A3B registry entries now default to presence_penalty 0.0 (was 1.5; 14 tags), so greedy requests to those models are no longer silently demoted from DFlash to AR by a penalty the user never sent (4b56086b04).
  • Vision is on by default. vision_mode defaults to auto: a present vision-tower sidecar is loaded and images are served with no configuration. The Qwen3.5/3.8-VL tower is streamed from host per block through two fixed VRAM staging slots, so a vision load costs about 128-142 MiB of idle VRAM rather than ~0.94 GiB (vision_mode=resident keeps the whole-tower upload; off restores the text-only load). Image turns prefill in batched chunks, and the tower uses WMMA flash attention (head_dim 72) and the LDS-tiled GEMM: a 1024x1024 turn is 0.51 s to first token on the R9700 (was 37.3 s). Streaming other media towers is tracked in #988.
  • Batched prefill for the whole LLaMA family. LLaMA, Mistral and SmolLM2 now take Qwen3's batched prefill on gfx11/gfx1201 with Q8 KV (serve and generic DFlash/n-gram spec), numeric rather than byte-identical; an hd64 decode tile removes the context fall-off for SmolLM2, Llama-3.2-1B and TinyLlama-class models (also numeric).
  • Strix Halo (gfx1151) Flash-Next and DFlash wave (@fivetide, @HUSRCF): streamed long-prefill MoE gate/up, shared RW4 HC gates, a fused HC-down, persistent HC-up v2 and the one-pass MoE front (the last three byte-identical, default on, kill switches below), the XLDS and VerifyAttn twins for 9B DFlash, head_dim-128 VerifyAttn for Qwen3-8B, sampled decode on mixed-codec slab models without the per-token host idle (1.70 -> 0.41 ms before each forward), and a Flash-Next reload that is no longer refused for RAM left in TTM's page pool (the 0.4.3 known issue).
  • Flash-Next on the R9700: decode-share lends the long-prefill scratch to the coldest resident experts' VRAM and spends the reclaimed reserve on more resident experts (default on for admitted loads; read the trade-off under Known issues). New exact gfx1201 kernels: MQ6G256V2 stream-split verify (gfx1201 and gfx1100), MQ4G256V2 gate+up tile loop (27B DFlash decode code 291.8 -> 314.7 tok/s), the A3B .mq4rs fused routed down, QSA norm+RoPE in the int8 GEMM epilogue, and a shared-expert INT8 prefill (numeric).
  • Long-context DFlash on the R9700 (gfx1201): the native-fp8 split-KV verifier is default on (#921, #922, @paleluna82-del). It runs on exact gfx1201 for a dense 27B-class model (H24/KV4/D256) with fp8 KV (the gfx1201 default), live context at or above 8,192, and at most 16 verify rows. Off switch: HIPFIRE_GFX12_FA2_SPLIT_VERIFY_FP8=0. Stock config (R9700, 27B mq4-xts with the registry DFlash draft, block 16, verify graph on; median per-process decode tok/s over 3 fresh processes per arm): 4K 65.90 -> 64.33 (ranges overlap, below the floor, identical text), 16K 53.30 -> 59.60 (+11.8%), 32K 40.40 -> 51.40 (+27.2%), 64K 29.93 -> 43.37 (+44.9%), 128K 18.93 -> 33.70 (+78.0%). NUMERIC, not byte-identical (coherent output). It is capture-safe with the verify graph (a route-class re-capture of about 35 ms per switch), and lane-batched CB verify equals singleton. MTP and ngram verify blocks above 8,192 take it too (kernel trace). HIPFIRE_GFX12_FA2_SPLIT_COUNT (Q8 KV, up to 64 splits) stays opt-in.
  • Multi-slot serve: swap budgets, private spill directories and bounded sessions (@ghazni101, #945; closes #943 and #944). Host and disk budgets and the swap directory are configurable (--swap-host-bytes, --swap-disk-bytes, --swap-dir; defaults 16 GiB and an 8 GiB disk cap), snapshots carry a real model fingerprint, abandoned parked sessions are swept (serve.multi_slot_session_ttl_secs, default 3600), and /health, /stats and /metrics publish the swap tier.
  • Security and CI. TLS crates (rustls, rustls-webpki, webpki-roots) and Tokio 1.53.3 are bumped to clear six RustSec advisories (#942, @HUSRCF), the Windows/MSVC link of the daemon is repaired by gating the native HSA loader to Linux, and the TUI golden and railgun inventory tests that kept beta red are fixed.
  • Windows installer and runtime (beta 0298052a63, #995): scripts/install.ps1 no longer starts with a double UTF-8 BOM, so the installer parameters work again (every v0.4.1 to v0.4.3.1 release ignored them; #979). The MSVC build links (the #979 link half does not reproduce). Release kernel packs are accepted on native Windows, amdhip64.dll is loaded so the driver's System32 copy cannot shadow the HIP SDK's, and Windows UMA releases the model mapping like Linux. Windows CI now smokes the packaged binaries with a System32-only PATH and checks PE imports. A volunteer test kit (scripts/windows-check.ps1) and issue form (windows-test-report.yml) are included. Windows GPU support is still untested; see Known issues.
  • Fixes to v0.4.3 regressions: MoE models with a hidden size that is not a multiple of four 256-wide groups no longer produce all-zero gate/up (#956); ParoQuant safetensors directories load again under VMM KV (#955); a retired DFlash request's grammar error is a validation error again (#950); a pre-row-padded LLaMA .mq4 that crashed the GPU on its first generate is refused at load (#954).
  • Greedy DDTree tree verify is deterministic across fresh runs (#971); Gemma 4 EAGLE draft acceptance no longer depends on the previous request (#957); a DeepSeek-V4 MQ2R prefill scratch read on gfx1151 is fixed (#973); hipfire pull now records the verification proof for a split base and its sidecars (a repeat pull no longer re-hashes ~70 GB).
  • Developer and research: an opt-in, certified railgun DFlash window for 27B on gfx1201 (HIPFIRE_DFLASH_RAILGUN_WINDOW=1, default off, see Known issues), railgun conditional nodes (IF and bounded WHILE, #867), the NPU co-processor as a generic heterogeneous stage (#864), a K = 5120 NPU stage for 27B dense in-projections, declarative layer programs for LFM2 continuous batching and DeepSeek-V4 decode, and hipfire setup fixes for the amdrocm layout (@amittell).
  • Refactor and lab: #997 (@fivetide, 98b90ba011) moves assistant-turn cache publication behind one seam; no default change. #996 (@fivetide, f7d7545523) is a lab example only.

7900 XTX (gfx1100)

  • Verify GEMV (#987) and verify attention (#989): see Headline numbers. Both are NUMERIC and default on for exact gfx1100 only. The GEMV takes every 2..=5-row MQ4G256V2 projection with k % 256 == 0 and k <= 32768, eager only (HipGraph capture and Redline recording keep the WMMA tiers); the attention takes dense Qwen H24/NKV4/HD256 shapes past HIPFIRE_FA_PERTOKEN_MIN_CTX (4096).
  • fwht4 / fwht3 KV: a VerifyAttn twin (attention_verify_gqa_fwht{4,3}_gfx1100, hardware-gated 63/63 oracle cases), the FA2 prefill twin for fwht4 K (default on for --kv-mode fwht4 with Q8_0 V), and the DeltaNet chunk-scan prefill for fwht4/fwht3 (about 0.4 s per 8K tokens saved by the scoping estimate; NUMERIC, KLD-gated).
  • MQ6G256V2 stream-split few-row verify kernels (byte-identical, HIPFIRE_MQ6_ROWS_SPLIT=0 restores).
  • Llama-3.2-1B HFQ4G256 serve prefill 463 -> 8,130 tok/s and decode 153 -> 508 tok/s at 1.1K (gate, numeric).

R9700 (gfx1201)

  • Image path (above): 0.51 s TTFT at 1024x1024, tower 9.2 s -> 0.80 s (attention) -> 0.14 s (GEMM), batched image prefill.
  • Flash-Next: decode-share (default on, admitted loads), shared-expert INT8 prefill (numeric), QSA norm+RoPE epilogue, MQ6 stream-split verify, A3B .mq4rs routed-down and shared-expert k512 two-row kernels (tg128 156.8 -> 197.0 tok/s HIP, 231.7 -> 242.5 PM4 by the merge-log gate), gate+up verify tile loop (27B DFlash decode code 291.8 -> 314.7, prose 58.7 -> 60.5 tok/s). All of these except the shared-expert INT8 prefill are byte-identical.
  • Dense TP2/TP4 on the generic mesh route (table above).
  • fwht4 K FA2 prefill and the fwht3 FA2 ceiling raised to 262,144 tokens (was 32,768), VerifyAttn fwht twins, DeltaNet chunk-scan for fwht KV.
  • Split-KV DFlash verifiers (#921, #922, @paleluna82-del): the native-fp8 split verifier is default on for exact gfx1201, 27B-class H24/KV4/D256, fp8 KV, live context at or above 8,192 and at most 16 verify rows (the stock-config table is under Highlights; off: HIPFIRE_GFX12_FA2_SPLIT_VERIFY_FP8=0). The Q8 KV split count HIPFIRE_GFX12_FA2_SPLIT_COUNT accepts up to 64 splits (16 is the useful value for Q8 KV) and stays opt-in; 64K serve DFlash with it 34.1 -> 48.0 tok/s vs the default route (+41%). Maintainer-measured on hiptrx R9700, 3 fresh processes per arm; not byte-identical (coherent). Independent regate: splitkv-v2-regate-runs/.
  • The 0.4.3 Flash-Next defaults (16,384-row chunk, hot set) are unchanged, and the 0.4.3 R9700 hot-set decode trade-off is gone on real text: measured on one R9700 (split base, 3 fresh processes per arm, hotset-confirm-044/), the hot set now makes --spec auto decode 34% to 50% faster than HIPFIRE_QWEN4_EXPERT_HOTSET=0 (prose 49.7 vs 37.2 tok/s, code 80.6 vs 55.2 and 78.6 vs 52.4) with pp16384 unchanged (6,648 vs 6,646 tok/s); what remains is pp512 4.2% slower (1,344 vs 1,403) and synthetic-id tg128 4.8% slower (36.0 vs 37.8, random token ids with no routing locality). Decode-share keeps 8,839 experts resident, up from 5,919 in 0.4.3.

Strix Halo (gfx1151)

  • Flash-Next prefill: persistent HC-up v2 (kernel 1.26x/1.29x at 8,192/16,384 rows), fused HC-down + activation (kernel 1.37x to 1.46x at 8,192/16,384 rows, crossover at 1,024 rows), one-pass MoE front (0.952 -> 0.646 ms at 8,192 rows). All byte-identical by GPU oracle (152/152, 36/36, 55/55) and a 16K+64 long-prompt greedy identity, default on, with HIPFIRE_QWEN4_HC_UP_V2=0, HIPFIRE_QWEN4_HC_DOWN_FUSED=0 and HIPFIRE_QWEN4_MOE_FRONT_FUSE=0 as kill switches. Native MQ6 PM recording/capture admission is not in this release; the opt-in prepare/replay API from #974 is (see below).
  • DFlash: llama-family prefill/decode (#958), 9B XLDS + GQA-4 VerifyAttn + key-split draft attention (#961, #962), Qwen3-8B head_dim-128 VerifyAttn (#977, contributor 2K 12.1 -> 15.1, 4K 9.7 -> 14.3, 8K 23.2 -> 28.0 tok/s; gate 1K/2K/8K 12.9/11.0/20.2 -> 15.3/13.7/26.8; AR 45.8/44.0/38.6; DFlash stays slower than AR on this model at tau 0.2-0.8).
  • Slab sampled decode (#975): idle gap 1.70 -> 0.41 ms, main-thread CPU per sampled token 7.6 -> 5.6 ms, sampled decode 32.9 -> 34.9 tok/s (gate, 3 fresh processes per arm), byte-identical text.
  • TTM page pool: a Flash-Next reload after an exit is no longer refused for RAM the exited process left parked (0.4.3 known issue); three back-to-back loads now place identically (receipt: MemAvailable 119.7 -> 42.5 GiB after the first exit, pool estimate 74.0 GiB).
  • hipfire pull no longer downloads the 305.5 MB trunk overlay on a host that will not apply it (0.4.3 known issue; gfx1151 and no-GPU hosts still fetch it, and HIPFIRE_QWEN4_TRUNK_OVERLAY=1 forces it).
  • NPU: the coprocessor stage (#960; byte-identical to beta on Halo at 16K and 57K, on == emulate == auto, 16K prefill 2,364.35 vs beta 2,363.98 tok/s), a K = 5120 BF16 stage for 27B dense in-projections (kernel/program level, no serving route yet; GDN 22.9 TOPS busy / 20.7 wall, FA 23.4 / 21.2, contributor-measured), and live-HIP-identity verification (@amittell, #985).
  • Strix Halo DFlash draft GEMMs (#994, @fivetide, a9e98a70b5): the gfx1151 Qwen3-8B DFlash draft F16 GEMMs, and the legacy MQ4G256 v1 verify lm_head, run faster and are byte-identical to beta. Qwen3-8B DFlash at 1K/2K/8K goes 15.7/18.0/26.1 -> 24.5/26.0/30.5 tok/s (mq044b-pr994/). DFlash is still below AR on this model (AR 46.8/46.4/40.2 tok/s on the same prompts), so AR remains the faster choice there.
  • Native gfx1151 MQ6 railgun prepare/replay (#974, @HUSRCF, b53122c1a4): an opt-in API for native MQ6 prepare and replay. No Redline capture. The Halo oracle is exact (mq044b-pr974/).
  • Contributor-measured gfx1151 results in this release are marked in the CHANGELOG; the merge gates ran the Halo oracles named there.

Multi-GPU and serve

  • Dense TP generic mesh default (above, LM head on rank 0 only as of 3385df7ea5); Glimmer TP on the same executor (developer). Physical gfx1201 TP2, 481/481 argmax equal to the single-card run.
  • Flash-Next EP shared rank execution and owner-routed schedule (#999, @fivetide, f3ae992a17): EP2 and EP4 serve chains are identical to beta, native MTP and AR, 5 turns each (mq044b-pr999/). No default-path change.
  • --tp N penalties: repeat, presence and frequency penalties run on rank 0's logits (the penalty stage of the single-GPU host sampler, over generated tokens only) on Qwen dense TP and MoE EP; the tp>1 penalty refusal is gone (2c1d4157fb).
  • serve swap: configurable budgets, disk cap 8 GiB, private per-pid spill directory ($XDG_RUNTIME_DIR/hipfire/swap/<pid>, 0700, owned by the user, symlinks refused), a TTL for parked sessions (default 3600 s, 0 disables), a session cap (auto, max(64, 16 x slots)), snapshots stamped v3 with a model fingerprint (older spill files are rejected and removed at start), and swap-tier fields on /health, /stats and /metrics. serve -d warns when swap flags were not applied.
  • qwen_jinja off-ladder reasoning_effort aliases map to the nearest supported tier (high and max -> xhigh, med -> medium, minimal -> low) instead of dropping to medium (@CallumD, #986). On Strix Halo Flash-Next (greedy, 3,000-token cap) high is now byte-identical to xhigh (9,450 chars) rather than to medium (8,844).
  • DFlash2 session cache (#984, @fivetide, 9c7312722d): DFlash2 reuses the Qwen4 session cache with a resume guard. Opt-in DFlash2 only; see Known issues.

Fixes

  • MoE gate/up zero output for hidden sizes with K/256 % 4 != 0 (#956); ParoQuant dirs under VMM KV (#955); retired-DFlash grammar error class (#950); LLaMA .mq4 row-tail crash is a load refusal (#954, rank-2 tensors only); KernelGuard refuses multi-wave flag writers (#991); batched ParoQuant MoE prefill reads the routed rotation (opt-in, numeric, #972); grouped E8-SoA WMMA launchers cover every output token (#973); DDTree commit determinism (#971); Gemma 4 EAGLE stale row (#957); ModelSlot::load sizes flash partials to max_seq (#920, demo/lab callers); VMM KV reset panic after growth (#927); hipfire pull records verification proofs; developer scripts no longer kill other agents' serves and daemons (#953, #992, #993).
  • gfx1030 (RDNA2) with an MQ6/HFQ6 model (e.g. qwen3.5:9b-mq6) no longer crashes at pre-warm with no implementation for GemvMq6G256Prerotated (#478, 10a9626bd0): the MQ6/HFQ6 GEMV rows are generic wave32 kernels and are now admitted on RDNA1/RDNA2. Resolve-tested on CPU only, not on RDNA2 hardware, so the load-time "UNVALIDATED on RDNA2" warning stays.
  • DDTree combined with CASK/TriAttention eviction is refused at load (DDTree ... cannot be combined with CASK/TriAttention eviction) instead of panicking mid-run (#272, 10a9626bd0). CASK is deprecated since 0.4.0; drop the CASK sidecar or set ddtree_budget = 0.
  • Qwen DFlash: a greedy request demoted to AR by a non-neutral penalty now logs why (ffbfd4b794); routing and output are unchanged from v0.4.3.1. (qwen3.5:9b still carries presence_penalty 1.5 in its registry profile; send presence_penalty 0 or set generation.presence_penalty 0 to use DFlash for greedy requests on it.)
  • --kv bf16 DFlash serve (5301c41c81): every request failed at verify-graph capture (HipError 906; present in v0.4.3). Graph output is now byte-identical to HIPFIRE_VERIFY_GRAPH=0 (bf16-regate/).
  • hipfire setup: an explicit --hipcc counts in the runtime-only fallback, the amdrocm layout needs no --rocm-root, and a missing cargo is named (@amittell, #968).
  • Windows: install.ps1 double BOM (#979; every v0.4.1 to v0.4.3.1 release ignored -Tag/-Ref/-Branch/-Commit); release kernel packs accepted on native Windows; amdhip64.dll loaded so the driver's System32 copy cannot shadow the HIP SDK's; Windows UMA releases the model mapping like Linux; Windows CI smokes the packaged binaries with a System32-only PATH and checks PE imports (#995). The Windows MSVC CI job is now required (still no GPU; CI 085dd189b5).
  • The pre-compiled kernel probe honors HIPFIRE_HOME (#1001, @amittell): with HIPFIRE_HOME pointing at an install without a kernel dir, the loader used to pick up another install's objects and every load failed packaged object rejected (index mismatch). Probe order is unchanged; with HIPFIRE_HOME unset the home is still ~/.hipfire.
  • Generic DFlash on a LLaMA-arch target with non-batchable weights (MQ-V2 including --format mq4, ParoQ4G128, Q4K, F16/BF16) no longer panics the daemon on its first prompt (#1005, @fivetide; a regression from the LLaMA batched-prefill change in this cycle). The per-token fallback now captures each token's hidden row, as the verify fallback does; Llama-3.2-1B --format mq4 and Llama-2-7B PARO G128 text equals AR on gfx1151.
  • Developer: the Qwen3.5 AR hipGraph / Redline replay refuses int8, int8c and hfq4 KV caches, whose decode attention sizes its LDS from a seq_len the replay froze at capture (#1004, @fivetide). No KvMode builds these tiers, so production routes are unaffected.

Upgrade notes

  • vision_mode now defaults to auto. A present vision sidecar is loaded; a text load with a sidecar on disk pays ~128-142 MiB of idle VRAM and the automatic max_seq also reserves the tower's activation scratch for the largest admitted image (327 MiB at 2 MP). On the 24 GB 7900 XTX the automatic max_seq for qwen3.8-27b.mq4-xts + vision drops 223,323 -> 210,453 (card-bound, q8 KV); on a 32 GB R9700 it stays the model's 262,144. hipfire config set vision_mode off restores the text-only load; hipfire bench pins off.
  • Dense serve --tp N takes the mesh route. HIPFIRE_DENSE_TP_ROUTE=hand restores the hand route. MTP (speculation.mtp = on) and uneven splits still take the hand route.
  • Registry sampling: Qwen3.5 and Qwen3.6-35B-A3B tags now default to presence_penalty 0.0 (was 1.5). An explicit presence_penalty (request, CLI or generation.presence_penalty) is unchanged.
  • Penalties at --tp N are now applied instead of refused (Qwen dense TP and MoE EP).
  • Numeric changes: LLaMA-arch batched prefill (HIPFIRE_PREFILL_BATCHED=0 restores per-token), the hd64 decode tile, gfx1100 verify GEMV and attention (switches above), shared-expert INT8 prefill (HIPFIRE_QWEN4_SHARED_INT8=0), batched image prefill (HIPFIRE_VL_PREFILL=token), batched ParoQuant MoE prefill (opt-in). Greedy text can differ from 0.4.3.1 at near-ties.
  • Multi-slot swap: new defaults (TTL 3,600 s, auto session cap, 8 GiB disk cap, private swap directory); older swap spill files are dropped at start.
  • Flash-Next on the R9700: decode-share is on for admitted loads; HIPFIRE_QWEN4_DECODE_SHARE=off restores the previous placement.
  • qwen_jinja aliases: clients that sent high or max now get xhigh thinking.
  • Split-KV DFlash verifiers (gfx1201): the native-fp8 split verifier is on by default for the shape in Highlights; HIPFIRE_GFX12_FA2_SPLIT_VERIFY_FP8=0 turns it off. The Q8 KV split count HIPFIRE_GFX12_FA2_SPLIT_COUNT (up to 64) stays opt-in.
  • Kill switches for the byte-identical kernel routes: HIPFIRE_QWEN4_HC_UP_V2=0, HIPFIRE_QWEN4_HC_DOWN_FUSED=0, HIPFIRE_QWEN4_MOE_FRONT_FUSE=0 (gfx1151), HIPFIRE_MQ6_ROWS_SPLIT=0, HIPFIRE_GFX12_GATE_UP_TILE_LOOP=0, HIPFIRE_MOE_NINEPATH=0 (also turns off the existing d4 ninepath routes of the other routed dtypes) and HIPFIRE_MQ4V2_K512_R2=0, HIPFIRE_QWEN4_QSA_NORM_EPI=0, HIPFIRE_GFX1151_XLDS=0.

Known issues / trade-offs

  • Windows (native and WSL2) is untested on GPU hardware. 0.4.4 repairs the daemon's MSVC link (the native HSA code loader is gated to Linux, #942) and keeps the 0.4.3 Windows fixes, but no 0.4.4 build was run on a Windows GPU. WSL2 is no longer called the recommended route. The six Windows reports (#623/#669, #635, #639, #644, #655) have commit-cited fixes from 0.4.3 and no reporter retest yet. Volunteers can run scripts/windows-check.ps1 and file the windows-test-report.yml issue form. The install.ps1 double BOM (#979) is fixed in 0298052a63; the #979 MSVC link half does not reproduce.
  • Short prompts (16 to 32 tokens) prefill 2.5% to 3.3% slower on the generic TP route than on the hand route (pp16 479.6 vs 496.2 tok/s, pp32 859.0 vs 880.7; TP2 on 2x R9700, Qwen3.8-27B mq4-xts, --spec off, 3 fresh processes, median of run medians; tp-followup-runs/bench-tp2/). The cause is not yet identified. TP2 prefill at 512 and 2,048 tokens is +14.9% and +22.3%, and TP2 decode ties the hand route (54.89 vs 54.95 at 128 context, 54.19 vs 54.24 at 4K). HIPFIRE_DENSE_TP_ROUTE=hand restores the hand route.
  • DeepSeek-V4 decode through sealed MoE (#980) landed without hardware identity. Score layers always take the new path (no switch). The merge side had no DS4 model, so only compile, unit tests and static review ran; the author's gfx1151 ds4_rows_parity rows (9 of 9 digests identical to beta) are the only hardware evidence, and the native gfx1201/gfx942 MQ2-Lloyd backend and expert-parallel shards were not run on a card.
  • hd64 decode tile (#978) and LLaMA-arch batched prefill (#982) are numeric, not byte-identical. Greedy text can differ from 0.4.3.1 after a matching prefix; the measured cases read coherent (SmolLM2-360M run 30/32 identical, Llama-3.2-1B 8/9 identical, hd64 text identical at 712/3,445/8,196 tokens on gfx1151 and differing after 35 characters at 7,066). MQ4G256V2 (.mq4) LLaMA-arch files pass the new batched-prefill gate but still prefill per token.
  • The gfx1100 verify GEMV (#987) and verify attention (#989) are numeric and have evidence on one card only (the 7900 XTX); gfx1201 and gfx1151 are unchanged by registry diff, not by a run. Prose-32k tau drops 1.38 -> 1.32 with #987 (still net +14%). The #987 gate noted that the daemon prints one [q8x-verify] route active stderr line per module by default.
  • gfx1100 draft vocabulary (#998) is numeric for greedy MTP text. Only checked on the 27B at up to 39K context; the 170 MiB lm_head copy is paid at load. Tau is unchanged on code-1K and code-32K and 1.38 -> 1.43 on prose-32K. Kill switch HIPFIRE_GFX1100_DRAFT_VOCAB=0.
  • DFlash2 resume path (#984) is untested. It is opt-in, and no Flash-Next DFlash2 draft exists yet, so the resume guard has no hardware run behind it.
  • DFlash on Strix Halo is still not a win everywhere. Qwen3-8B DFlash stays below AR at tau 0.2-0.8 (the DFlash cycle still costs about 170 ms at 2K and 280 ms at 8K against a 22-26 ms AR token); 9B DFlash output differs from AR with or without the XLDS and VerifyAttn changes (pre-existing on gfx1151); llama-family DFlash on gfx1100 diverges from AR at a near-tie (margin not measured).
  • Decode-share costs the second token. After a 16K-token prompt the second token takes 78.7 ms instead of 22.7 ms (R9700, AR, decode-share off for the comparison) because the restore of the lent expert VRAM is paid at the next decode; TTFT is unchanged (2,483.7 vs 2,469.9 ms). HIPFIRE_QWEN4_DECODE_SHARE_SYNC=1 restores at prompt end instead. Pp16384 is within 0.1% of decode-share off (6,677.5 vs 6,686.7 tok/s). A back-to-back loop of one-token MTP requests (hipfire bench --ttft with MTP on) pays about 8 ms more TTFT per request than the first 0.4.4 RC, because every request now restores and the next one rebuilds the 1,477 MiB weight shadows; with --spec off, or with any request that decodes past its first token, nothing changes.
  • Streamed vision tower: Qwen3.5/3.8-VL only. EmbeddingGemma 2, qwen35-embed, LFM2-VL and dots towers stay fully resident whenever vision is enabled (#988).
  • Opt-in DFlash railgun window (HIPFIRE_DFLASH_RAILGUN_WINDOW=1, gfx1201, 27B) is default off because short-turn serve decodes 3% to 6% slower with it on (each new shape pays planning on the window thread); the long-turn gain is +3.7% prose / +2.1% code.
  • Batched ParoQuant MoE prefill (HIPFIRE_PARO_BATCHED=1) is opt-in and NUMERIC.
  • Split-KV DFlash verifiers (#921, #922) are NUMERIC, not byte-identical (coherent output, measured on one R9700). The fp8 split module is registered in the gfx1201 kernel inventory, so nothing compiles after ready.
  • The R9700 default 16,384-row chunk still costs a little decode against HIPFIRE_PREFILL_CHUNK_ROWS=4096: --spec auto is 2.7% slower on prose (49.7 vs 51.1 tok/s) and 1.9% to 3.7% slower on code (80.6 vs 82.2, 78.6 vs 81.6), synthetic tg128 is unchanged at short context (36.0 vs 36.0), and pp16384 is 25.7% faster (6,648 vs 5,288 tok/s); about a third of the 0.4.3 figure (−9.8% prose, −10.3% code). Set HIPFIRE_PREFILL_CHUNK_ROWS=4096 only for decode-bound workloads with short prompts. The hot set itself costs only pp512 (−4.2%) and synthetic tg128 (−4.8%).

Not in this release

  • Open community PRs that were not ready: #870 (@nwoolmer, DeepSeek4 expert paging, draft; does not compile), #939 and #940 (@HUSRCF, tiered KV prototype; gfx1100 TP2 lm_head sharding), #959 (@fivetide, DSpark verify drafter context; tau drop unexplained), #970 (@HUSRCF, Flash-Next vision sidecar export and image serving; #938).
  • Vision for Flash-Next (#938, #970), streaming the remaining vision towers (#988), tiered KV (#925), and the design issues #861 to #868 beyond the landed slices.
  • A phase-aware R9700 hot-set budget as a separate mechanism (decode-share is the 0.4.4 answer for gfx1201 Flash-Next), the pruned coder model in the registry, the XTX slow-link redesign, Flash-Next row-exact wide verify and the derived-rate MTP depth chooser (default off): not claimed.
  • Default-on for the railgun DFlash window and for the NPU K = 5120 stage (kernel level only).
  • Windows GPU serving as a supported platform.

Contributors

Thanks to the contributors whose work landed in 0.4.4. Authorship was preserved on every landed commit.

  • @fivetide (Björn Bösel): merged on GitHub: #948 (dispatch family conventions), #949 (G5 fixture), #950, #951, #952, #953, #954, #955, #956, #957, #958, #961, #962, #964, #971, #972, #973, #975, #976, #977, #978, #980, #981, #982, #983, #984, #990, #991, #992, #993, #994, #996, #997, #999, #1004, #1005; landed with authorship preserved and closed on GitHub: #960 (NPU coprocessor stage), #963 (railgun conditional nodes), #965 (NPU K = 5120 stage).
  • @Ben1078: gfx1100 int8-activation small-batch verify GEMV (#987) and WMMA split-K verify attention (#989), plus eval_hipfire --scored-max-batch; the two PRs behind #946 and #947; gfx1100 MTP draft vocabulary subset (#998).
  • @HUSRCF: CI repair, Linux-only HSA loading and TLS advisories (#942); gfx1151 Flash-Next prefill/decode optimizations (#941); native gfx1151 MQ6 railgun prepare/replay API (#974).
  • @ghazni101: multi-slot swap budgets, private spill directories and bounded session lifecycle (#945; closes #943, #944).
  • @amittell (Alex Mittell): NPU live HIP identity and serving transport verification (#985) and NPU transport adapter tests; hipfire setup fixes (#968) and the PRO 495 / 8065S hardware record (#969), cherry-picked from master-targeted PRs with authorship kept; the HIPFIRE_HOME kernel probe fix (#1001), the local-compile recipe for hosts outside the pack's ROCm range (#1000) and the measured DeepSeek V4 kv_cache=f16 note for gfx1151 (#1003).
  • @CallumD: qwen_jinja off-ladder reasoning_effort normalization (#986).
  • @paleluna82-del: ModelSlot::load flash-partial sizing (#920) and the gfx1201 split-KV FA2 DFlash verifiers (#921, #922).
  • Issue reports acted on: @icocoon (#979, the MSVC link failure), @ghazni101 (#943, #944) and @rakaarwaky (#478, gfx1030 MQ6 pre-warm crash).
  • Co-authored-by trailers in the range: Claude (Anthropic assistant, 21 trailers) and multica-agent (3 trailers). Both are tooling, not accounts.
  • Draft PRs that did not make this release stay open: #870 (@nwoolmer), #939, #940, #970 (@HUSRCF), #959 (@fivetide).

Receipts

Internal evidence for the numbers above (workspace release-0.4.3):

  • Merge log: MERGE-QUEUE-LOG.md (every landing named here, with gates).
  • Decode-share: decode-share/LAND-S2-S8.md, decode-share/runs/; 16K second-token and shadow numbers are merge-log lines 9cb35862c4 and a0e571597f. The release-code timing (on vs off, 3 rounds) and the MTP==AR fix evidence are rc-044b/timing/RESULT.md and rc-044b/fn/RESULT.md; gates for the release head are in RC-GATES-0.4.4-RESULT.md.
  • Vision: vision/ (batched prefill and TTFT runs), vision-tower-streamed-run/ and vision-tower-streamed-regate/ (resident vs streamed embedding md5s, gfx1201).
  • gfx1100 verify: mq044-pr989/ (gate REPORT and KLD), pr987/REPORT.md and pr987/evidence/ (#987 gate: KLD pins, ABBA, serve battery), PR #987 and #989 bodies (author tables).
  • Verify bandwidth kernels: verify-bw/PLAN.md, mq4rs-kernels-044-evidence/RESULT.md, fn-10k/ (shared-expert INT8, fn-10k/shared-i8-receipts/), fwht4-kv/LANDING.md.
  • Halo: halo-ttm-044/gates.txt, pull-overlay-044/, mq044-pr977/, mqo-pr958/, mqo-pr961/, mqo-pr965/, mq044-pr980/, mq044-pr982/, mq044-pr989/.
  • Dense TP default (#927, ef3d42edb0): generic-tp-hiptrx-runs/ (final-perf-tp{1,2,4}/ bench, regate-hwtest.log parity, regate-serve-tp{2,4}/ battery and chain, regate-bench-*); the commit's docs/multi-gpu.md carries the same table.

Install

curl -fsSL https://hipfire.dev/install.sh | bash

To pin this release and its prebuilt kernel packs:

curl -fsSL https://hipfire.dev/install.sh | bash -s -- --tag v0.4.4

Full notes: CHANGELOG.md.

Contributors

CallumD, amittell, and 8 other contributors
Assets 23
Loading