vllm-sly-radiance 1.3.2

Fixed

• Lazy GDN without speculative decoding is deterministic again. In 1.3.1, RADIANCE_GDN_LAZY=1
  without a drafter gave different logprobs for the same cold request from the second request on
  (eager: bit-identical). Cause: the pre-forward materialize (mode 0) zeroed the stash headers at
  block-table column dst_col + 1. Without a drafter there is no stash block, so that column lies past
  the request's blocks and can hold a stale id of a live block; the zeroing hit live GDN state.
  Proven on the GPU window: masking that column made lazy bit-identical to eager.
  • radiance_gdn_lazy.temporal_enabled(): the lazy temporal path (skip of vLLM's temporal copies +
    materialize) runs only when the mamba groups own a stash block (num_speculative_blocks > 0).
    Without a drafter vLLM's exact Triton copies run, as in eager.
  • libr4d gdn_lazy_materialize (rx10 patch): new mode bit LZ_MODE_STASH. The kernel reads or
    zeroes column +1 only when the bit is set, so a wrong caller cannot hit a foreign block.
  • CPU tests: sly/tests/test_gdn_lazy_cpu.py.

Measured

• GPU window 2026-10-13, image 1.3.2-rocm10.1, production argv, bf16 SSM state, fp8 KV:
  • No spec, one cold 30,926-token chat request, 2 runs per arm: eager run 1 = run 2 (bit-identical);
    lazy runs 1 and 2 = eager (tokens and logprobs bit-identical, 148 tokens). 1.3.1 lazy: run 2
    diverged at token 0.
  • Spec (DFlash2, 7 tokens), lazy, turnbench EXACT, 3 sessions x 7 turns up to 110k tokens:
    HEALTH PASS, every cold turn exact, cached vs cold drift 16 of 18 GPU-path turns (eager 1.3.1 on
    the same bench: 18 of 18). The cached-path drift is the known prefix-cache drift of eager too,
    not a lazy defect.
• The acceptance night gate has not run on 1.3.2; this release is cut by hand (release.yml).

Commits since v1.2.1

-21c3fa6 fix(accept): request the served model name of docker-vllm7.service
-7bdb660 docs(sly): note the 1.3.2 LZ_MODE_STASH guard on r4d_extras_rx10.patch
-32b1110 docs(changelog): 1.3.2 GPU-window measurements
-7199c96 fix(lazy-gdn): no stashless materialize — lazy no-spec deterministic again (1.3.2)
-8d56012 Update renovatebot/github-action action to v46.3.8 (#137)
-50e090a fix(lazy-gdn): guard the stash column on speculative decoding
-8d0226a feat(lazy-gdn): bf16-state variants for lazy_update + lazy_materialize
-4da3dbb fix(lazy-gdn): dict-lookup for 0.30 mamba_state_copy_funcs + relax turnbench verdict
-06289f7 Wire up the lazy GDN snapshots for vLLM 0.30 (RADIANCE_GDN_LAZY)
-609f20c docs: add FASTBOOT.md — aiter-bake + compile-cache boot runbook (#134)
-8556b72 ci: sly/tests/*.py is offline-only, not in the final image — no bump needed
-17734e8 docs(wsl2): use the librocdxg 1.2.2 bridge with the 1.x images
-b1a8c81 Point the GPTQ checkpoint references at slybase/Swift-1.5-Qwen3.8-27B-heretic-MXFP4-GPTQ

---

Image: ghcr.io/slybase/vllm-sly-radiance:1.3.2-rocm10.1 (published by the build workflow for this tag).
acceptance gate passed: vllm-sly-radiance:1.3.2-rocm10.1, profile vllm7-mxfp4, mode fast

cc @slydlake

vllm-sly-radiance 1.2.1

Fixed

• The KV-cache offload modes (KVCACHE=ram|disk) work again on vLLM 0.30, i.e. on every image since
  0.4.0. The six "always" patches were written against vLLM 0.27.1; four of them stopped part-way on
  the 0.30 tree, which left an offload tier that stored but never served (0 external hits). vLLM 0.30
  rebuilt the offload scheduler and carries part of the set itself: the load now starts at the local
  boundary (mixed-hit lookback + assertion), Mamba states are stored per retained checkpoint and
  --prefix-cache-retention-interval (default 0) keeps only the replay boundary and shared-prefix
  junctions (mamba-stride), store reachability is judged on the absolute grid (swa-align). Those hunks now
  report SKIP through the new _patchlib.skip_if_upstream; eagle-groups got a second anchor shape;
  reconcile-reask, touch-all and align-last-block apply unchanged; vLLM 0.31 (the pin on main since
  1.1.0: multi-line annotator call, no _touch, _make_boundary_key with request context) is carried
  too. The 0.27.1 shapes still apply, and ci/patch_dryrun.sh now runs the ram-mode list against the
  pinned vLLM so the next bump fails ci instead of a boot WARNING.
• Tier hits landed two turns back. With the eagle drafter annotated, the scheduler commits the Mamba
  boundary state one block before the aligned prompt end, while the drafter group's retained tail ended
  at the aligned end, so the next turn's lookup found no complete window there and fell back to the
  previous turn's junction (37-56k tokens recomputed per tier turn). patch_swa_eagle_replay_tail.py
  keeps one more drafter block per reachable boundary (RADIANCE_SWA_EAGLE_REPLAY_TAIL=1, a superset of
  the stock mask); the tier hit now lands where a GPU hit lands. Without the eagle annotation (upstream's
  default for this model) the hit lands 4-5 turns back, so the annotation stays on.

Added

• kv-cache/patch_offload_boundary_trace.py: RADIANCE_OFFLOAD_BOUNDARY_TRACE=1 logs every Mamba
  boundary-state hand-off and its fate (stored / dropped and why). Silent otherwise.
• TIERBENCH_API_KEY for tierbench.py, equivbench.py, turnbench.py against a server started with
  VLLM_API_KEY.
• kv-cache/README.md, "vLLM 0.30 and later": what each patch does on the new tree and why.

Measured

• turnbench, 3 sessions x 7 turns, step 18k (398k tokens against a 331k GPU pool), KVCACHE=ram 16 GiB,
  production launch + vision, second start: tier hits C6/A7/B7/C7 at 94,080 / 112,896 / 117,376 / 113,792
  tokens (one block before the previous prompt end) against 77,952 / 94,976 / 100,352 / 77,952; 20-21k
  tokens recomputed per tier turn against 37-56k; tier turn wall 17-20 s against 29-41 s. Repeated on image
  1.1.1 (vLLM 0.31.0): tier hits at 94,160 / 112,640 / 117,920 / 114,400, 19-21k recomputed, 16.7-19.8 s,
  cold twins first divergent token 63 / 19 / 2 / none, max |dlogprob| 0.13-0.18. HEALTH pass on both images. Cold twins of the four tier turns (bf16 SSM cache): first divergent token 52 / none / 3 /
  none, max |dlogprob| 0.24 / 0.33 / 0.06 / 0.16 -- the same positions as without the patch and the same
  kind of drift as the GPU-hit turns of the run, so the tier adds none of its own; bit-exactness is not
  shown on this build. Single-stream decode with the tier on is unchanged. Image binaries unchanged:
  kv-cache/ rides the /patches mount and is applied at container start.

Commits since v1.2.0

-caf368b night gate green: Measured numbers and baseline for 1.2.1
-4c66809 Kv cache/vllm 0.31 (#131)
-ba41eb2 Update dependency ruff to v0.17.0 (#132)
-f2d0751 ci: register ubuntu-26.04 in actionlint label list
-dbfea79 Update GitHub Actions

---

Image: ghcr.io/slybase/vllm-sly-radiance:1.2.1-rocm10.1 (published by the build workflow for this tag).
released manually by github-actions[bot] without a green acceptance gate: night gate green: https://github.com/SlyBase/vllm-sly-radiance/actions/runs/38118524404

cc @slydlake

vllm-sly-radiance 1.2.0

Added

• Sharded MXFP4 checkpoints (low-RAM loading) + local source reuse. fp8_mtp.py
  (the one-time build behind setup-mxfp4.sh) now reads a source checkpoint in either
  layout — AMD's single model.safetensors or the standard HF sharded layout
  (model.safetensors.index.json + model-0000X-of-0000Y.safetensors) — and writes sharded
  output by default (5 GiB shards; RADIANCE_SHARD_GIB to resize, RADIANCE_SINGLE_FILE=1 to
  restore the single-file output). A sharded checkpoint is what unblocks 16 GiB hosts: vLLM loads
  it shard-by-shard (one shard mapped at a time), so the peak host RAM for the weight load is the
  largest shard instead of the whole ~19 GiB model. The serve path (serve-mxfp4.sh) is already
  layout-agnostic — it hands vLLM the checkpoint directory — so a sharded build serves with no
  further change. Every copied body tensor is byte-identical to the source (verified), and the
  fp8 MTP head is unchanged (same F8_E4M3 dtype, per-channel scale, ~2 % rel err).
• SRC_LOCAL in setup-mxfp4.sh: reuse an already-downloaded source snapshot. Point
  SRC_LOCAL at a directory holding model.safetensors or model.safetensors.index.json (any
  layout) and the ~19 GiB pull is skipped — machines that already have the weights via another
  workflow no longer re-download them. The container mount now bind-mounts the source at /src
  (was: the HF cache path), so a local directory anywhere on the host works, not just a snapshot
  under ~/.cache/huggingface.

Changed

• setup-mxfp4.sh step 3 reports "source checkpoint" (was: "AMD's MXFP4 release"); the
  reclaim-~19 GiB hint now only prints when a download actually happened (not for SRC_LOCAL).

Measured: no kernel change, so decode/prefill/KV pool are unaffected. Engine memory with a
16 GiB container cap, fresh boot per run, 4 runs: file-backed RSS of the engine 12.3-12.7 GiB
(single file) vs 5.0-5.3 GiB (5 GiB shards), anonymous RSS 2.2 GiB in both, no OOM kills, load
time unchanged (about 210-250 s to ready). The container cgroup still reaches its cap, and under
real host pressure the single file swapped heavily; see docs/TECHNICAL.md. The change moves the one-time
build and the serve-time weight load: a 16 GiB host loads a sharded checkpoint with peak RAM at one
5 GiB shard instead of the full model (minutes vs. hours of swapping); a SRC_LOCAL snapshot skips
the ~19 GiB pull. Copy fidelity is byte-exact (verified against a synthetic single-file and
sharded source); the fp8 MTP head is numerically unchanged (~2 % rel err, MXFP4 was ~11.6 %).

Commits since v1.1.1

-8cc12e0 changelog: add measured engine memory numbers to 1.2.0
-ce23c74 docs: correct checkpoint name and host-run labels in memory measurements
-c015668 docs: measured effect of sharded checkpoints on engine memory
-f71f5a9 1.2.0: sharded MXFP4 checkpoints (low-RAM load) + SRC_LOCAL source reuse (#106)

---

Image: ghcr.io/slybase/vllm-sly-radiance:1.2.0-rocm10.1 (published by the build workflow for this tag).
released manually by kore-bot without a green acceptance gate: Release 1.2.0 with measured memory results (PR #125, closes #106); no accept gate run for this change

cc @slydlake

vllm-sly-radiance 1.1.1

Fixed

• Split-K decode GEMM could read stale partials. The W4A8 decode kernel
  (radiance_mxfp4_fp8_gemm_decode, every MXFP4 linear at M <= RADIANCE_MXFP4_DECODE_MAX_M and, since 1.0,
  the wide band up to RADIANCE_MXFP4_WIDE_MAX_M) splits K over 2 or 4 blocks on narrow shapes (o_proj,
  out_proj, down, in_proj_ba) and the last block to arrive sums the partials. Only thread 0 fenced before the
  counter bump, and __syncthreads is workgroup scope, so the other threads' partial stores were not
  guaranteed visible to the reducing block on another CU. Now every thread fences, thread 0 bumps the
  counter with an agent-scope acq_rel atomic, and the reducing block fences again before it reads
  (the pattern libr4d's tiled W4A8 split tail documents for the same bug). Production runs this kernel
  on every decode/verify step. The same fix is in the opt-in AutoRound, EsCHA (both kernels) and the
  second ParoQuant decode kernel (the first already fenced on every thread) and in ggz14's root copy of
  the MXFP4 kernel. Written 2026-10-04 as 0.7.1 and never released; ported onto 1.1.0 unchanged.
• Lazy GDN: a prefill step after a prefix hit could run from a stale state (opt-in, only with
  RADIANCE_GDN_LAZY=1; production does not set it). r4d_gdn_lazy_materialize in sly/r4d/r4d_extras_rx10.patch
  returned without migrating the state when the stash column bt[base_col + 1] was null, even with no
  candidates to replay. That column is null when a prefill step spans more than one mamba block (block 896,
  --max-num-batched-tokens 2048) on a request's first step after a prefix-cache hit, so the step started from
  whatever the new running block held. The stash is now required only when count > 0; with none the kernel
  copies base -> dst. Decode (the lazy update kernel) is unchanged. Ported from libr4d-rx13 rx17 as published in
  zzpanic/qwen3.6-vllm-gfx1201-launchers 6147726 (validated there: offload-tier resume bit-identical, deep
  multi-turn gate 48/48 against 7/48); their stale-stash counters are not part of this fork's kernel.

Added

• sly/mxfp4/check_decode_splitk.py: alternates two inputs launch after launch and requires every
  output to be bit-identical to its golden, per production shape x M x split, including the wide band.

Measured

• Pending: check_decode_splitk.py on 1.1.0 (control) and 1.1.1, decode step A/B, GSM8K 200;
  lazy GDN: a prefix-hit multi-turn run with RADIANCE_GDN_LAZY=1 against eager GDN.

Commits since v1.0.0

-0719b42 night gate green: Measured numbers and baseline for 1.1.1
-24de2f8 accept-night: cron off the full hour (17 2,3)
-1cf0fe9 ci: renovate needs a checkout for docker-cmd-file (#123)
-599ed65 ci: stale-issues workflow (warn after 7 days, close after 10) (#122)
-e7aff63 1.1.1: lazy GDN materialize no longer needs a stash block when nothing is replayed (rx17)
-c0103e2 accept-night: gate and release an unreleased main
-3b77e84 release: mention @slydlake in the GitHub release body (RELEASE_NOTIFY)
-b37a2e8 1.1.1: split-K decode reduction fences on every thread
-9415750 night gate: baseline calibrated on the current release (no merge)
-fa21510 accept-night: run_now (owner-approved window now, owner accept-manual) and a mode input; health default vLLM
-d81b882 accept.yml: no automatic main-build gate while the automated maintenance is on
-8afd203 check_constraints: hand pins (flydsl) are not stale; --update bootstraps setuptools inside vLLM's range; renovate caps for vLLM 0.31
-2e381b1 radiance_attn_decode: use_2d_kernel wrapper forwards aiter 0.1.23's backend argument
-052439e resolve_stack: derive aiter's runtime pins (flydsl) from aiter's requirements.txt
-5d25ae0 1.1.0: install flydsl 0.3.4.1 (aiter 0.1.23 runtime dep), assert it in the stack check
-d2ac80e 1.1.0: do not strip AMD's torchvision extensions (strip makes _C.so unloadable)
-60643da accept-night: calibrate the current release when the baseline is incomplete, never gate against it
-c636a1c Automated maintenance: stack PRs, resolve_stack, night GPU gate, Hermes brief
-52c8eef 1.1.0: tools/use_existing_torch.py, libr4d a3e4833
-9778b6f 1.1.0: vLLM 0.31.0 on torch 2.13 / triton 3.8.0 / torchvision 0.28.0 / aiter 0.1.23
-964e6e9 chore(deps): update renovatebot/github-action action to v46.3.7
-aa458de Update README.md

---

Image: ghcr.io/slybase/vllm-sly-radiance:1.1.1-rocm10.1 (published by the build workflow for this tag).
released manually by github-actions[bot] without a green acceptance gate: night gate green: https://github.com/SlyBase/vllm-sly-radiance/actions/runs/37747756137

cc @slydlake

vllm-sly-radiance 1.0.0

Longer prefills are faster, the KV pool is 11.5 % larger, concurrent decode is ~5 % faster, and the image moves to
ROCm 10.1 with torch 2.12. All measured on one R9700 at 210 W with the production launch, A/B/A in one GPU window,
second start.

Added

• --attention-backend R4D_HYBRID: libr4d's paged prefill attention for prefill runs of 512 tokens or more
  (RADIANCE_R4D_PREFILL_MIN_Q), AITER unified attention for everything else, so decode, graphs and KV pool stay as
  they were. Prefill 2k / 8k / 32k / 64k +2 / +3 / +13 / +23 % (2375 / 2410 / 1976 / 1555 -> 2428 / 2506 / 2229 /
  1919 tok/s). Plain --attention-backend R4D is not recommended: decode 60.9 instead of 34.9 ms per step.
• RADIANCE_ADAPTIVE_WIDTH=perseq: per-request verify width (port of the Radiance engine's adaptive_k) on
  varlen FULL decode graphs. 8 concurrent requests with arrivals +4.7 % (401.7 / 403.5 -> 421.4 tok/s); one stream is
  untouched (bit-identical greedy output). Why ggz14's dynamic width lost and this one does not: TECHNICAL.md.
• RADIANCE_MXFP4_WIDE_MAX_M=192: the split-K decode GEMM kernel for M 129..192 instead of the padded 256-row tile.
  Time to first token of a 160-token prompt -13.8 % (109.6 -> 94.4 ms), c8 with arrivals +1.7 %. 192 is the edge:
  the folded tile is faster again at 256.
• Test packages of this work in sly/tests/lessons/.

Changed

• KV pool 391,193 -> 436,097 tokens (+11.5 %) at 262k context: --gpu-memory-utilization 0.98 with
  --compilation-config.cudagraph_mode FULL_DECODE_ONLY (was 0.9655 with FULL_AND_PIECEWISE). CUDA graph memory 1.24 ->
  0.22 GiB, peak activation 1.69 -> 0.65 GiB. Decode step, c8 and TTFT unchanged; 64k prefill runs without OOM. Util
  0.98 alone gives 404,947.
• ROCm 10.1 (rocm/dev-ubuntu-24.04:10.1.0-full, HIP 7.16) and AMD's torch 2.12.0+rocm10.1.0 (AMD ships no 2.11
  for 10.1), torchvision 0.27.0; triton 3.6.0, aiter 0.1.22.post1, vLLM 0.30.0 unchanged. Image tag
  1.0.0-rocm10.1. The torch >= 2.12 CPU spin that held torch back is fixed in ROCm 10.1. Performance is neutral (step
  35.35 / 35.59 / 35.13 ms, prefill within 1-2 %). Fresh compile caches on first start; to stay on 10.0 see TECHNICAL.md.
• README reference launch, options table and docs layout updated; the KV pool reference in AGENTS.md is now 436,906 (the full 1.0 launch, warm compile cache).
• tests-lessons/ moved to sly/tests/lessons/; the working notes are folded into TECHNICAL.md.

Removed

• NOTES-A/D/F/H.md (content is in docs/TECHNICAL.md, section 1.0.0).

Credits

• The adaptive verify width and the band findings (wide decode tile) come from the Radiance engine fork
  (StillDeadcode/radiance, our fork slydlake/radiance); the prefill
  kernel is libr4d.

Measured

• Prefill (tok/s, control / R4D_HYBRID / control): 2k 2375 / 2428 / 2380, 8k 2410 / 2506 / 2450, 32k 1976 / 2229 / 1970,
  64k 1555 / 1919 / 1554. Greedy step 35.12 / 35.05 / 35.14 ms, c8 arrivals 395.7 / 391.8 / 395.9 tok/s, short-prompt
  greedy text identical, prompt NLL delta -0.0007.
• KV pool 436,097 (util 0.98 + FULL_DECODE_ONLY); greedy 34.89 ms (control 34.88 / 36.68), c8 401.9 (control 401.7 /
  414.4), TTFT 72 / 256 / 904 tokens unchanged, 64k prefill 1572.
• Wide band (edge 256 in the server run): TTFT 160 tokens 109.6 / 109.4 -> 94.4 ms, other prompts within 0.6 % except 256
  tokens (+9.7 %, hence the edge 192), c8 405.4 / 406.2 -> 412.8, greedy identical for prompts <= 160 tokens.
• Adaptive width: c8 arrivals off 401.7, uniform 408.8, perseq 421.4, off2 403.5; c4 steady unchanged; 30 % of decisions
  shortened, 4.15 verify rows saved per step; 8-way concurrent greedy diverges at near-ties (batch-shape numerics).
• ROCm 10.1: greedy 35.35 / 35.59 / 35.13 ms, prefill 2k 2357 / 2344 / 2385, 8k 2432 / 2382 / 2401 tok/s, KV 385,934 on
  the second start of the test run (fresh cache dirs), greedy text 4 of 8 identical (A/A: 8 of 8).
• Release candidate as shipped (ROCm 10.1 image, all 1.0 flags) against 0.7.0, same window, 210 W, base1 / rc / base2:
  KV pool 391,193 / 430,433 / 391,193 (+10 %; the varlen graphs of perseq and the 10.1 runtime take ~6k of the
  436k from the util/graph change alone); greedy 34.84 / 35.04 / 35.06 ms; TTFT 160 tokens 103.8 / 93.1 / 106.3 ms;
  prefill 8k / 32k / 64k 2666 / 2737 / 2556, 2128 / 2376 / 2083, 1645 / 2025 / 1627 tok/s (+5 / +13 / +24 %);
  c8 arrivals 418.3 / 411.0 / 407.9 (neutral in the combination; the isolated +4.7 % of perseq does not show here);
  multi-turn follow-up with a prefix hit 3.3–3.6 s on both.
• GSM8K 200 (cot zero-shot, greedy, flexible-extract): 0.84 (0.7.0 in the same window: 0.83).
• BetterBench full at 300 W (1.0.0 then 0.7.0, same window): weighted decode 135.8 / 137.4 tok/s (within noise);
  prefill 1.5k / 6k / 12k / 24k / 47k 3132 / 3136 / 3105 / 2949 / 2635 against 3108 / 3049 / 2946 / 2648 / 2164 tok/s
  (+1 / +3 / +5 / +11 / +22 %); concurrency 1 / 2 / 4 / 8 / 16 120 / 213 / 338 / 461 / 466 against 121 / 219 / 338 /
  446 / 449 tok/s; TTFT 47k 17.9 s against 21.7 s; KV pool 436,906 against 391,193 tokens.
• Not adopted (docs/NOT-ADOPTED.md): gated gate_up + SwiGLU fold, int4 lm_head LEAN configs, fp16 SSM state, draft refill
  after a prefix hit, plain R4D backend.

Commits since v0.4.5

-ccb3401 1.0.0: reference run in BENCHMARKS.md, KV reference 436,906, transformers 5.18.0 in the docs
-b52a656 1.0.0: 300 W BetterBench reference run (1.0.0 vs 0.7.0) in README and CHANGELOG
-138e9f7 1.0.0: combined release-candidate run and GSM8K in the changelog, KV reference 430,433
-a414f57 Release 1.0.0: R4D_HYBRID, util 0.98 + FULL_DECODE_ONLY, wide band 192, adaptive width perseq, ROCm 10.1
-9138369 Revert "Merge branch 'worktree-agent-a9c4ebf4753bee325' into release/1.0"
-97a87aa Revert "Merge branch 'worktree-agent-a24b76d422b9e2726' into release/1.0"
-2a23d0d R4D_HYBRID attention backend: libr4d prefill for long runs, AITER for the rest
-b4bd870 ROCm 10.1 base, torch 2.12.0+rocm10.1.0, torchvision 0.27.0 (candidate for 0.8.0)
-7984bc9 int4 lm_head: RADIANCE_LMHEAD_INT4_LEAN knob (split-K DEQ1/UNPACK1 configs for M>16), gfx1201 spill report, C test package
-2a751e3 Adaptive verify width for the DFlash2 drafter, graph-safe (RADIANCE_ADAPTIVE_WIDTH=0|uniform|perseq)
-2056ccd RADIANCE_MXFP4_GATED_FOLD: gate_up GEMM + SwiGLU epilogue for the W4A8 decode band (default off)
-8ae654b mxfp4: wide decode band for M 129..256 behind RADIANCE_MXFP4_WIDE_MAX_M (default off), band map, kernel bench + A/B test script
-9cdd3aa 0.7.0: AMD's torch wheel instead of a source build; hold aiter at 0.1.22.post1
-2171ec5 0.6.2: torch wheel from ghcr.io, triton from PyPI
-400737b 0.6.0: MXFP6-PARO (W6A8) serving — port ggz14's kernel onto this fork's diverged base (#97)
-e29cf25 0.5.0: opt-in KV-cache offload + MXFP6 build tooling (#96)
-f455956 chore: bump VERSION to 0.4.7 with changelog for the build changes
-038de12 build: kernels stage, parallel quant compiles, ccache for the stack build
-fb53849 ci: append commit list to GitHub release notes
-b9e613e Fix #90: ship radiance_lmhead_int2.py into the image + fix stale README quickstart (#92)
-9d8c19d chore(deps): update dependency rocm/aiter to v0.1.24 (#55)
-8829523 docs: KFD launch-flags note — don't reboot when a sibling container already uses the GPU
-e2ff42a docs: run on Windows 11 through WSL2
-5ca3e5c chore(deps): update dependency ruff to v0.16.10
-7d2304a fix(ci): aiter drift justification may live in a PR comment (renovate bodies are templates) (#85)
-9621078 fix(ci): aiter drift graded by distance; 2+ releases above sanctioned stand needs a PR-body justification (#84)
-262e1d4 fix(ci): aiter contract gate -- AST-check the pinned aiter tag before merge (#83)
-3d49c69 ci: add trio compatibility gate + PR build check (#82)
-6d1971f chore(deps): update ubuntu:24.04 docker digest to a853f94
-f766a2e chore(deps): update dependency ruff to v0.16.10
-b6fec5b constraints: repair the resolution of this update
-4e0ce70 chore(deps): update dependency transformers to v5.18.0

---

Image: ghcr.io/slybase/vllm-sly-radiance:1.0.0-rocm10.1 (published by the build workflow for this tag).
released manually by slydlake without a green acceptance gate: 1.0.0 measured by hand 2026-10-07 instead of the accept gate: 300 W BetterBench full 1.0.0 vs 0.7.0 same window (prefill 47k +22 %, c8 +3 %, KV 436,906), GSM8K 200 0.84 vs 0.83; gate baseline profile still on the Quark checkpoint

vllm-sly-radiance 0.4.5

Added

• Self-service HSA/KFD diagnostics (issue #73): when GPU enumeration fails while rocm-smi
  still sees the card, the startup precheck now prints the host-side facts that triage it
  without a GPU window — host kernel, /dev/kfd access, container pids limit — plus the
  recovery order (reboot first, raise the pids limit, then capture strace/dmesg evidence
  before rebooting). docs and radiance_preamble only; the default GPU path is unchanged.

---

Image: ghcr.io/slybase/vllm-sly-radiance:0.4.5-rocm10.0 (published by the build workflow for this tag).
acceptance gate passed: vllm-sly-radiance:0.4.5-rocm10.0, profile vllm7-mxfp4, mode fast

vllm-sly-radiance 0.4.4

Added

• Greedy int2 lm_head (RADIANCE_LMHEAD_INT2=1, sly/mxfp4/radiance_lmhead_int2.py +
  sly/patch_lmhead_int2.py): two-stage vocabulary head — a 2-bit coarse pass over all 151,936 rows
  and a bf16 re-rank of the top-16 — with roughly 7.5× less lm_head weight traffic per decode step
  than the int4 head (≈1.26 GiB saved at [151936, 5120]). Greedy-only: it declines under sampling
  and whenever RADIANCE_LMHEAD_INT4 is active, so the default path is byte-identical.
  Pre-flight: sly/check_lmhead_int2.py (real-weight coverage gate) and 11 CPU unit tests.
• Drafter bf16 pre-expansion (RADIANCE_DFLASH_BF16=1, default off): the DFlash2 drafter's W4A16
  rows are dequantized once at the post-load hook into per-layer bf16 weights, and its apply_weights
  runs plain bf16 GEMMs; +3.22 GiB VRAM for 3.88× the hot-path weight budget. The default path is
  byte-identical (template parse + packer round-trip validated).
• W4A8 prefill cell sweep (sly/mxfp4/bench_prefill_cells.py): the prefill counterpart of
  bench_decode_cells.py — M = 96…8192, row-major vs fragment-tiled arms, --check validates every
  cell against an independent fp32 reference before any timing counts.
• sly/audit_kernel_binary.sh: a CPU-only VGPR/spill audit of the compiled kernel binaries
  (--vgpr-max 213 --spill-max 0 --strict), run against the image's .so as a per-kernel record.

Measured

• CPU only so far: all 11 int2 unit tests pass and the packer round-trip is bit-exact; the greedy
  coverage gate (≥ 99.5 % in the coarse top-16 on real weights) runs in a 300 W GPU window, and the
  production numbers follow at the acceptance gate.

---

Image: ghcr.io/slybase/vllm-sly-radiance:0.4.4-rocm10.0 (published by the build workflow for this tag).
acceptance gate passed: vllm-sly-radiance:0.4.4-rocm10.0, profile vllm7-mxfp4, mode full

vllm-sly-radiance 0.4.1

Added

• Split-K for the W4A16 Triton GEMM (sly/patch_w4a16_tiles.py, knob RADIANCE_W4A16_SPLITK, default on):
  the stock kernel runs the whole K loop in one workgroup, so the N = 5120 projections at decode M get
  160–320 workgroups for 64 CUs and stay latency-bound (INT4 target down_proj: 139 µs against 82 µs
  for the MXFP4 split-K kernel on the same shape). Per-shape entries in _GFX12X_SPLITK; a shape
  without an entry runs exactly as in 0.4.0. Affects compressed-tensors INT4 targets and the DFlash2
  W4A16 drafter. The same kernel carries two cheaper dequant schemes (scale applied to the fp32 tile
  result; bf16 magic-number nibble conversion with the zero point folded into the tile result) and an
  interleave-free unpack, selectable per table entry. Split-K partials stay <= 1 MiB (the CUDA-graph
  memory estimate charges them to the KV pool); RADIANCE_W4A16_SPLITK_TABLE=<json> loads a swept
  table without a rebuild.
• Serial split-K reduce: the program that finishes a tile last sums the partials in split order
  (per-tile counter, self-resetting) -- deterministic, and one launch less per split call (134 per
  step on the INT4 target at one request).
• Tiled W4A16 weight layout (RADIANCE_W4A16_TILED, default on): 16-row x 128-K blocks of 1 KB, so
  every row group a tile reads is one contiguous burst (the MXFP4 kernel's WPERM idea). The HIP
  skinny path for M <= 5 reads the row layout and is skipped for tiled weights.
• Fusions for W4A16 layers (sly/patch_w4a16_fuse.py): MLP gate_up + silu(gate) * up in one GEMM
  (RADIANCE_W4A16_SILU); GDN in_proj_ba re-quantized to int4 rows of in_proj_qkvz, one GEMM with two
  contiguous outputs (RADIANCE_GDN_BA_W4, INT4 target; frees the bf16 ba weights); the DFlash
  context-KV projection on the drafter's own int4 rows instead of a 105 MB bf16 copy read every step
  (RADIANCE_DFLASH_KV_W4, exact, frees ~78 MB for the KV cache); the DFlash2 grouped-conv
  kernel_projection (bf16, 10 calls per step) re-quantized to int4 (RADIANCE_DFLASH_CONV_W4; changes
  proposals only, the target verifies every token).
• The int4 lm_head in the tiled layout too (RADIANCE_LMHEAD_INT4_TILED, default on with
  RADIANCE_W4A16_TILED); it is the largest W4A16 GEMM of a step (2 x 656 MB).
• sly/check_w4a16_fuse.py: the post-load transforms end to end on stand-in layers against the unfused
  computation (fused silu, merged split, context-KV, conv projection, plain calls into transformed layers).
• sly/check_w4a16_splitk.py (numerics against an fp32 reference and bit-identity of the new paths,
  CPU via the Triton interpreter or GPU) and bench_w4a16_tiles.py --splitk.

Measured

• One R9700, 300 W, production arguments, one GPU window (2026-09-28), step gap from streamed greedy
  requests (median chunk interval, deterministic): RedHatAI/Qwen3.8-27B-INT4 42.70 → 37.43 ms
  (−12.4 %), 36.69 ms (−14.1 %) together with RADIANCE_SKINNY_GEMM=all; the control arm repeated
  at 42.76 ms. Quark MXFP4 (production) 34.47 → 34.15 ms (−0.9 %, drafter GEMMs). At unchanged
  acceptance (4.29 / 4.21 tokens per step) INT4 decode is ~125 instead of 107.8 tok/s; the gap to
  MXFP4 shrinks from −19 % to −7 % step time. At 8 concurrent requests INT4 stays far behind
  (297 vs 426 tok/s: M = 40–64, where the W4A16 kernels reach only ~250 GB/s).
• GSM8K 200 (cot zero-shot, greedy): INT4 0.825 (0.820 before), MXFP4 0.850 (baseline 0.835–0.845).
• Window E (2026-09-29, 300 W, tiled weights, serial split-K, all fusions): kernel sweep at M = 8 --
  down_proj 150.8 -> 87.7 us, out/o 70 -> 40, GDN qkvz 110.6 -> 79.2, attention qkv 98 -> 72, gate_up
  206 -> 176, drafter context-KV 194 (bf16) -> 54, drafter qkv / o / fc 47 / 48 / 207 -> 37 / 32 / 126; the
  tiled layout alone is worth 1.1-1.6x at M <= 16. Quark MXFP4 (production), all new knobs off -> on:
  step gap 34.52 -> 33.72 ms (-2.3 %), KV pool 384,316 (the split-K loss of 405 tokens is
  offset now that the drafter's context-KV no longer needs a bf16 copy). The INT4 target arms of that
  window were not run (stopped early).
• Window D (2026-09-29, split-K table of window B): INT4 step gap 42.68 -> 36.53 ms, BetterBench decode
  (128 runs) INT4 127.5 tok/s against MXFP4 128.1 in the same window; GSM8K INT4 0.83.
• Window F2 (2026-09-30, image 0.4.1-rc5 with the window-E table, second starts): INT4 all knobs off ->
  on (with RADIANCE_SKINNY_GEMM=all) 42.73 -> 34.61 ms (-19.0 %), within 3 % of MXFP4 (33.61 ms);
  KV pool 370,561 -> 375,820; GSM8K 200 0.835. Tokens per step 4.52 -> 4.24 on the four fixed prompts is
  a different greedy text, not the drafter (MXFP4 moves as much between windows, 4.41 / 4.70).
  MXFP4 (production) all knobs on: 33.61 ms, BetterBench decode 131.0 tok/s (128 runs; 0.4.0 in
  production: 130.4), KV pool 384,316, concurrency 1 / 8: 121.5 / 416.1 tok/s. The tiled int4 lm_head
  alone: 33.74 -> 33.61 ms (-0.4 %, bit-identical output).
• Not adopted: merging GDN in_proj_ba into in_proj_qkvz (96 extra rows add a 129th tile -- 93.9 us
  merged vs 79.2 + 3.6 separate), so RADIANCE_GDN_BA_W4 defaults to 0; a decode-attention retune
  at long context (the shipped rule is within 1 % of the best cell, ~470 GB/s at 32k); W4A8 int8.
• Kernel level (M = 8, DRAM-cold, bench_w4a16_tiles.py --splitk): down_proj 150 → 95 µs,
  out/o 61 → 48, gate_up 200 → 178, GDN qkvz 106 → 82, attention qkv 89 → 76, drafter qkv 53 → 39,
  drafter fc 197 → 131.

---

Image: ghcr.io/slybase/vllm-sly-radiance:0.4.1-rocm10.0 (published by the build workflow for this tag).
acceptance gate passed: vllm-sly-radiance:0.4.1-rocm10.0, profile vllm7-mxfp4, mode fast

vllm-sly-radiance 0.4.0

Changed

• vLLM 0.29.0 → 0.30.0, aiter 0.1.21.post2 → 0.1.22.post1, torchvision 0.24.1 → 0.29.0, ubuntu:24.04
  digest, all Python pins re-resolved for vLLM 0.30 (constraints.txt: 209 pins, 34 changed, new
  mooncake-transfer-engine-rocm and msgpack; setuptools held at 79.0.1 because vLLM requires <80).
• Four patch anchors moved in vLLM 0.30 and were re-anchored for both versions (apply_any):
  sly/patch_quark_mxfp4 (multi-line _resolve_backend_kernels), sly/patch_lmhead_fp8 / _int4
  (get_quant_method now opens with get_quant_method_target), patch_autoround / patch_escha
  (__all__ gained resolve_quant_method). patch_unpad, patch_qwen3_toolparse and
  sly/patch_mamba_align_retire are no-ops now: vLLM ships the same fixes.
• CI: ruff 0.16.8, renovate action v46.3.3, create-github-app-token v3.
• Renovate: no hourly or concurrent PR cap any more (updates were piling up rate-limited).
• Docs: README cut down to stack, reference numbers, quickstart, changes, options; details moved to
  docs/; this changelog; GitHub releases carry the changelog section.

Measured

• Performance-neutral against 0.3.6 (prefill ±0.3 %, step gap unchanged, KV pool 384,316), GSM8K 0.855.
  Reference run: 133.2 tok/s decode, 3,166 tok/s prefill at 1.5k, 410 tok/s at conc 8 (300 W);
  124.5 / 2,527 / 362 at 210 W (details).
• New recommendation for shorter contexts: --max-model-len 131072 --max-num-batched-tokens 4096,
  +3.5 … +4.8 % prefill on 8k–64k prompts.

---

Image: ghcr.io/slybase/vllm-sly-radiance:0.4.0-rocm10.0 (published by the build workflow for this tag).
released manually by slydlake without a green acceptance gate: gate 36115764240: all checks green except conc 16 (0.926x of the 0.2.9 baseline; every 0.3.x full run at 210 W measured 360-370 t/s there, the baseline predates them) and GSM8K aborted by a runner venv bug (lm-eval without [api], fixed). Same-window A/B vs 0.3.6 (2026-09-25, 300 W): prefill +-0.3 %, step gap equal, KV 384,316; reference run GSM8K 0.855.