vllm-sly-radiance

The fastest way to run Qwen3.8-27B on one AMD Radeon AI PRO R9700. A vLLM image for gfx1201 (RDNA4) with hand-written MXFP4 kernels, DFlash2 speculative decoding and the full 262k context on a single 32 GB card: ~136 tok/s single-stream decode, ~3,100 tok/s prefill (2,635 at 47k), 461 tok/s at 8 concurrent requests, 437k-token KV pool (1.0.0 reference run at 300 W, see Performance), with every number measured and reproducible.

docker pull ghcr.io/slybase/vllm-sly-radiance:1.1.0-rocm10.1

Reference run of image 1.0.0 on 2026-10-07 against 0.7.0 in the same GPU window (1.0.0 first, then 0.7.0), 300 W (firmware fan curve), Swift-1.5 Qwen3.8-27B GPTQ-MXFP4 + DFlash2 k = 7, production arguments of each image, BetterBench 0.4.0 default config (3 warmup + 20 passes per category, prefill 8 runs per depth, concurrency 1–16 × 48 requests, T 0.7 / top_p 0.95 / top_k 20):

The older reference run of image 0.4.0 (Quark checkpoint, 300 W and 210 W) is kept below for comparison.

Reference run of image 0.4.0 on 2026-09-25, production arguments from the quickstart (--max-model-len 262144, fp8 KV, bf16 SSM state, DFlash2 k = 7, KV pool 384,316 tokens), BetterBench 0.4.0 default config: single-stream decode 3 warmup + 20 passes per category, prefill sweep with unique prompts, concurrency 1–16 × 48 requests, sampling temperature 0.7 / top_p 0.95 / top_k 20. Two power settings, no other traffic: 300 W (firmware fan curve) and 210 W (fan capped at 2,800 rpm, what this card runs 24/7).

What that means: code, JSON, math and edits decode at 140–175 tok/s (the drafter lands 5–6 tokens per step), free prose at ~85; long contexts stay fast (99k tokens of context cost ~7 ms per step instead of 83). Per-category tables, the measurement method and every earlier release are in docs/BENCHMARKS.md. Decode differences below ~5 % between single runs are noise; see there for why.

Requirements: one R9700 (or another gfx1201 card with 32 GB), ROCm-capable kernel driver (/dev/kfd, /dev/dri), Docker, ~20 GB of disk for the model + drafter, and enough free host RAM for the model load. Since 1.2 the MXFP4 checkpoint is sharded by default, so vLLM loads it shard-by-shard and the peak host RAM is the largest shard (5 GiB) instead of the whole ~19 GiB model — a 16 GiB box now loads it in minutes rather than swapping for hours. On the maintainer's 32 GB host the other VMs are paused while the model loads; on 16 GiB hosts they no longer need to be. Windows 11 through WSL2: see docs/WINDOWS-WSL2.md.

# 1. model + drafter into the Hugging Face cache
hf download amd/Qwen3.8-27B-Quark-AWQ-MXFP4
hf download syvai/Qwen3.8-27B-DFlash2-W4A16

# 2. serve (OpenAI-compatible API on :8000)
docker run -d --name vllm --restart unless-stopped \
  --device=/dev/kfd --device=/dev/dri --group-add video --group-add render \
  --security-opt seccomp=unconfined --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -v vllm-cache:/root/.cache/vllm -v triton-cache:/root/.triton -v aiter-cache:/root/.aiter \
  -e HIP_VISIBLE_DEVICES=0 -e GPU_MAX_HW_QUEUES=2 \
  -e RADIANCE_MXFP4=1 -e RADIANCE_MXFP4_W4A8=1 -e RADIANCE_MXFP4_W4A8_MIN_M=0 \
  -e RADIANCE_MXFP4_DECODE_MAX_M=128 -e RADIANCE_MXFP4_A_TILED_MIN_M=513 -e RADIANCE_MXFP4_WPERM=1 \
  -e RADIANCE_LMHEAD_INT4=1 -e RADIANCE_FUSED_NORM_QUANT=1 \
  -e RADIANCE_KV_GROUP_SIZE=8 -e RADIANCE_EMBED_INT8=1 -e RADIANCE_EMBED_BITS=4 \
  -e RADIANCE_MXFP4_WIDE_MAX_M=192 -e RADIANCE_ADAPTIVE_WIDTH=perseq \
  ghcr.io/slybase/vllm-sly-radiance:1.1.0-rocm10.1 \
  --model amd/Qwen3.8-27B-Quark-AWQ-MXFP4 --quantization quark \
  --max-model-len 262144 --gpu-memory-utilization 0.98 \
  --kv-cache-dtype fp8 --mamba-ssm-cache-dtype bfloat16 \
  --speculative-config.method dflash \
  --speculative-config.model syvai/Qwen3.8-27B-DFlash2-W4A16 \
  --speculative-config.num_speculative_tokens 7 \
  --speculative-config.draft_sample_method probabilistic \
  --speculative-config.attention_backend TRITON_ATTN \
  --attention-backend R4D_HYBRID \
  --max-num-seqs 8 --max-num-batched-tokens 2048 \
  --compilation-config.cudagraph_mode FULL_DECODE_ONLY \
  --compilation-config.cudagraph_capture_sizes '[8,16,24,32,40,48,56,64]' \
  --enable-prefix-caching --skip-mm-profiling --enable-mm-embeds \
  --limit-mm-per-prompt.image 0 --limit-mm-per-prompt.video 0 \
  --enable-auto-tool-choice --tool-call-parser qwen3_xml --reasoning-parser qwen3 \
  --chat-template /opt/qwen-fixed.jinja \
  --default-chat-template-kwargs '{"reasoning_effort": "medium"}' \
  --override-generation-config '{"temperature": 1.0, "top_p": 0.95, "top_k": 20, "min_p": 0.0, "presence_penalty": 0.0}'

# 3. test
curl -s localhost:8000/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"amd/Qwen3.8-27B-Quark-AWQ-MXFP4","messages":[{"role":"user","content":"Hello!"}]}'

• First start compiles kernels and CUDA graphs (~10 min) and reports a ~2 GiB smaller KV pool; restart once — the warm start takes 4–6 min and gets the full 436k-token KV pool. Keep the cache volumes.
• Power: 210 W costs ~6.5 % decode and ~20 % prefill against 300 W (table above). The card allows up to 330 W: +4 … +8 % prefill and ~+1 % step time over 300 W, but the junction runs at 102 °C average (106 °C peak) and the fan at ~4,200 rpm — not worth it for 24/7 use.
• iGPU: HIP_VISIBLE_DEVICES=0 keeps ROCm off an integrated GPU next to the R9700.
• Other checkpoints: NVFP4, INT4, ParoQuant; several GPUs: docs/MULTI-GPU.md.

On top of vLLM, StillDeadcode/vllm-radiance (the RDNA4 build, libr4d kernels, gfx1201 fixes; FP8 on two cards) and ggz14/radiance-vllm-mxfp4 (MXFP4 on RDNA4, the W4A8 kernel), this fork makes one card with one MXFP4 model as fast as it gets. Each item was an A/B on the production arguments; the numbers are the gain it measured when it went in (details, per release).

What was tried and rejected, with reasons: docs/NOT-ADOPTED.md.

Recommended = what the reference run above uses. All knobs are environment variables read at start; everything not listed is off by default and documented in docs/TECHNICAL.md and sly/README.md.

RADIANCE_ATTN_DECODE_TUNE, RADIANCE_ATTN_DRAFTER_TUNE, RADIANCE_LOOKUP_DRAFT (prompt lookup), RADIANCE_USE_R4D (libr4d kernels), RADIANCE_MXFP4_DECODE_TUNE16, and for the W4A16 GEMMs (INT4 targets and the drafter): RADIANCE_W4A16_TILES (tile table), RADIANCE_W4A16_SPLITK (split-K / fast dequant, per-shape table; RADIANCE_W4A16_SPLITK_TABLE=<json> replaces it), RADIANCE_W4A16_TILED (weights in 1 KB 16-row blocks), RADIANCE_LMHEAD_INT4_TILED (the int4 lm_head too), RADIANCE_W4A16_SILU (gate_up + silu in one GEMM), RADIANCE_DFLASH_KV_W4 (drafter context-KV on its int4 rows instead of a 105 MB bf16 copy), RADIANCE_DFLASH_CONV_W4 (drafter conv projection int4). Set any to 0 only for an A/B. RADIANCE_GDN_BA_W4=1 (GDN in_proj_ba as int4 rows of in_proj_qkvz, one GEMM) stays off: measured slower.

Less context, more prefill: if your requests stay well below 262k tokens, set --max-model-len 131072 --max-num-batched-tokens 4096: +3.5 … +4.8 % prefill on 8k–64k prompts, and the KV pool (~326k tokens) still holds two full 131k requests. Measured with image 0.4.0 on 2026-09-25 (300 W, prefill sweep; KV pools are 0.4.0 values, 1.0.0 holds ~11 % more):

Decode and concurrency are unchanged within noise. (These are prefill-only sweeps, which read ~5 % higher than the prefill phase of the full run in the table above; compare within one table.)

A second-tier prefix cache behind the GPU prefix cache for long-context / agent-style sessions, where every turn re-sends the whole conversation and the prefix outgrows the GPU pool. Off by default: with KVCACHE unset (or off) the serve command is byte-identical to the unmodified script — nothing is mounted, patched or passed to the engine.

The offload patches, kvwatch.py and the turnbench/tierbench benches live in kv-cache/ and ride the repo's existing /patches mount; serve-mxfp4.sh applies them at container start (not baked). The disk tier additionally needs its own filesystem plus a host systemd reaper (your step, not the image's). Measured (one R9700, TP=1, KVCACHE_RAM_GIB=16, 110k-token agent sessions): a ~81k-token request drops from 38.8 s (no cache) to ~13 s (RAM tier) — see kv-cache/README.md. This reduces re-prefill latency for long sessions; it does not raise --max-model-len.

This image stands on other people's work:

• StillDeadcode — vllm-radiance and libr4d: the RDNA4 build, the gfx1201 kernels and fixes everything here runs on.
• Radiance engine fork (StillDeadcode/radiance, our fork slydlake/radiance) — the adaptive_k verify-width policy behind RADIANCE_ADAPTIVE_WIDTH and the M-band findings (wide decode tile) behind RADIANCE_MXFP4_WIDE_MAX_M, ported to vLLM's graph model.
• libr4d — the paged prefill attention kernel behind R4D_HYBRID.
• ggz14 — radiance-vllm-mxfp4: MXFP4 on RDNA4 and the W4A8 kernel, the libr4d extras, TP=3 padding, and the NVFP4, ParoQuant and AutoRound paths.
• vLLM, AITER, the DFlash authors; sly/gdn/radiance_gdn_decode.hip ports vLLM's fused GDN decode kernel (Apache-2.0).
• syvai for the DFlash2 W4A16 drafter, AMD for the Quark MXFP4 checkpoint, froggeric for the fixed Qwen chat template.
• turboderp/exllamav3 (MIT) — escha/ derives from it; z-lab for the ParoQuant format.

License: same as upstream vllm-radiance (see LICENSE).