v0.2.2

Highlights

• Speculative decoding. --speculative-algorithm mtp drafts with the checkpoint's own MTP layer, and --speculative-algorithm dflash drafts with a separate DFlash 2 checkpoint (#836, #844, #856, #857).
• New model. Ripple-65B Preview, from BF16 safetensors, the ModelOpt NVFP4 export or GGUF (#871).
• Faster CPU experts in hybrid. The NVFP4 W4A8 CPU kernels are faster; rerun ft bench bw so hybrid uses the new CPU rate (#851).

Speculative Decoding

Each decode step drafts k tokens and verifies them on the target in one forward, so one step can commit several tokens. With sampling, the output distribution stays the target's. Speculative decoding is off by default, and the draft stays on the GPU, so it takes some KV and expert cache space. See the flags.

Target	off	MTP (faster of k=3, k=7)	DFlash 2, k=7
Qwen3.8-27B (BF16)	50.1	144.4 (2.88x)	215.0 (4.29x)
Qwen3.6-35B-A3B (BF16)	193.1	359.6 (1.86x)	475.7 (2.46x)
Qwen3.8-Flash-Next-NVFP4	92.9	149.7 (1.61x)	174.2 (1.87x)

tok/s on an H100, one request, AIME25 problem 1, temperature=1.0, top_p=0.95, top_k=20, median of 5 runs.

• [Feature] --speculative-algorithm mtp: the Qwen3.5 family (Qwen3.5, Qwen3.6, Qwen3.8-27B) and Qwen3.8-Flash-Next, from HF or GGUF. A Flash-Next GGUF needs a Strata MTP pack (#836, #844, #856)
• [Feature] --speculative-algorithm dflash: Qwen3.8-27B, Qwen3.6-35B-A3B and Qwen3.8-Flash-Next, with a DFlash 2 draft passed through --speculative-draft-model-path, such as incoai/Qwen3.6-35B-A3B-DFlash2 (#857)
• Not supported yet: TP above 1, and targets with sliding-window or DSV4 window-tier KV.

GGUF Support

• [Feature] Serve gpt-oss GGUF checkpoints on gpt_oss (#838)
• [Feature] Serve image input on qwen3vl GGUFs from a DeepStack qwen3vl_merger mmproj (#827)
• [Feature] Serve image input on muse-glimmer GGUFs from a muse-glimmer mmproj (#830)
• [Feature] Serve image input on minimax-m3 GGUFs from a minimax_m3 mmproj (#829)
• [Feature] Add an IQ1_M MMQ kernel (#835)
• [Feature] Read NVFP4 weight scales on qwen4exp GGUFs (#845)
• [Perf] Sort MoE MMQ routes with a counting sort (#872, #878)
• [Fix] Skip inactive GGUF routes in hybrid decode (#823)
• [Fix] Log why mmproj auto-discovery skips a file (#860)
• [Docs] Document GGUF support (#867)

Model Support

• [Feature] Add Ripple-65B Preview support (#871)
• [Fix] Read stop ids from config.json without generation_config.json (#874)

Engine

• [Perf] Speed up the CPU MoE NVFP4 W4A8 kernels (#851)
• [Perf] Give only the routes the CPU computes work items in a hybrid step (#866)
• [Fix] Stop the SWA prefill commit from freeing pages the request still reads (#651)
• [Fix] Send graph-capture KV writes to the dummy page (#877)
• [Fix] Add once-per-message logger methods (#858)
• [Docs] Add a guide to speeding up decode (#859)

Kernels

• [Fix] Avoid overflow in Triton KV cache offsets (#854)
• [Fix] Size the decode KV tile to the shared-memory budget (#840)

Server and CLI

• [Feature] Report per-request prefill and decode timings in API responses with --enable-timings-report (#814)
• [Feature] Add continuous_usage_stats and count shell tok/s from it (#868)
• [Fix] Recover Qwen tool calls after unclosed think (#825)
• [Fix] Refuse n != 1 on /v1/completions instead of ignoring it (#842)
• [Fix] Reject empty completions prompt and keep error code (#850)
• [Fix] Bound top-k before int32 conversion (#790)

AMD ROCm (WIP, experimental)

• [Fix] Publish ROCm CPU expert inputs before notification (#764)

Breaking Changes & Deprecations

• qwen3vl / qwen3vlmoe, muse-glimmer and minimax-m3 GGUF models with a fitting *mmproj*.gguf in their folder now build the vision tower. --text-model-only keeps them text-only (#827, #829, #830).
• /v1/completions with n != 1 now returns 400 instead of one choice (#842).

Contributors

@jason-fxz, @ischencheng, @void-mckenzie, @LZ-QWQ, @madbrain76, @festeraeb, @A5bhinav, @zihaomu, @wenlive, @techsaurabh-394

First contributions this release:

• @madbrain76 in #825
• @festeraeb in #840
• @A5bhinav in #842
• @wenlive in #790
• @techsaurabh-394 in #850

Full changelog: v0.2.1...v0.2.2