Skip to content

v0.5.17

Latest
Latest

Choose a tag to compare

View all tags
@Fridge003 Fridge003 released this 08 Aug 00:19
· 211 commits to main since this release
v0.5.17
2948168
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Highlights

582 PRs from 194 contributors.

Kimi K3 day-0 support: A 2.8T-parameter multimodal LatentMoE (896 experts, top-16, routed in a 3584-dim latent space) with a 1M-token context, 69 KDA linear-attention layers interleaved with 24 MLA layers, and a MoonViT3d vision tower, shipping as a native MXFP4 checkpoint. SGLang serves it from day 0 with DCP, DSpark speculative decoding, chunked-prefill PP with TP decode, KDA-aware prefix caching, HiCache L2 over DCP, LoRA on the quantized weights, and reasoning, tool-call and OpenAI-compatible serving, verified on NVIDIA GB300 and AMD MI35x (#32541, #32828, #32890, #33025, #33112, blog, cookbook, roadmap).

MiniMax-H3 day-0 support: MiniMax's video generation model that produces a video and a synchronized stereo audio track in one request, served natively on SGLang-Diffusion across all three public task profiles: text-to-video-and-audio (t2va), first/last-frame conditioning (fl2va), and image/video/audio reference conditioning (ref2va, which also covers video-to-video). Verified on B200 (TP2 + Ulysses4), H100 (TP2 + Ulysses2), and 2x RTX 5090 with layerwise offload (#33275, cookbook).

Other new models added: EmbeddingGemma and LFM2.5 embedding models, nvidia/MiniMax-M3-NVFP4, plus cookbook recipes for Poolside's Laguna-S-2.1 family and Inkling-Small.

Initial Support for Rust Frontend: Initially supporting rust frontend, which migrates front-half (everything from network ingress up to the point a tokenized request is handed to the GPU scheduler) from Python to a multi-threaded Rust implementation. #29799

DCP communication backends and q-replicate (Helix): The DeepSeek-MLA decode context-parallel path gains pluggable comm backends. a2a exchanges packed attention output plus fp32 LSE in a single NCCL collective per layer, with fp8 KV carried as uint8 byte transport; fi_a2a delegates the cross-rank exchange to the FlashInfer MNNVL kernel on GB200. --dcp-replicate-q-proj projects full-head Q locally and skips the per-layer Q head-dim all-gather. Select with --dcp-comm-backend {ag_rs, a2a, fi_a2a} (#21637).

DWDP for MoE prefill: A new prefill parallelism strategy that prefetches peer expert weights over NVLink P2P and computes all experts locally, removing EP all-to-all token dispatch. On 4x B200 with gpt-oss-120b, prefill-only, DWDP4 reaches 1.92x over DEP4 at MNT 32K / ISL 32K, and 506K vs 329K tok/s (1.54x) at saturation (CONC=128, ISL=8K). Enable with --dwdp-size; the authors mark it early-development (#29778).

Session-reference-aware Unified Radix Cache: For agentic and RL-rollout workloads, requests can carry a stable session_id so eviction knows which prefixes an active session still references, instead of evicting purely by cache policy. Release the references with /close_session. Opt in with --enable-session-radix-cache (#29173).

SM90 FP8 MegaMoE for DeepSeek-V4: Adds the DeepGEMM MegaMoE A2A path on SM90 for DeepSeek-V4-Flash/Pro FP8, including the pre-dispatch JIT kernel and FP8 expert weight preparation. Guarded behind SGLANG_OPT_USE_DEEPGEMM_MEGA_MOE=1 (#29016).

Faster engine recovery: Large-model restarts cost 3 to 6+ minutes today, about 6.5 minutes for Qwen3-235B FP8 on 4 GPUs, because weights reload from storage and CUDA graphs recapture. A weight-cache daemon holds weights per GPU so a restarting engine can recover from cache instead (#27139).

Lower host overhead in hybrid-linear MTP decode: Under spec-v2 overlap scheduling each decode step runs draft, verify and extend CUDA graphs, and the eager seams between them become GPU idle time at low concurrency. This trims that host work so the host stays off the critical path (#32219).

Dependencies: flashinfer 0.6.15.post1 (#31927), sgl-deep-gemm 0.1.5.post1 (#32345, #33143), helion 1.4 (#32562), mooncake 0.3.12.post1 (#32302), dynamo-tokenizers 1.7.0 (#32981). PyTorch stays at 2.11.0 and the CUDA base image at 13.0.1.

Full release notes by category below; breaking changes and known issues are at the end.

New Model Support

Model Type PRs Cookbook
Kimi K3 Autoregressive (Multimodal) #32541, #32828, #32890, #33025 link
MiniMax-H3 Diffusion #33275 link
MiniMax-M3-NVFP4 Autoregressive #31989
EmbeddingGemma Autoregressive (Embedding) #32375, #32383 link
LFM2.5 Autoregressive (Embedding) #28691 link

Kimi K3

  • [Kimi] Support kimi-k3: #32541 ⭐
  • [Kimi] Support DCP + DSpark (ported from kimi-k3 branch): #32828
  • [Kimi K3] Add reasoning, tool-call, and OpenAI serving support: #33025
  • feat(kernels): port standalone Kimi K3 kernels: #32890
  • [Feat] DCP + HiCache L2 Support (ported from kimi-k3): #33112
  • Replace Kimi K3 DeepGEMM patch with 0.1.5.post1: #33143
  • docker: add Kimi K3 images: #32760

Rust Server

A native Rust serving layer: tokenizer manager, ingress validation and egress, an OpenAI-compatible API server, and PD disaggregation support, shipped as prebuilt release artifacts.

  • support rust sglang server: #29799
  • create rust workspace: #32014
  • init sglang rust server project: #32256
  • add the rust server tokenizer, detokenizer, and egress modules: #32872
  • add the rust server ingress request validation and api server common types: #32873
  • add the rust server ingress tests, guard, and submit modules: #32874
  • add the rust server api frame codec and http server entry: #32875
  • add the rust server native api handlers and runtime threads: #32876
  • wire the rust server modules into lib, runtime, and tokenizer manager: #32877
  • sglang rust server tokenizer manager, ring and runtime: #32358
  • feat: rust sglang server openai apis: #33103
  • [rust-server] PD disaggregation support: #33125
  • [rust-server] Reland: fix TCP-layer TTFT stalls (#33026): #33269
  • rust server build release artifacts: #33096

Speculative Decoding

  • [MTP] Cut spec-v2 host-seam overhead in hybrid-linear MTP decode: #32219 ⭐
  • [KDA] Add target_verify support for speculative decoding: #26888
  • [DSPARK] Grammar-constrained decoding, incl. tool_choice=auto: #31753
  • [DFLASH] Support grammar-constrained decoding in speculative verify: #30096
  • [Spec] Enable grammar overlap scheduling for STANDALONE speculative decoding: #32110
  • Overlap grammar (constrained decoding) with speculative decode verify: #31488
  • [Spec] Support sampling in the DSPARK graph-folded draft proposal: #33298
  • [Spec] Add trtllm_mha support for Gemma 4 MTP draft attention backend: #25545
  • [Perf] Fast-path chain-style draft token organization in multi-layer EAGLE: #32887
  • [Perf] Skip the target-verify tree mask fill when the backend never reads it: #32886
  • [Spec] Compact the target-verify mask when nothing reads it: #32920
  • [Perf] Fold dspark dense draft embedding into the draft graph via forward_embed: #31985
  • [Perf] Stack dspark dense draft per-layer ctx KV projection into one GEMM: #31986
  • [Mamba] Support speculative decoding with extra_buffer_lazy: #30437
  • [gdn] fused replayssm ring write into flashinfer gdn mtp verify kernel: #33102
  • [gdn] support replayssm with extra buffer: #32692
  • [6/6][kimi-deterministic] Use deterministic seeded coins for EAGLE rejection sampling: #30822
  • [Speculative Decoding] Fix GPT-OSS EAGLE3 hidden states: #32334
  • [EAGLE] Handle NaNs in fused top-k=1: #32396
  • Pack aux hidden states into a preallocated buffer: #28956
  • Support SGLANG_SIMULATE_ACC_LEN for DFLASH: #32595
  • Fix stop boundaries for grammar-constrained speculative decoding: #31738
  • Fix DSpark loading for hybrid DSV4 NVFP4: #33276
  • Fix DSpark and DP/EP: #33098
  • [Fix] Clamp degenerate all-sentinel draft rows to token 0 in dspark _online_combine_kernel: #32277
  • [Fix] Bound FULL_MASK verify-mask reuse by the captured max_bs: #33127

Piecewise & Breakable CUDA Graph

  • Turn on breakable prefill cuda graph for dp attention by default: #31682
  • [FullCG] Support chunked cached-prefix prefill: #30825
  • [FullCG] Preserve attention LSE through the custom-op boundary: #31050
  • [BCG][3/N] Enable bcg on dsa & deepep a2a backend: #31987
  • [BCG][4/N] Enable bcg on megamoe & flashinfer a2a backend: #33150
  • [CUDA Graph] Allow custom decode graph runners: #33553
  • [cuda_graph] Gate breakable-CG capture_inputs retention to DP-gather paths: #32678
  • Enable multimodal prefill BCG for VL and audio models: #30872
  • Enable post-capture KV sizing with DP attention: #33427
  • fix(server): capture legal multi-request prefill CUDA graph batches: #30206
  • Fix BCG circular import during server startup: #33371

Attention Backends

  • [Attention Backend] Add HPC-Ops attention backend: #30540 (blog)
  • [Attention Backend] Extend hpc_ops dynamic-scheduled decode to bf16: #32304
  • [DSA] Q8KV8 FP8 Sparse Prefill on GLM-5.2 & DeepSeek-V3.2: Q8-Path & Shared-Path Optimizations: #31888
  • [unified-memory] Support MLA-hybrid-Mamba (Kimi-Linear) on the Triton backend: #32971
  • [unified-memory] Let Kimi-Linear use the paged MLA attention backends: #32972
  • [unified-memory] Support fa3, the default MLA backend on pre-Blackwell hosts: #33046
  • feat(inkling): migrate short convs onto the ShortConv attention backend: #33023
  • [GDN] Support FlashInfer GDN prefill with extra-buffer radix cache: #29735
  • fix(gdn): skip the -1 padding sentinel in the chunked extend kernel: #33810
  • [Kernel] Fuse KV-cache writes for asymmetric K/V (head_dim != v_head_dim): #32813
  • [Perf][DSA] Pass topk_length to flash_mla_sparse_fwd in the sparse attention path: #31128
  • [Perf] Skip page-table columns past kv length in DSA draft-extend metadata kernel: #31981
  • [Perf] Skip blocks past per-request live length in full-width Triton kernels: #32109
  • Support a same-size mixed q dtype in the fused RoPE kernels: #31834
  • Avoid TRTLLM prefill output copy: #33306
  • Fix attention backends for models with per-layer head counts (num_attention_heads_per_layer): #32625
  • [Fix] Route asymmetric-KV models to fa4 on SM100 and pin MiMoV2 FP8 MoE to flashinfer_trtllm: #32818
  • [Fix] Fix trtllm_mla backend + fp8 kv cache without rope: #32181
  • [Fix] Support ENCODER_ONLY target-verify in the trtllm_mha backend: #32178
  • [Fix] missing max_context_len on HybridAttnBackend: #32690
  • fix(dsa): correct packed FlashInfer top-k and backend selection semantics: #32490
  • fix(dsa): fail fast on fp8_e4m3 KV with tilelang DSA backend on CUDA: #31346
  • [KDA] Fix mixed exponent bases in Triton chunk prefill: #31904
  • [FIX] Prevent Lightning Attention extra-buffer mamba state corruption: #29973
  • Add configurable FlashInfer autotune skips: #31389
  • Fix FlashInfer MNNVL workspace size check: #32318

MoE & Expert Parallelism

  • [Feature] Add DWDP (Distributed Weight Data Parallelism) for MoE prefill: #29778 ⭐
  • Add SM90 FP8 MegaMoE support for DeepSeek-V4: #29016 ⭐
  • [MoE Backend] Add HPC-Ops FP8 MoE runner backend: #30541 (blog)
  • Integrate pplx a2a backend: #30756
  • Support DeepGEMM for standard MoE dispatch: #33128
  • [DSV4] Support megamoe for CP: #29569
  • [MoE] Make DeepEP auto serve flashinfer_cutedsl FP4 (coerce to low_latency) + guard: #29523
  • [2/N] elastic-ep: Enable EPLB after scale-up: #30553
  • [Elastic EP] Centralize Mooncake PG configuration: #31708
  • [Elastic EP] Fix recovery lifecycle and add manual coverage: #31744
  • perf: speed up marlin moe with occupancy-aware launch specialization: #31552
  • Migrate CompressedTensorsW4A4Nvfp4MoE TRT-LLM path onto MoeRunner: #32248
  • Load initial expert location metadata on CPU: #32435
  • Fix silently wrong EPLB output with --moe-a2a-backend none (rank-invariant dispatch): #32962
  • Fix MoE reduce-scatterv eligibility check: #32663
  • Fix pad-row top-k masking with custom_routing_function under DP attention: #31838
  • Add DeepSeek-reference 1e-20 epsilon to top-k renormalization to prevent 0/0 NaN: #31017
  • [Fix] Clear stale FlashInfer BF16 MoE index cache: #33016
  • Optimize LongCat-Flash router GEMM with the HPC-Ops bf16xfp32 kernel: #30247
  • [Fix] Reject online weight updates while the HPC-Ops router GEMM split cache is active: #31943

Quantization

  • Support CuteDSL GEMM BF16 on SM100 on by default when allowed by heuristic: #30567
  • [Quant] Support NVFP4_AWQ checkpoints in ModelOpt FP4 path: #31825
  • [Quant] Keep the flashinfer_deepgemm FP8 GEMM to 1 <= M < 32: #32843
  • Qwen3.5-MoE: support modelopt_fp4 checkpoints that quantize attention (+ load baked FP8 KV scales): #31220
  • [Qwen3.5][MTP] Support FlashInfer CuTe DSL for online NVFP4 draft MoE: #31382
  • Fixing MXFP8 online quantization pipeline: #31510
  • [Fix] Restore online MXFP8 quantization for linear layers: #32953
  • [Bugfix] compressed-tensors: mixed-precision checkpoints silently load unquantized: #32736
  • Fix compressed-tensors NVFP4 MoE W13 layout: #32430
  • fix(marlin_nvfp4): only apply routed_scaling_factor in moe_sum_reduce: #31762
  • [Perf] Halve the non-finite sanitization overhead in per_token_group_quant: #32296
  • [JIT] Trait-driven per_token_group_quant: unify the quant kernel family (flat + masked): #30924
  • [JIT] Restore the previous division behavior in per-token group quantization: #32616
  • Fix FP8 Triton dtype selection on A100: #31340

Parallelism & Disaggregation

  • [Feature] DCP: A2A + FlashInfer-MNNVL comm backends and q-replicate (Helix): #21637 ⭐
  • [DeepSeek V4] CP decode opt: slice repeat attention weights to local TP partition: #27657 ⭐
  • [4/N][CP] Support interleave strategy for cp v2: #30482
  • [CP] Fuse zigzag attention into a single call: #33137
  • [CP] Support breakable CUDA graphs for zigzag strategy: #33136
  • [CP]: FIx some issue for glm5.2 cp v2: #33100
  • Support DCP for Kimi Linear model: #32612
  • feat: support Kimi Linear PD disaggregation with DCP: #32837
  • [Bugfix] Fix Kimi-Linear state transfer across heterogeneous TP: #32262
  • [DCP] Match the replicated draft KV pool's page granularity to its allocator: #33348
  • [DCP] Bound a request by the aggregate KV pool, not one rank's share: #33448
  • [Fix] Honor FlashMLA natural-log LSE in DCP reduction: #33065
  • fix(mem_cache): state the MLA KV bound in the DCP index space: #33432
  • [comm] Enable multi-node custom-AR v2 on a single NVLink clique: #32339
  • [Fix] Enable graph capture and MSCCL++ for attention TP groups: #31629
  • Disable extra NCCL CUDA event synchronization with symm mem: #27089
  • [PD] NIXL connector: shard by destination: #32025
  • [PD] pool decode bootstrap HTTP sessions: #31543
  • [PD] Handle abort requests in PP mode: #32797
  • [PD+PP] Honor PP consensus for bootstrap and prealloc: #31869
  • [PD] Prevent decode scheduler from blocking on ZMQ sends to a stalled prefill peer: #31144
  • [PD] Fix false health-503 during decode retraction re-admission: #33118
  • [PD] Drain NIXL completion notifications before enforcing the WaitingForInput timeout: #32267
  • [Disagg][NIXL] Fix heterogeneous attn-TP KV transfer for replicated GQA heads (NIXL_ERR_NOT_FOUND): #31968
  • [Disagg][StagingBuffer][1/2] Robustness and failure handling: #31217
  • fix(disagg): support pipeline-parallel hybrid-linear transfer: #32270
  • [BugFix][EPD] Harden zmq_to_scheduler receiver failures; sync error info across TP: #31592
  • [EPD] Fix HTTP dispatch lock blocking cross-request encoder batching: #31275
  • [EPD] Make encoder register/unregister health-check robust: #31576
  • [BugFix][EPD] Fix Mooncake source-MR lifecycle for multi-TP /send: #32071
  • [BugFix][EPD] Early-release mooncake GPU embeddings; fix gpu_id via scheduler.ps: #31591
  • [HiSparse]Fix DeepSeek V4 HiSparse PD Transfers with Separate Host and Device KV Indices: #31901
  • support dp attn with client lb: #33105
  • Fix token count localization for replicated attention-TP forwards: #32411

Scheduler & Runtime

  • [FEAT] Support fast engine recovery through weight cache: #27139 ⭐
  • feat(grpc): add generation request semantics: #32588
  • [Scheduler] Enable decode retraction ordering under speculative decoding: #32023
  • Allow optimistic prefill with L2 hierarchical cache and write-back policy: #33545
  • Bound prefill delayer all-branch delay and decay the max_prefill_bs high-watermark: #32880
  • Negotiate PrefillDelayer only after KV-budget admission checks: #31835
  • [Bugfix] Fix prefill suspension caused by delayed negotiate_should_allow_prefill invocation: #32389
  • Fix SWA admission livelock on cached-prefix resumes: #32379
  • [core/loader] Add presharded load format: #24256
  • Support fastsafetensors no-GDS loading and page-cache release: #31859
  • Fix async loading of RunAI-streamed tensors: #32896
  • Fix DeepSeek V4 loading with RunAI Model Streamer.: #30240
  • [Fix] Fail fast when a safetensors index references missing shard files: #32279
  • AutoWeightLoader support Sglang native models 1: demo: #28671
  • Fix reward/classification models broken by load_weights v2 dispatch (#28671): #31988
  • perf(startup): skip unused PyTorch headers for KV VMM allocator stub: #33126
  • Reduce startup log noise and fix Dynamo / CUDA-graph edge cases: #33428
  • [Feature] Add leveled invariant-check primitive for nan/inf/oob validity checks: #32308
  • [Observability] Add startup, memory, and hybrid SWA diagnostics: #33375
  • [metrics] Split tokenizer request metrics by stream: #32734
  • Add stream label to TTFT metrics: #32363
  • fix(metrics): clear forward occupancy on idle: #33562
  • Report accelerator type in /v1/loads: #32348
  • [Router] Report accelerator count in /v1/loads: #33548
  • Add prefill and decode load counters to LoadSnapshot: #32245
  • [PD] Add a queues.prealloc_ready counter to the load snapshot: #33133
  • [Fix] Include disagg prefill waiting queue in FPM: #32122
  • fix: prevent ReqTimeStats from being dropped during IPC serialization: #31339
  • config: make ServerArgs read-only with a single audited mutation entry: #31811
  • runtime_context: per-role namespace enforcement behind SGLANG_ROLE_NAMESPACES: #33172
  • [misc] Deep-merge nested config overrides and parse request bodies with orjson: #33351
  • [Perf] Free KV pages by segment in the paged allocator without a device sync: #32701
  • [mem_cache] Build empty-prefix last_loc sentinel on-device to avoid per-call H2D sync: #32575
  • [Kernel] Skip KV writes to reserved padding slots: #32477
  • Skip dist_init/nccl port prechecks when the dist init method is overridden: #31410
  • [BugFix] Prevent TBO crash when return_logprob is enabled: #32180
  • [BugFix] Fix DS/Kimi crash on non-first PP ranks when resolving input length: #31752
  • [Fix] Two root causes of the H100 deepep TBO CI break: scale-tensor use-after-free + missing non-finite quant sanitization: #32188

HiCache & Radix Cache

  • feat: Session-reference-aware Unified Radix Cache for agentic multi-turn workloads: #29173 ⭐
  • Radix Cache Split: Spin off TreeCore: #29901
  • [Hicache][1/2]Support Mamba branching in Unified Radix Cache with HiCache: #31181
  • feat(hicache): Add shared memory allocator for host KV cache: #29326
  • [HiCache] Add model-aware key isolation to Mooncake Store: #31920
  • Add Mooncake tenant id support: #30256
  • Eliminate redundant DSA state transfers (Mooncake): #32620
  • [HiCache] Merge HiCache event checks to reduce decode overhead: #30511
  • [HiCache]: Optimize hybrid/DSA L3 prefetch result sync and usable-prefix clamping: #31443
  • fix(hicache): support staged write-back for asymmetric MHA: #30981
  • Write-back policy fix for unified tree: #31845
  • Fix --hicache-size allocating ~2x host memory on hybrid SWA: #32373
  • Fix --hicache-size allocating ~2x host memory on hybrid Mamba: #32915
  • [UnifiedTree] fix: drop prefetched host refill under an un-backed-up parent: #31902
  • [UnifiedRadixCache][mamba] Fix mamba state corruption and slot leak when load_back aborts: #30986
  • [Fix] Evict only the KV shortfall in evict_from_tree_cache: #32016
  • [Fix] Unify pinned host pool release on graceful shutdown: #32029
  • fix(hisparse): correct DSA KV memory budget: #31992
  • perf(hisparse): eliminate redundant swap output fill: #32483
  • [Mamba] Add a per-path cap for cached states: #31230
  • Skip mamba lock during decoding: #32228
  • [Fix] Reserve the mamba pool's +1 padding slot in the memory budget solve: #32184
  • Fix unnecessary gather/scatter on CPU for non-contiguous Mamba statepool: #31754
  • [XPU] Add XPU device support for LMCache radix cache integration: #23534

LoRA

  • [LoRA] Support LoRA under the breakable/full prefill CUDA graph: #30988
  • Split #32584 into 2/2: [LoRA] Shard attention LoRA by attn-TP and allow dynamic LoRA with dp attention: #32708
  • Split #32584 into 1/2: [LoRA] Guard DP-attention idle forwards against stale LoRA batch state: #32707
  • [LoRA] 1/n Per-rank tensor serialization for load_lora_adapter_from_tensors under dp_size > 1: #32580
  • [LoRA] Guard TMA down path for LoRA hooks: #31608
  • [lora] Fix WAR race: never write MoE runner output into hidden_states in place: #31870
  • [LoRA] Fix Marlin MoE kernel import: #32884
  • LoRA: Ascend: Update ascend LoRA backend to support new kernels: #15912

Multimodal

  • [mm] sglang-mm: server vision pipeline core (fetch/driver/pipeline) + Qwen VL: #32364
  • vlm: parallelize multimodal preprocessing with customized worker num: #31438
  • [Perf] Broadcast single-image DP vision embedding instead of pad-to-max all-gather: #33307
  • [Perf] Speed up the Kimi-K2.5 vision path and match PIL bicubic in the GPU resize: #33349
  • [EPD][VLM] Fix Kimi-VL 2D encoder grids: #32104
  • [mm] Accept per-item embedding lists from DataEmbeddingFunc: #31826
  • [mm] Handle per-item embeddings in cache misses: #32498
  • Decode input_audio media containers with PyAV & Update memory profiler: #31832
  • fix(vlm): materialize Qwen3-VL features on the vision device: #31596
  • vlm: reject moss vision metadata mismatches: #31957
  • [Bugfix] Place empty Qwen encoder-DP embeddings on the communication device: #31663
  • [Fix] --mm-process-config crash when video config contains: #30260
  • [Fix] Drop deprecated multimodal processor residency state: #33308

Model Support & Optimizations

  • [Feature] Support return_hidden_states="last": #30177
  • Add return_token_ids support to completions and chat completions APIs: #30917
  • embedding: centralize capabilities and complete OpenAI compatibility: #32481
  • [Model] Support standalone text-only Qwen3.5 checkpoints: #32401
  • model: serve bare Qwen3Model backbone natively as an embedding model: #32457
  • [feat] Opt-in flat response format for prompt top logprobs: #32078
  • [feat] Optional base64 encoding for the flat prompt top logprob arrays: #31960
  • [perf] Assemble flat prompt top logprobs scheduler-side as numpy arrays: #32223
  • Use native batched llguidance mask generation: #32412
  • [Spec] Hold the grammar bitmask in one GrammarMask type across all decode paths: #32409
  • [Spec] Consolidate the grammar sync decision into ScheduleBatch.grammar_needs_sync: #32353
  • [Spec] Share the grammar mask build and verify-tree staging across spec workers: #32393
  • [Spec] Derive NGRAM grammar tree links on the host instead of reading back retrive_next_token: #32380
  • support regex that compatible with python re lib however apply more l...: #32676
  • fix(sampling): reject conflicting structural tag constraints: #32525
  • [Fix] Treat an empty grammar constraint as unset in SamplingParams: #33328
  • Add 'anyOf' schema support for qwen3_coder tool call parser: #30832
  • Fix dropped tool calls when a stream delta carries several: #31860
  • Fix Inkling tool-call parsing recovery, content handling, and streaming: #32861
  • Fix LFM 2 tool parser.: #27614
  • [Fix]: render tool_reference schema regardless of tool_result part order: #32522
  • Treat partial_json_parser AssertionError as incomplete JSON: #31975
  • Flush dropped reasoning at stream end when stream_reasoning=False: #32225
  • fix(reasoning): let --enable-strict-thinking works for DeepSeek-V4: #32400
  • fix(reasoning): honor Poolside template thinking defaults: #32540
  • Guard min_new_tokens penalizer against None eos_token_id: #31973
  • [Inkling] Add minimal DFLASH support: #31840
  • [Inkling] Hold the short-conv per-step state on one metadata struct: #33116
  • Fix broken Nemotron DP attention: #33123
  • [Nemotron] Fix decode track-save reading the stale tail of the CUDA-graph track buffer: #32555
  • [Nemotron] Hoist mamba track-mask host syncs out of the per-layer prefill path: #32589
  • [LFM2] Wire Lfm2MoeForCausalLM into the LFM2 serving override tables: #30780
  • Fix LongCat n-gram token-table crashes on padded batches: #31312
  • fix(qwen3.5): restrict MoE weights to local PP layers: #32022
  • [minimax-m3] fp8 attention GEMMs on SM100 (fp8_e4m3 KV + trtllm_mha): #30971
  • Prewarm DSV4 MHC post kernel at model load: #30741
  • fix mqa preshuffle layout issue for deepseek v4: #31563
  • [DeepSeek-V4] Fix nvcc 13 crash building the topk_v2 kernel: #32910
  • [RL] DSV4: dispatch indexer topk_transform_512 through DSATopKBackend: #31087
  • [RL] DSV4: add env to quantize SWA KV cache from bf16-rounded values: #31086
  • [RL] Support FlashInfer TRT-LLM NVFP4 MoE in the RL weight checker: #31085
  • [DLLM] vectorized joint/low-confidence decoding and skip redundant attn init: #21094
  • [Bugfix] Fix Cohere2MoeConfig import crash from huggingface_hub @strict: #31769

Kernel Library (sglang.kernels, RFC #29630)

  • [Kernel] RFC #29630 finale: retire sglang.jit_kernel into sglang.kernels: #32072
  • [Kernel] Move sgl-kernel under sglang.kernels.aot: #32648
  • [Kernel] Phase 3+4: move JIT infra + operator groups into sglang.kernels (RFC #29630): #31666
  • [Kernel] Phase 4 batch-2: migrate JIT operator groups into kernels.ops (no shims) (RFC #29630): #32015
  • [Kernel] Phase 4 batch-3: migrate tangled JIT subsystems + new groups into kernels.ops (RFC #29630): #32045
  • [Kernel] Sweep missed dedicated kernels into kernels.ops (moe/quant siblings + dspark) (RFC #29630): #32160
  • [Kernel] Classification cleanup: unify jit naming, drop empty/model groups, add elementwise (RFC #29630): #32148
  • [Kernel] Reclassify kernel tests by ops group + move helpers out of the package (RFC #29630): #32128
  • Delete sgl-kernel AOT bmm_fp8, use flashinfer.bmm_fp8: #31202
  • Delete sgl-kernel AOT router GEMM and fused A GEMM: #30280

SGLang-Diffusion

  • [diffusion] model: support minimax-h3: #33275
  • [diffusion] FLUX.2 VAE decoder fast path behind quality=high (H200: 1024^2 97.6->29.2 ms, 2048^2 437.2->168.5 ms): #33451
  • [diffusion] Fuse DiT FFN tanh-GELU into up-proj GEMM (cublasLt epilogue) behind quality=high (Qwen-Image 1024^2 denoise 12.36 -> 12.05 s on H200): #33536
  • [diffusion] feat: add dynamic cuDNN SDPA attention backend: #30090
  • [diffusion] feat: support resident layers for DiT: #31538
  • [diffusion] feat: unify encoder folding and batch data-parallel encoding: #30211
  • feat(diffusion): add regional torch compile: #32696
  • perf(diffusion): decode Wan VAE in BF16: #32697
  • [diffusion] optimization: accelerate CUDA video output finalization: #32784
  • [diffusion] optimization: support cuda-ipc zero-staging all-to-all for 2-rank Ulysses: #31854
  • [Diffusion] offload rollout weights to pinned host memory: #32032
  • [diffusion] fix: preserve tensor stride when offloading rollout weights to pinned host memory: #32420
  • [Diffusion] Return scheduler sigmas snapshot in rollout dit_trajectory: #32683
  • [diffusion] fix: reject ring parallelism where it would silently miscompute: #33353
  • fix(diffusion): size VSA top-k from padded blocks: #32695
  • [diffusion] fix: keep fused qk-norm-rope out of dynamo tracing: #31849
  • [diffusion] fix: fix dual-DiT models crash with (1,)-placeholder weights after compile-time offload: #32743
  • [diffusion] fix: per-shard FP8 scale shape for single-GPU fused linears: #32157
  • [diffusion] fix: fix local-path detection for MiniMax-H3 and other non-diffusers models: #33365
  • [diffusion] fix: don't self-kill diffusion worker when PID 1 is the real parent: #31361
  • [Diffusion] Fix component accuracy topology reuse: #33317
  • [vla] fix: pi05 models does not apply scale factor for language embeddings: #33367

Local & Desktop AI

  • feat: SM120 (Blackwell Desktop) support for GLM-5.1 inference: #26928
  • Add FP4 Indexer for DeepSeek V4 on SM120: #27059
  • Enable GPT-OSS FlashInfer MXFP4 on SM120: #32668
  • Change the FP8 per-tensor GEMM backend on SM120 to cuBLAS: #31961
  • [SM120] Only split touched SWA pages in FlashMLA page-split kernel: #32320
  • [DSV4/SM120] Allow fused MHC opt-in with standalone TileLang pre disabled: #30954
  • Sm120 scatter fallback: #31669
  • [MLX] Fix overlap-loop request bookkeeping and graceful shutdown: #32447
  • [MLX] Size request capacity by attention DP: #32115
  • [MLX] Handle configs without quant_method in Humming: #31460
  • [diffusion] fix: fix diffusion output stability on mps: #30017

New recipes: RTX 5090 DeepSeek-V4, DGX Spark for Inkling-Small, MiniMax-H3 on 2x RTX 5090 (TP2 + layerwise offload).

AMD / ROCm

  • [AMD] Add fused all-reduce RMSNorm per-group quant for Qwen3.5 FP8: #24651
  • [AMD][MXFP4] Reland "Online MXFP4 quantization 2/N - FP8 to MXFP4 requantization on AMD GPUs": #28291
  • [AMD] Cache AITER expert mask across decode: #31889
  • [AMD] Derive AITER verify tokens-per-req from input shape: #31221
  • [AMD] Speed up DSV4 MoE weight loading from mmap views: #32315
  • [AMD] Fix DeepSeek-V4 FP4 MoE expert memory bloat: #31450
  • [AMD] DSv4: bring HIP compress-state pool into the memory_saver KV_CACHE region: #31747
  • [AMD] Fix DeepSeek-V4 fused-RMS FP8 scale metadata on gfx950: #31727
  • [AMD] [Fix] Enable aiter hd256 FP8 prefill FMHA on gfx950: #33399
  • [AMD]Qwen3.5 integration gfx950 fmha fp8 hd256: #32046
  • [Fix][AMD] Qwen3.5 MoE: disable global-slot shared-expert fusion under per-rank EP backends (MoRI + dp-attention init crash): #31793
  • [AMD] MiniMax-M3: opt-in custom/quick all-reduce on ROCm: #32230
  • [AMD] Minimax-M3 : unblock mxfp8 block convert on gfx950: #32036
  • [AMD] Enable mamba JIT transfer kernel on ROCm (fix transfer_kv_mamba NameError): #31741
  • [AMD] add Gemma3RMSNorm.forward_hip to unbreak ROCm: #32613
  • [AMD][Fix] Restore aiter-padded MoE weight dims for serialized checkpoints: #33090
  • [AMD] Fix JIT compile failure in sgl_kernel/warp.cuh: #33195

NPU / Ascend

  • [NPU] Optimize DeepSeek-V4 performance: #31931
  • ✨ [llm][npu][quant] Add W8A8 MXFP8 quantization for Qwen3 MoE on Ascend NPU: #30768
  • [NPU] adapt dflash v2 on npu: #31739
  • [NPU] Adapt MiMo-V2.5-W8A8: #29131
  • [NPU] Enable automatic ascend_attn selection for vision attention and graph runners: #31948
  • [NPU] ascend fuseep use moe ep group: #32040
  • [NPU] Fix MTP IndexShare warm-up for attention DP and prefill CP: #32210
  • [NPU] fix dsv4 mtp condition on NPU graph: #32711
  • [NPU] [FIX] Fix performance degradation of Qwen3.5-397B-A17B: #32130
  • [Bugfix] [NPU] Fix w4a8 MoE performance degradation: #32113
  • [NPU] Fix LLaDA2 MoE OOM after the FRACTAL_NZ cast, re-enabling the NZ speedup: #31772
  • [NPU] Determine the topk norm_type through scoring_func: #31393
  • [NPU] Acc fix for afmoe model introduced by topk refactor.: #31280
  • [NPU] bugfix for extra device memory on Ascend: #30112
  • [NPU][Fix Issue]: Send expert weights contiguous tensor across cards during EPLB rebalance: #32001
  • [NPU]Add Ascend transfer version compatibility.: #31189
  • [NPU] Update non-vit vision part for cumulative seqlen: #31867
  • 🐛 [llm][npu][quant] Fix ModelSlim MXFP4 packed weight loading: #32013
  • add fill_draft_extend_prepare_buffers_native for NPU: #32427
  • sgl-kernel-npu tag update to 2026.7.27: #32596

CPU / Intel / XPU

  • [Intel GPU] Add XPU Platform support: #31949
  • [XPU] Enable breakable prefill CUDA graph on XPU: #30273
  • [XPU][GDN] add XPU path for causal_conv1d_fn and causal_conv1d_update: #31250
  • [Intel GPU] calculate free memory based on allocated memory for XPU: #32044
  • [sgl-kernel][CPU] add kernel for shm_allgather_into_tensor and shm_reduce_scatter_tensor: #13397
  • [CPU] refactor rope kernels: #31897
  • [CPU] Fix mxfp4 padding size: #31334
  • [Bugfix] Fix CUDA import on non-CUDA platforms: #31919

Dependencies

  • Bump FlashInfer to 0.6.15.post1: #31927 ⭐
  • Bump sgl-deep-gemm to 0.1.5: #32345
  • Replace Kimi K3 DeepGEMM patch with 0.1.5.post1: #33143
  • Bump helion version to 1.4: #32562
  • chore: bump mooncake version to 0.3.12.post1: #32302
  • bump dynamo-tokenizers to 1.7.0: #32981
  • [AMD] ROCm AITER pin reverted to 9127c94, then updated to d9e5ef7: #32879, #32939
  • Fix PyPI release: drop the git-only sgl-eval dep from packaged metadata: #32354

Breaking Changes & Upgrade Notes

  • helion jumps 0.2.6 to 1.4, a major-version move for anyone depending on helion-backed kernels: #32562
  • sglang.jit_kernel is retired into sglang.kernels, completing RFC #29630. Imports from the old module path must move: #32072, #31666, #32015, #32045
  • Breakable prefill CUDA graph is now on by default for DP attention: #31682
  • CuteDSL BF16 GEMM on SM100 is on by default when the heuristic allows it: #30567
  • sgl-kernel AOT bmm_fp8 is deleted in favor of flashinfer.bmm_fp8; the AOT router GEMM and fused A GEMM are also removed: #31202, #30280
  • The FP8 per-tensor GEMM backend on SM120 switches to cuBLAS: #31961
  • Unfetchable or unparseable multimodal inputs now return HTTP 400 instead of 500, and text-only models reject media input outright: #31417, #32914
  • Diffusion request-level quality is restricted to two validated tiers, lossless (default) and high: #33453
  • ServerArgs is being made read-only behind namespace config bags. Code that mutated ServerArgs at runtime must route through the new accessors: #31809, #31811, #33334, #33338

Known Issues

  • The FlashInfer rmsnorm + quant fusion for SM90/SM100/SM120 was landed and reverted this cycle (#32994, #33455). It is not in this release.
  • Breakable CUDA graph is disabled for NemotronH pending a fix: #33130
  • test_update_weights_from_disk is skipped on ROCm pending a reload fix (#31924): #31925
  • The SM120 DeepGEMM release-pipeline test is temporarily skipped: #32193

Full Changelog: v0.5.16...v0.5.17

New Contributors

Full Changelog: v0.5.16...v0.5.17

Contributors

ormandj, vroomfondel, and 44 other contributors
Assets 2
Loading