Releases · vllm-project/vllm-ascend

Release list

v0.23.0 Latest

Latest

yiz-liu released this 16 Aug 22:18

v0.23.0

5cb98ca

v0.23.0 - 2026.08.16

We're excited to announce the official vLLM Ascend v0.23.0 release, aligned with upstream vLLM v0.23.0. This note summarizes the cumulative user-facing changes since the previous official release, v0.18.0, including the v0.19.1rc1, v0.20.2rc1, v0.21.0rc1, v0.22.1rc1, and v0.23.0rc1 development cycles. Please follow the official documentation to get started.

PR references marked with † were merged into the v0.23.0 release branch after v0.23.0rc1.

Highlights

Ascend 950 and DeepSeek V4: Added end-to-end DeepSeek V4 support on Ascend 950, including DSA attention, MTP, piecewise graph execution, distributed inference, sparse attention, CPU binding, and MXFP quantization and communication paths. #9757 #9935 #10236 #11014
Model and hardware coverage: Expanded support and deployment guidance for GLM-5.2, GLM-4.7-Flash, Qwen3.5/Qwen3.6, Qwen3-ASR, Qwen3-Omni, Bailing MoE, Gemma4, Step3, and MiniMax 2.x models across A2, A3, Ascend 950, and Atlas 300I DUO. GLM-5.2 supports long-sequence inference up to 1M tokens on Atlas 800 A3. #8657 #9560 #10441 #10697 #11091 #11264 #12115
Context parallelism and sparse attention: Added SFA DCP with a replicated indexer, compact KV gather, C8 support, and device-side metadata paths for long-context and P/D-disaggregated deployments. #9638 #9809 #11819 #11871 #11981
KV-cache lifecycle and offload: Added hybrid/Mamba attention prefix caching; CPU and SSD offload in AscendStore that covers all backends. #8743 #9533 #9731 #10393
Graph and speculative execution: Added FULL_AND_PIECEWISE graph mode, which is enabled by default and requires no manual configuration; DFlash FULL_DECODE_ONLY; zero-bubble async scheduling; P-Eagle and PARD; and expanded MTP/Eagle3 support. #7640 #8118 #9572 #10042 #10566

Features

Added multimodal DFlash, FlashComm support for Qwen VL/MoE models, and PCP-aware multimodal reasoning. #7486 #7897 #8038 #9340
Added HCCL weight transfer for reinforcement-learning workloads and D2D NetLoader support for speculative draft models. #9152 #9893
Expanded Model Runner V2 with initial MoE and Eagle support. #7885 #7922
Expanded EPLB with additional observability and dynamic load-balancer examples. #9536 #10627
Extended C8 INT8 KV cache to sparse-attention paths with packed layouts and a DCP replicated indexer, and added W8A8FP8 and W4A16 MXFP quantization paths for Ascend 950. #10236 #11014 #11846 #11871

Hardware and Operator Support

Added and optimized recurrent GDN, causal Conv1D, sparse-attention, LightningIndexer, compressor, and fused quantization operators. #7798 #7926 #9382 #9491 #9825 #10730
Expanded Atlas 300I DUO support for Qwen3.5, Qwen3.6, Qwen3-ASR, Qwen3-VL, quantized MoE paths, MTP, and graph execution. #7674 #7725 #10309 #12115 #13262†
Added Python 3.12 support and moved release images to Python 3.12. #9558

Performance

Unless stated otherwise, these optimizations are selected automatically for the targeted path and need no additional configuration.

Replaced npu_fusion_attention with _npu_flash_attention_unpad for supported A2/A3 attention workloads. It is selected automatically; no manual setting is needed. #8671
Avoided projecting unused tail KV tokens during MLA prefill with PCP. Enable PCP with --prefill-context-parallel-size; the optimization then applies automatically. #8787
Reduced scheduler issuance bubbles for workloads using asynchronous scheduling. Enable it with --async-scheduling. #8766
Added zero-bubble scheduling for asynchronous speculative decoding. Enable it with --async-scheduling together with a speculative decoding configuration. #7640
Batched KV-cache offload copies with aclrtMemcpyBatchAsync for CPU-offload workloads. Configure KV cache CPU offload as documented; batching is automatic within that path. #7819
Reduced PCP/DCP KV-cache all-gather traffic by selecting the required blocks before communication. Enable PCP or DCP with --prefill-context-parallel-size or --decode-context-parallel-size; no separate optimization switch is needed. #8050
Optimized split_qkv_tp_rmsnorm_rope kernels for supported quantized model paths. Kernel selection is automatic; no manual setting is needed. #8059 #9830
Removed prefill host-device synchronization in Qwen3-Next and Qwen3.5 paths. It applies automatically to those models. #7967
Reduced SFA prefill KV all-gather communication for PCP/DCP. Enable the corresponding context-parallel mode; the optimized communication path is automatic. #8043
Added a Triton penalty kernel for requests using repetition, frequency, or presence penalties. It is selected automatically when penalties are requested. #7569
Optimized Model Runner V2 temperature and top-k log-softmax kernels. They are selected automatically for sampling workloads on Model Runner V2. #8083
Optimized the Model Runner V2 min-p kernel. It applies automatically when min-p sampling is requested. #8243 #7767
Added a Model Runner V2 Triton kernel for bad-word filtering. It applies automatically when bad_words is supplied. #8030
Optimized the Model Runner V2 bincount kernel. It is selected automatically for sampling paths that require token counts. #7757
Optimized the Model Runner V2 ranks kernel. It is selected automatically for the corresponding sampling path. #7767
Reduced avoidable Triton recompilation caused by runtime function parameters. The cache-friendly path is automatic; no manual setting is needed. #7481 #7483
Reused equivalent HCCL process groups to reduce distributed initialization overhead. Reuse is automatic for matching groups. #7654
Deferred CPU binding until worker warmup completes to avoid interfering with initialization. CPU binding is enabled by default on supported ARM servers; no manual setting is needed unless it was explicitly dis...

Assets 2

v0.23.0rc1 Pre-release

Pre-release

yiz-liu released this 19 Jul 13:55

v0.23.0rc1

f4a08bd

v0.23.0rc1 - 2026.07.20

We're excited to announce v0.23.0rc1, the first release candidate for the vLLM Ascend v0.23.0 release line. This release aligns the plugin with upstream vLLM v0.23.0 and expands model, context-parallel, KV-cache offload, and Ascend 950 support. Please follow the official documentation to get started.

Highlights

Expanded model support: Added GLM-5.2 support on A2 and A3, and Ascend 310P support for Qwen3-ASR-1.7B, Qwen3.5, and Qwen3.6. #10441 #11264 #10257 #12115
Sparse attention and context parallelism: Added SFA DCP with a replicated indexer, compact KV gather, and C8 support. #11819 #11981 #11846 #11871
KV-cache lifecycle and offload: Added recompute KV-cache offload for P/D decoder nodes, AscendStore coordination, and layerwise KV Pooling with a Memcache backend. #10742 #10393 #11585
Ascend 950 quantization and communication: Added W4A16 MXFP4, all-gather EP MXFP4, and low-accuracy token-dispatch paths. #11014 #11287 #11718 #11766

Features

Added DeepSeek V4 MTP graph support. #11062
Added Virtual Width Network Eagle3 and Eagle3 support with chunked pipeline parallelism. #10042 #10566

Experimental Features or Optimizations

Added experimental Step3P7 and Step3P5 support, including Step3P5 MTP. #10697
Added experimental Gemma4 support on A2 and Ascend 950. #11091 #10643
Improved the DeepSeek V4 prefix-cache hit rate. #11107

Performance

Optimized SFA DSA-CP output merge with All-to-All communication and PCP FlashAttention restore/output merge. #12137 #11842
Avoided H2D synchronization in context-parallel speculative decoding metadata and snapshotted query locations before asynchronous H2D copies. #11862 #12071
Parallelized KV-cache receive with a thread pool and enabled asynchronous all-gather for DSA-CP output-projection TP weights. #10548 #10694
Vectorized local sequence-length computation in SFA metadata. #11816

Stability and Bug Fixes

Fixed GLM-5.1 IndexCache weight loading and a GLM-4.7-Flash IndexError on the first request with MTP and layerwise MemCache. #11363 #11829
Fixed Qwen3.5 GDN accuracy regressions across PCP, MTP, and DCP graph replay, including a mixed-length PCP out-of-bounds crash, while restoring the previous model-runner dispatch behavior. #11195 #11893 #12027 #12283
Fixed Qwen3.5 speculative-decoding accuracy, garbled output, and out-of-bounds failures on Ascend 310P with MTP/EAGLE and full-graph execution. #11337 #11408 #11920
Fixed Qwen MoE routing overflow and shared-expert gate matrix-multiplication failures on Ascend 310P. #11391 #11730
Fixed Qwen3-Omni ModelSlim W8A8 checkpoint loading failures caused by mismatched weight names and unquantized embedding metadata. #12321
Fixed Qwen3-VL rotary-embedding copy races on Ascend 310P and restored the device-specific VisionTransformer patch. #11679 #12132
Fixed the DeepSeek-R1-0528 W8A8 shared-expert no-clamp accuracy path without regressing the clamped DeepSeek V4 path. #11775
Fixed DeepSeek V4 Flash W4A8-MXFP4 all-gather EP inference on Ascend 950 by preserving routing-weight precision. #11498 #11663 #11718
Fixed malformed streamed tool-call arguments and TP8+EP startup compatibility for MiniMax-M2 and MiniMax-M2.5. #11505
Fixed silent prefix-cache output corruption and block-table overflow for Qwen3-Next, Qwen3.5, and other hybrid Mamba models using MTP/EAGLE, plus a 310P Mamba align-postprocess hang. #11353 #11659 #12038
Fixed Mooncake KV-transfer grouping for Kimi-K2.7 Code with Kimi-K2.5-DFlash when P/D nodes use unequal TP sizes and target/draft models have different global KV-head counts. #11887
Fixed the AscendStore parent-block hash chain when a KV block group is only partially missing. #12252
Disabled shared-expert multistream overlap when fused MC2 is enabled to avoid an unsupported configuration. #12245
Fixed DCP/DP service hangs and restricted the recompute scheduler to decode nodes. #12034 #11490
Delayed AscendStore initialization until the first real decode request. #11673
Fixed low MTP acceptance rates for SFA with DSA-CP and multiple speculative tokens. #10878

Dependencies

Upstream vLLM: v0.23.0.
Python: >= 3.10, < 3.13.
CANN: 9.0.1 for A2, A3, and Ascend 950; refer to the 310P installation guide for its platform-specific CANN package.
PyTorch / torch_npu: 2.10.0 / 2.10.0.post2.
Triton Ascend: 3.2.1.
Mooncake: 0.3.11.post1 in the release images.

Ready to Deprecate

The following features and optimizations are planned for deprecation in a future release:

Layer sharding.
FlashComm2.
The FlashComm3 multistream-overlap gate.
Hamming sparse.
Asynchronous exponential overlap.
Matmul all-reduce and matmul all-reduce RMSNorm fusions.
Weight prefetch.
Dynamic-batch SLO.
KV offload in KV Pool.
Fused MC2 mode 2 (enable_fused_mc2=2).
Paged attention and pa_shape_list.
Selected plugin environment variables; their configuration will be migrated to equivalent --additional-config options.

Known Issues

The combination of pipeline parallelism (PP) and prefill context parallelism (PCP) is not supported in v0.23.0. Support for this combination is deferred to a later release.
The former enable_sparse_c8 option has been split into enable_sparse_sfa_c8 and enable_sparse_li_c8. Existing --additional-config settings must use one or both new options depending on whether Sparse Flash Attention C8, LightningIndexer C8, or both are required. #12351
The load-balance proxy can swallow decode errors and return an empty HTTP 200 response. #12166
Qwen3-30B-A3B floating-point serving can show a 1-2 ms TPOT regression at batch size 1 in the reported TP4 full-graph configuration. #12337
In the reported DeepSeek V4 Flash W8A8 MTP P/D-disaggregated deployment, the second aisbench round can cause a worker process from another card to appear on an NPU device. #12338
On Ascend 950, Qwen3.5-397B-W8A8-MXFP8-FULL_QUANT in a P/D-disaggregated deployment without MTP can alternate between correct and incorrect outputs. #12339
DeepSeek V4 Pro on A3 and A5 can show continuously increasing memory usage in bo...

Assets 2

v0.22.1rc1 Pre-release

Pre-release

MengqingCao released this 30 Jun 13:45

v0.22.1rc1

5f6faa0

We're excited to announce the release of v0.22.1rc1 for vLLM Ascend. This is the first release candidate for the v0.22.1 release line, building on v0.21.0rc1 and aligning the plugin with upstream vLLM v0.22.1. Please follow the official doc to get started.

Highlights

Mooncake Connector for DeepSeek V4 / Hybrid KV Cache: Mooncake connector now supports DeepSeek V4 and hybrid KV cache disaggregated prefill scenarios with correct block stride handling, compressed KV transfer calculation, and hybrid Mamba token alignment. #10342
HCCL Weight Transfer for RL Workloads: Added an HCCL-based weight transfer backend for Ascend NPU so trainer and inference workers can synchronize weights in RL pipelines without a CUDA/NCCL dependency. #9152
Ascend 950 Expansion: Extended Ascend 950 support with W8A8/W4A8 dynamic quantization and platform-specific CPU binding support. #10236 #10483

Features

Added multimodal input support for DFlash workloads. #9340
P-Eagle and PARD are now stable parallel speculative decoding methods and have passed validation testing.
Added KV consumer partial-group caching for hybrid Mamba models. #10009
Added MiniMax M2 C8 cache-scale support in GQA load_weights. #10461
[Experimental] Added SSD support for multiple DP ranks on the same machine to avoid local-rank path collisions in Mooncake offload directories. #10477

Hardware and Operator Support

Added W8A8/W4A8 dynamic quantization support for Ascend 950. #10236
Added Ascend 950 CPU binding support for Ascend 950 server topology and process layout. #10483

Performance

Optimized split_qkv_tp_rmsnorm_rope with grid-stride loading and host-side reciprocal precomputation; the PR reports about a 5x kernel speedup on the tested MiniMax-M2.5 W8A8 QuaRot prefill workload. #9830
Reused prebuilt chunk host metadata for Ascend chunk ops to reduce host-device synchronization overhead on Qwen3.5 workloads. #9310
Skipped compute_slot_mapping for Mamba groups to reduce unnecessary work in hybrid cache paths. #10492
Enabled multistream DSV4 DSA overlap and removed redundant DSA v1 code paths. #10518

Documentation

Refreshed the context parallel, EPLB, and speculative decoding documentation. #10332
Added Kimi 2.6 and GLM5.2 documentation. #9969 #10544

Known Issues

MiniMax 2.7 dual-node 16-card deployments may hang or crash after 10-20 minutes under load. #10591
Llama LoRA can still hit an einsum tensor-dimension mismatch on Ascend. #10577
Qwen3.x with PD disaggregation plus MTP could still show precision issues because former KVCache blocks may remain dirty. #10961
In A3 four-machine 2P1D deployments, Kimi-K2.6 can trigger Error in KVCacheTransferThread. error=unhashable type: 'list' on the D node under concurrent terminal-bench2 testing. #10962
With CANN 9.0.0, GLM5.1 1P1D four-machine deployments may hang during 140K-context performance tests, and Kimi-K2.5 with MC2 enabled may hit OOM on single-node A3. #10963
Multi-level pooling remains an experimental feature and still has known issues, including DeepSeek-V4-Flash startup failures with Layerwise masks and service hangs in some Mooncake SSD scenarios. #10964

New Contributors

@Lin-Qingyang-Alec made their first contribution in #8799
@Maybe2191 made their first contribution in #8706
@ChefWu551 made their first contribution in #8398
@AlanisZomeg made their first contribution in #9018
@ccc000-cell made their first contribution in #8928
@vvaen made their first contribution in #8508
@quancs made their first contribution in #9049
@ZeroFadeAway made their first contribution in #9218
@mccube2000 made their first contribution in #9298
@yuhongming-2026 made their first contribution in #7886
@lihaofei-2026 made their first contribution in #8537
@panther-zhu made their first contribution in #9389
@nanxingMy made their first contribution in #9381
@Shelleyaaa made their first contribution in #9433
@MosCloud made their first contribution in #9450
@zhao-stack made their first contribution in #9238
@internel-error made their first contribution in #9616
@ZRICHARD9527 made their first contribution in #9201
@CXY-Katrina made their first contribution in #9525
@xszbuaa made their first contribution in #9344
@ningjingbengxiaohai made their first contribution in #9692
@Biuapha made their first contribution in #9765
@jyoung6652 made their first contribution in #9189
@KaiMa-endeavour made their first contribution in #9663
@ztzx3156 made their first contribution in #9721
@Bybbbb11 made their first contribution in #9382
@1-Y-C made their first contribution in #9601
@yilunh998 made their first contribution in #9893
@nomewang made their first contribution in #9757
@cfq0 made their first contribution in #9638
@ljy19911228 made their first contribution in #9481
@muziyuhui666 made their first contribution in #9972
@Fishermanykx made their first contribution in #9908
@nofushanquan made their first contribution in #9835
@zouzy5137 made their first contribution in #10004
@goodgoodname made their first contribution in #10032
@robellliu-dev made their first contribution in #9801
@baolongsun made their first contribution in #10266
@evan-ai-arg made their first contribution in #9476
@luobicangqiong made their first contribution in #10248
@2416602906 made their first contribution in #10178
@lHrHenry233 made their first contribution in #10009
@He1pa made their first contribution in #9830

Full Changelog: v0.19.1rc1...v0.22.1rc1

Contributors

ztzx3156, ljy19911228, and 41 other contributors

Assets 2

v0.21.0rc1 Pre-release

Pre-release

realliujiaxu released this 16 Jun 14:56

v0.21.0rc1

80610e4

We're excited to announce the release of v0.21.0rc1 for vLLM Ascend. This is the first release candidate for the v0.21.0 release line, building on v0.20.2rc1. Please follow the official doc to get started.

Highlights

DeepSeek-V4 for Ascend 950: Full end-to-end support for DeepSeek-V4 on Ascend 950, including piecewise graph mode, DSA attention, KV cache management, and MTP. #9757 #9935
Hybrid & Mamba Align Prefix Cache: New alignment-based prefix caching mechanism for Hybrid and Mamba architectures, improving cache hit rates across related sequences. #9533
FULL_AND_PIECEWISE Graph Mode: Introduced a hybrid graph compilation mode combining full-graph and piecewise strategies. Requires HDK 25.5.1+ / CANN 8.5.0+ to remove the old stream-budget limitation, enabling up to ~32K graphs on A3 and ~64K on Ascend 950. #9572 #9962
Python 3.12 Support: Dockerfiles and setup.py now officially support Python 3.12, and all base images have been upgraded from py3.11 to py3.12. #9558

Features

Added end-to-end support for DeepSeek-V4 on Ascend 950, including piecewise graph mode, DSA attention backend, KV cache management, distributed inference (with PP fixes), and MTP. #9757 #9473 #9935
Added Hybrid & Mamba Align Prefix Cache for improved prefix cache reuse in Hybrid and Mamba architectures. #9533
Added layerwise KV cache event callbacks for finer per-layer observability and control. #9468
Added GLM4.7-Flash model support with Flash Attention backend. #9560
Added FULL_AND_PIECEWISE graph mode, a hybrid compilation strategy mixing full-graph and piecewise approaches. Requires HDK 25.5.1+ / CANN 8.5.0+ to remove the old stream-budget limitation, enabling significantly more graph captures — approximately 32K on A3 and 64K on Ascend 950. Legacy capture-size pruning has been cleaned up accordingly. #9572 #9962
Added W4A8 MXFP4 quantization support for Ascend 950. #8265
Added MXFP8 FlashCommV3 support on Ascend 950. #9671
Added NZ layout support for W4A8 MoE compressed tensors and C8 quantization (GQA). #9625 #9721
Added Mooncake Connector hybrid PCP/DCP support for QWen3.5. #9809
Added D2D NetLoader weight loading for draft models in speculative decoding. #9893
Added Mooncake Connector hybrid attention support. #8850
Added Mooncake KV pool usage optimization. #7820
Added KV Pool support for loading failure block IDs without hybrid recompute. #9701
Added NPU storage metadata debug helpers for improved troubleshooting. #9189
Added torch reserved/allocated memory profiling in execute_model(). #9765
Added EPLB experts hotness metrics and EPLB time consumption data exposure. #9536
Added group_name parameter when creating HCCL config for better group management. #9667
Enabled prefix caching with PCP/DCP, allowing KV cache reuse across prefill and decode in disaggregated deployments. #9638
Added simple yet general CPU KV Cache Offloading support. #8743
Added Mooncake SSD offload with embedded client for large-scale KV cache storage. #9731
Re-added code start compilation caching for npugraph_ex (previously reverted), improving warmup time. #9914
Added ACL graph memory estimation before KV cache allocation to prevent OOM during graph capture. #9865
Added DeepSeek-V4 compressor block size [32,64,128] support to improve automatic prefix cache hit rate. #10354
Added batch_invariant_ops setup for reinforcement learning scenarios. #10034
Adapted load balance proxy example to shared scheduler workers. #9645
[310P] Added Qwen3.5 MTP and graph mode support. #10309

Hardware and Operator Support

Added custom GDN operator support for Ascend 950 with a new fused GDN gating AscendC operator (fused_gdn_gating). #9382 #9601
Added A2/A3 and Ascend 950 compressor operator paths. #9350
Adapted GDN and Conv1D operators for the Ascend 950 platform. #9224
Added Ascend 950 Dockerfiles and disaggregated PD endpoint configuration documentation. #9723 #9690
Removed unused MC2 prefill custom ops to streamline the operator surface. #9919
Added Sparse Flash Attention support on Ascend 950 devices. #9825
Added LightningIndexer and SparseFlashAttention ACLNN ops for improved sparse attention performance. #9491
Added Rehash for AscendStore grouped keys to support DeepSeek V4 and compressed layouts. #9789

Performance

Optimized 310P MoE routing path for improved throughput. #9105
Added NZ format support for W4A8 MoE compressed tensors, delivering better memory access patterns. #9625
Added irregular mask build optimization for PCP/DCP with speculative decoding, improving efficiency. #9678
Reconstructed reduce sampling to eliminate patch behaviors and support both DFlash and MTP. #9735

Stability and Bug Fixes

Fixed speculative decoding MLA shape mismatch with Eagle3 and added DeepSeek V2 Eagle3 support. #9703
Fixed draft lm_head preservation for DFlash with reduced (draft-to-target) vocabulary. #9795
Fixed a draft model index-out-of-range error caused by token_indices_to_sample on Ascend 950. #9867
Added validation of DCP for draft models to catch configuration mismatches early. #9717
Fixed multiple DeepSeek V4 PP issues. #9473
Fixed DSA compressed idle dummy graph out-of-bounds issue. #9818
Fixed HMA support in AscendMultiConnector. #9782
Patched GLM47 inline zero-argument streaming tool calls. #9901
Patched GLM tool-call final chunks for correct streaming termination. #9787
Fixed empty tool_calls being emitted in OpenAI-format chat responses. #9791
Backported MiniMax M2 tool call streaming support. #9742
Repaired 310P Qwen3.5 ACLGraph precision. #9727
Fixed precision of the causal_conv1d_v310 operator on 310P. #9720
Fixed ACL dtype mapping table for correct dtype conversions. #9826
Chunked wq_b matmul to work around the NPU 65536 dimension limit. #9780
Optimized router experts in eager mode and fixed communication handling. #9728
Lazy initialization of KV store on put to avoid early resource allocation. [#9771](#9771...

Contributors

weixinAc, ztzx3156, and 47 other contributors

Assets 2

v0.20.2rc1 Pre-release

Pre-release

weijinqian0 released this 03 Jun 15:24

v0.20.2rc1

367b8e6

We're excited to announce the release of v0.20.2rc1 for vLLM Ascend. This is the first release candidate for the v0.20.2 release line. Please follow the official doc to get started.

Highlights

DeepSeek V4 Support: Added end-to-end support for DeepSeek V4, including the model architecture, DSA attention backend, KV cache management, distributed inference, tool-call parser, MTP support, KV Pool adaptation, and custom operator enablement. #9270 #9385 #9228
A5 and XLite Quantization Expansion: Added MXFP4 flatquant with row parallelism for Ascend A5 and expanded XLite support to GLM-4.7 W8A8 quantization. #9391 #9415

Features

Added Flash Attention 3 support for training-inference consistency. The backend is ready in vLLM Ascend and will become directly usable once the FA3 package is publicly available. #9060
Added DeepSeek PCP/DCP adaptation to improve support for disaggregated deployments. #9058
Added a dedicated additional_config.enable_dsa_cp switch to decouple DSA-CP from FC1. DSA-CP now requires both FC1 and DSA-CP to be explicitly enabled, allowing FC1 to stay enabled while DSA-CP is disabled when needed. #9878
Added merged graph support for DFlash workloads. #9074
Added LoRA support for Qwen3.5 dense models. #9023
Added KV pool adaptation for DeepSeek V4 and separated MTP-layer KV cache sharding for DeepSeek V4 speculative decoding. #9385 #9367

Hardware and Operator Support

Added DeepSeek V4 custom operators required for the new model path, registered the operators for Ascend 910B, and switched the DeepSeek V4 hc_pre path to a fused operator. #9228 #9339 #9396
Enabled MXFP4 flatquant and row parallel support on Ascend A5. #9391
Enabled MC2 dispatch and combine support for MXFP4/MXFP8 quantization on Ascend A5. #9365 #9328
Improved 310P support by optimizing fused operators for Qwen3.5 Dense ACLGraph and simplifying the 310P RMSNormGated path. #9104 #9489

Performance

Added DeepSeek V4 DSA multistream overlap optimizations across compressor, indexer-select, CV parallel, and pure-prefill compute-communication overlap paths. #9450 #9441 #9433 #9504
Reused DSA topk_indices across decode steps with IndexCache to reduce repeated DeepSeek V4 index computation. #9390
Fixed the missing enablement for cv_indexer_qkv_prepare multistream parallelism in the new overlap path. #9530
Reduced host-device synchronization overhead by removing the sync point in PIECEWISE mode. #9025
Optimized shared expert overlap timing in FusedMoE. #9413
[Experimental] Added reduce sampling with enable_reduce_sample to lower Tensor Parallel communication overhead in distributed greedy, top-k/top-p, and rejection sampling paths. #8308

Stability and Bug Fixes

Fixed DeepSeek V4 MTP, serial inference, FlashComm, A2 tensor-output all-reduce, and P/D disaggregation KV cache edge cases. #9456 #9487 #9488 #9389 #9500
Fixed DeepSeek V4 hc_pre behavior and added a 4-card E2E regression test. #9452

Dependencies

Upgraded the matched upstream vLLM baseline to v0.20.2. #9270
Upgraded CANN to 9.0.0 and triton-ascend to 3.2.1. #9085
Upgraded PyTorch and torch-npu to 2.10.0. #9128

Breaking Changes and Migration Notes

Migrated a set of runtime options from environment variables to AscendConfig, including the FC1/FlashComm1 switch from VLLM_ASCEND_ENABLE_FLASHCOMM1 to additional_config.enable_flashcomm1. Please review configuration code and deployment manifests when upgrading. #9064
Disabled SwiGLU clamp by default, which may slightly change behavior for workloads that previously relied on the old default. #9438

Documentation

Refreshed deployment and feature documentation for the current main branch release line. #9309 #8968
Added documentation for the enable_dsa_cp additional configuration option for DeepSeek V3.2 and GLM5. #9910

Known Issues

GLM5/GLM5.1 W4A8 deployments have known issues in some advanced configurations. #9395
Qwen3.6-35B-A3B may shut down when MTP/speculative decoding is enabled, with numAcceptedTokens[0]=4 exceeds varlen segment length=3 reported during shape/dtype processing. #9956
GLM-5.1 can hang on the P node in 200K long-sequence 1P1D agent workloads after long-running service, with MoeDistributeDispatchV2/aclnnMoeDistributeDispatchV4 reporting an AICore timeout. #9958
GLM5 W4A8 deployments can see a significantly lower speculative decoding acceptance rate when MTP3 is used together with FlashComm. #9803
MiniMax-M2.7 W8A8/QuaRot can show lower-than-expected GPQA accuracy in long-sequence deployments when PCP/DCP is combined with Eagle3 speculative decoding. #9959
KV Pool feature for DeepSeek V4 now faces several known issues affects user-friendliness and performance, including special startup parameter requirements, special key storing behaviors, etc. For details, please refer to issue #9975.

Assets 2

v0.19.1rc1 Pre-release

Pre-release

MengqingCao released this 30 Apr 10:48

v0.19.1rc1

da421af

This is the first release candidate of v0.19.1 for vLLM Ascend, based on vLLM v0.19.1. This release includes significant performance optimizations, new model support, hardware expansion, and important bug fixes.

Please follow the official doc to get started.

Highlights

DFlash Attention Backend: Added DFlash attention backend with FULL_DECODE_ONLY support for improved inference performance (#8118, #8516, #8627)
Zero Bubble Async Scheduling: Implemented zero bubble optimization for async scheduling and speculative decoding, significantly reducing scheduling overhead (#7640)
A2/A3 Attention Operator Upgrade: Replaced npu_fusion_attention with _npu_flash_attention_unpad operator for better performance on A2 and A3 hardware (#8671)
Eagle3 + MiniMax-M2.5 Support: Applied Eagle3 speculative decoding to MiniMax-M2.5 model for faster inference (#7619)
C8 INT8 KV Cache for GQA: Added C8 (INT8 KV cache) support for GQA attention models, including DeepSeek-V3.1 with PD disaggregation (#7474, #7222)
Bailing Model Support: Full support for Bailing MoE model including linear adaptation and ModelSlim quantization (#8657, #8709)

Features

Flash Comm V1 for Qwen3-VL: Support Flash Comm V1 for Qwen3-VL multimodal models (#7897)
Eagle + PCP + Full Graph Mode: Support Eagle combined with PCP and full graph mode (#7924)
Multimodal Reasoning with PCP: Support multimodal reasoning when prefill context parallel feature is enabled (#8038)
Dynamic Chunk for PP: Support Dynamic Chunk for Chunked Pipeline Parallelism (#7896)
Hamming-based Sparse Attention: Added Hamming-based sparse attention inference framework and operators (#8564, #8346)
Optimized Causal Conv1d Operator: Added optimized causal conv1d operator (#8215)
Recurrent AscendC Operators: Added recurrent AscendC operators for specific model architectures (#8055)
GLM4.7 C8 Support: Support GLM4.7 with C8 (INT8 KV cache) scenarios (#8174)
Minitron-8B-Base Support: Verified and supported nvidia/Minitron-8B-Base model (#8157)
Bailing Model Support: Full support for Bailing MoE model with linear adaptation and ModelSlim quantization configuration (#8657, #8709)
Qwen3.5 MoE Flash Comm: Support Flash Comm for Qwen3.5 MoE models (#7486)
Initial MoE Support for MRv2: Add initial MoE models support for Model Runner V2 (#7922)
Xlite Backend Expansion:
- XLite GLM-4.7 support (#7935)
- Support Qwen3VLMoeForConditionalGeneration in xlite backend (#8046)
EPLB Enhancements:
- Swift balancer policy supports mix placement (#8035)
- EPLB adaptation to multimodal models (#7743)
Eagle Improvements for model_runner_v2:
- Fixed Eagle's acceptance rate problem in graph mode (#8365)
- Fixed Eagle's precision problems (#8230, #8033)
- Adapted Eagle for model_runner_v2 (#7885)
MTP Merged Graph: Support merged graph for MTP (Multi-Token Prediction) (#6860)
Unified MoE Expert Placement: Support unified placement for shared & router experts (#7188)
Dispatch V2 Hierarchy Communication: Support dispatch_v2/combine_v2 hierarchy communication for better MoE performance (#7583)
Xmask for Dispatch FFN Combine: Add xmask feature for dispatch_ffn_combine operator (w8a8 branch) (#8560)
Fused W4A8 Kernel: Fuse W4A8 dispatch + FFN + combine into a single fused kernel (#7779)
KV Cache Memory Accounting: Account for graph capture memory in KV cache planning (#8289)
Qwen3-Next Hybrid Attention: Support Qwen3-next hybrid attention in piecewise & full_decode_only modes (#7422)
GDN Optimization: Optimize GDN non-spec prefill fallback metadata (#7756)
Qwen3-VL Support: Support kv_rmsnorm_mrope for Qwen3-VL (#7762)
Mamba Prefix Caching: Layerwise connector supports Mamba prefill prefix caching (#7814)
Yuanrong KV Pool Backend: Add Yuanrong backend support to KV Pool (#6869)

Hardware and Operator Support

310P Enhancements:
- Qwen3.5 model adaptation synchronized with main (#8009)
- Support W8A8 dynamic linear method (#7725)
- Support shared experts path in fused MoE for Qwen3.5 (#7674)
- Add npu_causal_conv1d_310 AscendC Custom Op (#7798)
- Add recurrent_gated_delta_rule_310 AscendC Custom Op (#7926)

Performance

A2/A3 Attention: Replace npu_fusion_attention with _npu_flash_attention_unpad operator for better performance (#8671)
MLA PCP Prefill Optimization: Optimize MLA PCP prefill attention by avoiding projecting unnecessary tail KV tokens (#8787)
Async Scheduling Optimization:
- Asynchronous scheduling issuance bubble optimization (#8766)
- Zero bubble async scheduling and spec decoding (#7640)
KV Cache Optimization:
- Batch KV cache offloading via aclrtMemcpyBatchAsync (#7819)
- Optimize KV cache gathering by selecting blocks before all-gather (#8050)
Operator Optimizations:
- Optimize split_qkv_tp_rmsnorm_rope ops (#8059)
- Optimize host-device sync problem in prefill phase for Qwen3Next/Qwen3.5 (#7967)
- Reduce prefill KV all-gather communication for PCP/DCP (SFA) (#8043)
- Add penalty-related Triton kernel for better performance of penalties (#7569)
Triton Kernel Optimizations (model_runner_v2):
- Optimize _temperature_kernel and _topk_log_softmax_kernel (#8083)
- Optimize _min_p_kernel performance (#8243, #7767)
- Add bad-words-kernel triton kernel (#8030)
- Optimize bincount_kernel performance (#7757)
- Optimize _ranks_kernel performance (#7767)
- Optimize triton recompilation triggered by function parameters (#7480, #7481, #7483)
HCCL Process Group Reuse: Reuse equivalent HCCL process groups on Ascend (#7654)
CPU Binding Defer: Defer CPU binding until worker warmup completes (#7829)
Conv3d to Linear Conversion: Convert conv3d to linear when kernel size equ...

Contributors

idouba, triomino, and 33 other contributors

Assets 2

v0.18.0

zzzzwwjj released this 30 Apr 10:48

v0.18.0

e18643f

We're excited to announce the release of v0.18.0 for vLLM Ascend. This is the official release for v0.18.0. Please follow the official doc to get started.

Highlights

Model Support

Kimi-K2.x Model Support: [Experimental]Added support for Kimi-K2.x models. @aipaes @dragondream-chen @SparrowMu @LoganJane #6755
Minimax-m2.x Model Support: [Experimental]Added support for Minimax-m2.x models with eagle3. @SparrowMu @GDzhu01 #7105 #7714
GLM5 Support: [Experimental]Added support for GLM5 models without any code modification!
Qwen3.x Support: [Experimental]Added support for Qwen3.x models without any code modification!
DeepseekOCR Support: [Experimental]Added support for DeepseekOCR model and optimize RelPosAttention and CustomQwen2Decoder. @Wangbei25 #7737

Core Features

EPLB (Expert Parallelism Load Balance): EPLB is more stable with many bug fixes, and has better performance now. EPLB now works in most cases and is recommended for use. #6528 #7344 #7890 #6477
ACLGraph Enhancement: ACLGraph now support capturing a single merged graph for multi-step drafts, which greatly reduce host bound in multi-step spec decoding case! #5553 #5940
KV Pooling: Enhanced KV pool with Mooncake connector now support sparse attention, and LMCacheAscendConnector is added as a new KV cache pooling solution for Ascend, and support FabricMem Mode for HIXL interconnect, support yuanrong as a backend for AscendStoreConnector, and now MooncakeLayerwiseConnector can be activated together with KV Pooling. Compared with previous versions, KV Pooling has a huge performance optimization on TTFT! #6339 #6882 #6806 #6869 #7032
PD disaggregation: Mooncake layerwise connector now support hybrid attention manager and PCP feature. #7022 #6627
NPU Graph EX (npugraph_ex) Enabled by Default: The npugraph_ex feature is now enabled by default, providing better graph optimization with integrated inductor pass and MatmulAllReduceAddRMSNorm fusion. #6354 #6664 #6006
RL(Reinforcement learning): [Experimental]RL enhanced with implemented batch invariant feature with AscendC and triton op, and added routing replay feature. #6590 #6696
CPU Binding Enabled by Default: Enabled ARM-only CPU binding with global-slicing A3 policy, improving inference throughput in hostbound scenarios. #6686

Features

Prefix cache is now supported in hybrid model. #7103
Flash Comm V1 now supports VL models with MLA, removing a previous limitation for multimodal serving. #7390
VL MoE models now support SP, and sp_threshold is removed in favor of sp_min_token_num from vLLM. #7044
[Experimental]Pipeline Parallel now supports async scheduling, improving throughput for PP deployments. #7136
Eagle3 now supports QuaRot quantization without embedding. #7038
Refactoring eagle3/mtp, eagle3 and mtp are now using the same proposer. #6349 #7033

Hardware and Operator Support

First time support 310P, with huge performance optimization!:
- support W8A8 quantization. #6641 #6454
- support weightNZ with quant and unquant case. #6705
- support W8A8SC quantization. #7075
- fix post-sampling not working in graph mode. #8077
- Added addrmsnorm support for 300I DUO. #6704
- Fix ngram graph replay accuracy error on 310P. #7134
Custom Operators: Added multiple custom operators including:
- Added AscendC casual_conv1d_fn operator for Qwen3-Next. #6661
- Added Ascend Ops recurrent_gated_delta_rule operator. #6725
- Added GMM custom operator for MoE models. #7010
- Optimize split_qkv_rmsnorm_rope operator. #6827
- Triton rope now supports index_selecting from cos_sin_cache. #5450
- Added AscendC fused op transpose_kv_cache_by_block to speed up GQA transfer. #6366
- Optimized DispatchFFNCombine kernel performance and resolved vector error caused by unaligned UB access. #6468 #6707
- Refactor and optimize CausalConv1d. #7495

Performance

Initialize Performance: Optimized Triton operator recompilation to reduce redundant rebuilds and unnecessary recompilation triggered by function parameter optimization. #7647 #7645
Qwen3.x Performance: [Experimental]Optimized the Qwen3.x and Qwen3-Next performance by supporting full graph mode, PD disaggregation, mamba prefill prefix-caching and flashcomm1, prebuilding chunk metadata to reducing host-device synchronization overhead, and multiple op performance optimization including chunk_gated_delta_rule, chunk_fwd_kernel_o, solve_tril, recompute_w_u_fwd_kernel, split_qkv_rmsnorm_mrope, etc. @LoganJane @shaopeng-666 @ppppeng @SunnyLee151064 @hust17yixuan @Toneymiller @linfeng-yuan #7487 #6830 #7506 #7796 #7527 #7529 #7495 #7368
Kimi-K2.x Performance: [Experimental]Optimized the Kimi-K2.x performance by supporting eagle3 and flashcomm1, and reducing d2h overhead. @aipaes @dragondream-chen @SparrowMu @LoganJane @GDzhu01 @Yaphets24 @hust17yixuan #7342 #7390 #7521
Qwen3-VL Performance: Qwen3-VL gets stronger multimodal operator enablement with Flash Comm V1 and qkv_rmsnorm_mrope support, and enable 2.7x faster for convolution computation with aclnn BatchMatMulV2, support EAGLE speculative decoding. #7893 #7852 #7017 #6327
Qwen3-Omni Performance: Qwen3-Omni quantization adaptation and optimization is now available. #6828
DeepSeek-V3.2/GLM5 Performance: Performance optimizations, support W8A8C8 quantization, and optimized KV cache usage. @yydyzr @ZYang6263 @rjg-lyh @Nagisa125 #7029 #6610
GLM4.7-Flash Performance: Added W8A8 quantization support for GLM4.7-Flash. @aipaes #6492

Dependencies

vLLM: Upgraded to 0.18.0 and dropped 0.17.0 support.
CANN: Upgraded to 8.5.1. PS: AscendStoreConnector with FabricMem mode, 310P device supporting and Qwen3-Omni model need upgrades CANN version to 9.0.0, if you need these features, please upgrade manually.
torch-npu: Upgraded to 2.9.0.post1+git4c901a4 because of some known issue. This version can't install by default, pleas...

Contributors

LoganJane, Yaphets24, and 16 other contributors

Assets 2

v0.13.0rc3 Pre-release

Pre-release

wangxiyuan released this 24 Apr 03:30

v0.13.0rc3

bbe9bc0

What's Changed

[Doc][Misc] Update release notes and FAQ links for v0.13.0 by @wangxiyuan in #6585
[BugFix][v0.13.0] fix a bug that patch from PR #5786 does not take effect by @Angazenn in #6615
[v0.13.0][Ops] Make triton rope support index_selecting from cos_sin_cache by @Angazenn in #6602
[0.13.0][bugfix]fix profiler initialization bug with calling stack by @linfeng-yuan in #6714
[0.13.0] modify release note & supported matrix by @zzzzwwjj in #6751
[v0.13.0][Fusion]add checks to skip fusion where split_rmsnorm_rope is not supported by @Angazenn in #6749
[DOC] add request forwarding (cherry-pick from #6780) by @starmountain1997 in #6788
[Bugfix] Fix vllm-ascend 0.13.0 error: TypeError: apply_token_bitmask_inplace_cpu(): incompatible function arguments by @wjunLu in #6823
[Bugfix] mtp forces eager mode by @zhenwenqi2024 in #6760
[DOC] add layer_sharding and fix link by @starmountain1997 in #6808
[v0.13.0][CI] Upgrade to CANN 8.5.1 by @wxsIcey in #6865
[Bugfix] Resolve operator name collision for DeepSeekV3.2 in RL scena… by @Mind-s in #7034
[doc] Added Ascend PyTorch Profiler section by @herizhen in #6905
[0.13.0][cherry-pick][Bugfix][csrc] Add compile-time Ascend950/910_95 compatibility for custom ops between CANN8.5 and 9.0 by @zjchenn in #7116
[0.13.0][cherry-pick][Bugfix][Triton] Centralize Ascend extension op dispatch in triton_utils by @zjchenn in #7112
[Doc][Misc][v0.13.0] Updated the document configuration for DeepSeek-V3.2 by @Nagisa125 in #7957
[CI] Fix Releases/v0.13.0 CI tests by @wjunLu in #7952
[BugFix]Fix compilation errors for operators dispatch_gmm_combine_decode/moe_combine_normal/moe_dispatch_normal by @wangyibo1005 in #7840
[v0.13.0][Feature] Add DeepSeek v4 initial support by @wangxiyuan in #8648

New Contributors

@Mind-s made their first contribution in #7034

Full Changelog: v0.13.0...v0.13.0rc3

Contributors

wangxiyuan, zzzzwwjj, and 11 other contributors

Assets 2

v0.18.0rc1 Pre-release

Pre-release

yiz-liu released this 01 Apr 15:30

v0.18.0rc1

99e1ea0

This is the first release candidate of v0.18.0 for vLLM Ascend. Please follow the official doc to get started.

Highlights

C8(INT8 KV cache) is now supported for DeepSeek-V3.1 with PD disaggregation scenario. #7222
DeepSeek models are now supported on A5 through new MLA operators. #7232

Features

Flash Comm V1 now supports VL models with MLA, removing a previous limitation for multimodal serving. #7390
Support separate attention backends for target and draft models in speculative decoding, allowing finer backend tuning per model. #7342
VL MoE models now support SP, and sp_threshold is removed in favor of sp_min_token_num from vLLM. #7044
Qwen VL models now support w8a8_mxfp8 quantization. #7417

Performance

Optimized Triton operator recompilation to reduce redundant rebuilds and unnecessary recompilation triggered by function parameter optimization. #7647 #7645
Optimized the Qwen3.5 and Qwen3-Next GDN prefill path by prebuilding chunk metadata, reducing host-device synchronization overhead. #7487
Simplified the FIA prefill context merge path for better runtime efficiency. #7293

Documentation

Refreshed deployment and model docs for Kimi-K2.5, GLM-4.7, DeepSeek-V3.2, MiniMax-M2.5, and PD disaggregation guides. #7371 #7403 #7292 #7296 #7300

Others

Fixed a PD separation issue where decode nodes could get stuck because shapes were not aligned across DP nodes. #7534
Fixed a regression where hybrid attention plus mamba models on Ascend could start with an incorrect block size after the v0.18.0 upgrade. #7528
Fixed multi-instance serving OOM calculation on single-card deployments. #7427
Fixed DeepSeek v3.1 C8 when overlaying MTP with full decode and full graph modes. #7571
Fixed quantization config key mapping in AscendModelSlimConfig by switching from reverse mapping to forward mapping. #7716

Dependencies

To address issues triggered by multi-stream parallel operations within ACL Graph, we have integrated temporary dependency versions for torch_npu. These fixes are already included in our official Docker images. If you prefer to build your own environment from source, please manually install the specific versions as follows:

# Set environment variables
PYTHON_TAG=$(python3 -c "import sys; print(f'cp{sys.version_info.major}{sys.version_info.minor}')")
ARCH=$(python3 -c "import platform; m=platform.machine().lower(); arch_map={'x86_64':'x86_64','amd64':'x86_64','aarch64':'aarch64','arm64':'aarch64'}; print(arch_map.get(m,m))")

# Select the specific torch_npu wheel based on your environment
if [ "$PYTHON_TAG" = "cp310" ] && [ "$ARCH" = "aarch64" ]; then PTA_WHEEL="torch_npu-2.9.0.post1%2Bgit4c901a4-${PYTHON_TAG}-${PYTHON_TAG}-manylinux_2_28_${ARCH}.whl"; \
elif [ "$PYTHON_TAG" = "cp311" ] && [ "$ARCH" = "x86_64" ]; then PTA_WHEEL="torch_npu-2.9.0.post1%2Bgitdc51c2d-${PYTHON_TAG}-${PYTHON_TAG}-manylinux_2_28_${ARCH}.whl"; \
elif [ "$PYTHON_TAG" = "cp310" ] && [ "$ARCH" = "x86_64" ]; then PTA_WHEEL="torch_npu-2.9.0.post1%2Bgita74051c-${PYTHON_TAG}-${PYTHON_TAG}-manylinux_2_28_${ARCH}.whl"; \
elif [ "$PYTHON_TAG" = "cp311" ] && [ "$ARCH" = "aarch64" ]; then PTA_WHEEL="torch_npu-2.9.0.post1%2Bgitee7ba04-${PYTHON_TAG}-${PYTHON_TAG}-manylinux_2_28_${ARCH}.whl"; \
else echo "Unsupported PYTHON_TAG=$PYTHON_TAG ARCH=$ARCH"; exit 1; fi

# Install wheels
python3 -m pip install "https://vllm-ascend.obs.cn-north-4.myhuaweicloud.com/vllm-ascend/${PTA_WHEEL}"

There is an known issue on the current triton-ascend, as shown in #7782 Please upgrade triton-ascend to 3.2.0.dev20260322 to avoid this issue, please use the official docker images or manually install the specific triton-ascend version as following:

PYTHON_TAG=$(python3 -c "import sys; print(f'cp{sys.version_info.major}{sys.version_info.minor}')") && \
ARCH=$(python3 -c "import platform; machine = platform.machine().lower(); arch_map = {'x86_64': 'x86_64', 'amd64': 'x86_64', 'aarch64': 'aarch64', 'arm64': 'aarch64'}; print(arch_map.get(machine, machine))") && \
TRITON_ASCEND_WHEEL="triton_ascend-3.2.0.dev20260322-${PYTHON_TAG}-${PYTHON_TAG}-manylinux_2_27_${ARCH}.manylinux_2_28_${ARCH}.whl" && \
python3 -m pip install "https://vllm-ascend.obs.cn-north-4.myhuaweicloud.com/vllm-ascend/${TRITON_ASCEND_WHEEL}"

Known Issue

When running DeepSeek-R1 W8A8 with MTP and KV Pool enabled under high concurrency, a ValueError: Counters can only be incremented by non-negative amounts may occur. #7489
triton-ascend may fail to compile with a g++ internal compiler error (Segmentation fault). Workaround: update to triton-ascend==3.2.0.dev20260322 and clear the Triton cache (rm -rf ~/.triton/cache/*). #7782
FIA does not support all MHA head dimensions when using tp-size >= 16 on Ascend. Affected models will fail with an error on unsupported head dimensions. This will be resolved in a future release when FIA supports more head dimensions. #7729
While Minimax-2.5 now supports PD Disaggregation, internal testing has identified a 13% regression on the GPQA benchmark when this feature is enabled. We currently do not recommend enabling PD Disaggregation for this model and We are working on an optimization fix.

New Contributors

@GGGGua made their first contribution in #7295
@asunxiao made their first contribution in #7066
@liuhy1213-cell made their first contribution in #7300
@jiangmengyu18 made their first contribution in #7383
@ksiyuan made their first contribution in #7417
@yesyue-w made their first contribution in #7046
@lijiahang226 made their first contribution in #7232
@ZhuQi-seu made their first contribution in #7368
@GoMarck made their first contribution in #7392

Full Changelog: v0.17.0rc1...0.18.0rc1

Contributors

GGGGua, GoMarck, and 7 other contributors

Assets 2

v0.17.0rc1 Pre-release

Pre-release

MengqingCao released this 15 Mar 14:53

v0.17.0rc1

e20f0b1

This is the first release candidate of v0.17.0 for vLLM Ascend. Please follow the official doc to get started.

Highlights

Ascend950 chip is now supported. #7151
ACLGraph (graph mode) is now supported for Model Runner V2. #7110
Unified parallelized speculative decoding is supported, enabling parallel draft inference schemes simultaneously. #6766

Features

Auto-detect quantization format from model files, and remote model IDs (e.g., org/model-name) are also supported. --quantization ascend is not required now. #7111
Qwen3.5 is supported from this version on.
FlashLB algorithm for EPLB: supports per-step heat collection and multi-stage load balancing for better expert parallelism efficiency. #6477
LoRA with tensor parallel and --fully-sharded-loras is now fixed and working. #6650
LMCacheAscendConnector is added as a new KV cache pooling solution for Ascend. #6882
W8A8C8 quantization is now supported for DeepSeek-V3.2 and GLM5 in PD-mix scenario. #7029
[Experimental] Minimax-m2.5 model is now supported on Ascend NPU. #7105
[Experimental] Mooncake Layerwise Connector now supports hybrid attention manager with multiple KV cache groups. #7022
[Experimental] Prefix cache is now supported in hybrid model. #7103

Performance

Pipeline Parallel now supports async scheduling, improving throughput for PP deployments. #7136
Improved TTFT when using Mooncake connector by reducing log overhead. #6125
KV Pool lookup is optimized for short sequences (token length < block_size). #7146
Fix penalty ops in Model Runner V2, achieving ~10% performance improvement. #7013

Documentation

Added EPD (Encode-Prefill-Decode) documentation and load-balance proxy example. #6221
Added Ascend PyTorch Profiler usage guide. #7117
Fixed DSV3.1 PD configuration documentation. #7187

Others

Fix drafter crash in full graph mode for speculative decoding. #7158 #7148
Fix GLM5-W8A8 precision issues caused by rotary quant MTP weights. #7139
Fix ngram graph replay accuracy error on 310P. #7134
Fix FIA pad logic in graph mode after upstream vLLM change. #7144
Fix a precision issue caused by wrong KV cache reshape on Qwen3.5. #7209
Fix extra processes spawned on rank0 device. #7107
Graph capture failures now properly raise exceptions for easier debugging. #5644
Fix Qwen3.5 model by replacing torch_npu.npu_recurrent_gated_delta_rule by fused_recurrent_gated_delta_rule. #7109
Fix the bug when running Qwen3-Reranker-0.6B with LoRA. #7156

Known Issue

GLM5 requires transformers==5.2.0, and this will resolved by vllm-project/vllm#30566, will not included in v0.17.0.
There is a precision issue with Qwen3-Next due to the changed tp weight split method. Will fix it in next release.
The minimum number of tokens of prefix cache hit in hybrid model is large now. The exact number is related to tp size, e.g., with tp 2, the block_size is adjusted to 2048, which means that any prefix shorter than 2048 will never be cached.
GLM5 has an issue in the 2-node PD mixed deployment scenario where inference may hang when concurrency exceeds 8 (fixed in PR #7235 #7290).

New Contributors

@ppppeng made their first contribution in #7109
@SparrowMu made their first contribution in #7105
@tfhddd made their first contribution in #7127
@drizzlezyk made their first contribution in #7208
@chloroethylene made their first contribution in #6882
@bazingazhou233-hub made their first contribution in #7286

Full Changelog: v0.16.0rc1...v0.17.0rc1

Contributors

drizzlezyk, chloroethylene, and 4 other contributors

Assets 2

Uh oh!

Releases: vllm-project/vllm-ascend

Release list

v0.23.0

v0.23.0 - 2026.08.16

Highlights

Features

Hardware and Operator Support

Performance

Uh oh!

v0.23.0rc1

v0.23.0rc1 - 2026.07.20

Highlights

Features

Experimental Features or Optimizations

Performance

Stability and Bug Fixes

Dependencies

Ready to Deprecate

Known Issues

Uh oh!

v0.22.1rc1

Highlights

Features

Hardware and Operator Support

Performance

Documentation

Known Issues

New Contributors

Contributors

Uh oh!

v0.21.0rc1

Highlights

Features

Hardware and Operator Support

Performance

Stability and Bug Fixes

Contributors

Uh oh!

v0.20.2rc1

Highlights

Features

Hardware and Operator Support

Performance

Stability and Bug Fixes

Dependencies

Breaking Changes and Migration Notes

Documentation

Known Issues

Uh oh!

v0.19.1rc1

Highlights

Features

Hardware and Operator Support

Performance

Contributors

Uh oh!

v0.18.0

Highlights

Features

Hardware and Operator Support

Performance

Dependencies

Contributors

Uh oh!

v0.13.0rc3

What's Changed

New Contributors

Contributors

Uh oh!

v0.18.0rc1

Highlights

Features

Performance

Documentation

Others

Dependencies

Known Issue

New Contributors

Contributors