Skip to content

Releases: NVIDIA/TensorRT-LLM

Release list

v1.3.0rc24

v1.3.0rc24 Pre-release
Pre-release

Choose a tag to compare

View all tags
@tongyuantongyu tongyuantongyu released this 12 Aug 07:07
v1.3.0rc24
1cef02e
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

  • Known Issues

    • Workloads combining torch.compile with CUDA graphs can fail during initialization or inference with allocator assertions, missing outputs, or illegal memory accesses.
    • MLA models using MTP with multi-stream or piecewise CUDA graphs on SM120 can asynchronously crash with an illegal memory access.
    • Workloads using KVCacheManagerV2 may intermittently report a teardown failure after otherwise successful inference; this is observed with DeepSeek-V4-Flash.
    • Diffusion image and video pipelines can show output-quality regressions when conditioning defaults or fused normalization change numerical behavior; this is observed with Cosmos 3 Nano T2I/T2V and Wan 2.2 T2V.
    • Multi-GPU low-precision MoE models using fused residual normalization and all-reduce can suffer severe accuracy loss; observed examples include Nemotron V3 Ultra NVFP4 and GPT-OSS W4.
    • Qwen3 MoE models may fail during startup when using FP8 block-scale quantization on Blackwell.
    • Mistral Large-family NVFP4 configurations can show materially reduced GSM8K accuracy, as observed with Mistral Large 3 675B.
    • Multi-rank hybrid models using shared FlashInfer JIT artifacts can fail to start when ranks compile the same kernel concurrently.
    • High-concurrency disaggregated serving over Python NIXL with UCX 1.21 can crash during KV-cache transfer or request cancellation.
    • Hybrid recurrent-attention models using Mamba/GDN state caches can be rejected at startup because required live and dummy slots are underestimated; observed examples include Nemotron V3 Ultra NVFP4 and Qwen3-Next BF16.

  • Model Support

    • Add Kimi K3 with KDA kernels, optimized MoE, parsers, speculative decoding, and disaggregated serving (#17054, #17190, #17225, #17269, #17327, #17334)
    • Add MiniCPM-V 4.6 image and video support to the PyTorch backend (#15976)
    • Add Whisper support to the PyTorch backend (#16141)
    • Add Qwen-Image-Layered and Qwen-Image-Edit-2511 pipelines with CFG parallelism and TeaCache/Cache-DiT (#15096, #16095, #16384, #16339)
    • Add distilled Cosmos3 four-step text-to-image, image-to-video, and video-to-video generation (#16563, #16690, #16155)
    • Add the FastWan2.2 TI2V-5B three-step text-to-video pipeline (#16162)
    • Enable reference-image conditioning for FLUX.2 (#16644)
    • Enable dual-topology parallelism for LTX-2 two-stage generation (#16502)
    • Support mixed image, video, and audio inputs for Qwen2-VL and Qwen3-VL (#16337)
    • Enable multimodal encoder caching, side-stream integration, and partial cache hits for Qwen3.x and Gemma4 (#16817, #16662, #16554, #17231)
    • Add Gemma4 MTP assistant support (#15833)
    • Expand Qwen3, Qwen3.5, and Qwen3-Next compatibility with Qwen3.5-397B BF16/FP8 refit, safer loading, MoE/GDN/MTP fixes, and W4A8 execution (#16642, #16936, #17051, #17120, #17196)
    • Improve Mistral checkpoint, tokenizer, chat, and multimodal format handling (#15123)
    • Improve Nemotron-H quantized loading (#16833)
    • Handle EXAONE 4.5 33B memory constraints (#16992)

  • API

    • Expand TorchSampler with repetition, frequency, and presence penalties, min-p, no-repeat n-grams, and per-request seeds; reject NaN sampling values and raise top_logprobs to 100 (#16485, #16590, #16594, #17172, #17159, #16851)
    • Honor compatible generation_config.json sampling defaults through generation_config="auto" and --generation-config auto (#17213)
    • Add per-request priority to OpenAI chat and completion APIs (#15329)
    • Improve Qwen3 tool-call and DeepSeek-R1 reasoning-parser completion handling (#16866, #17157)
    • Configure disaggregated-serving HTTP keep-alive with server_keep_alive_timeout (#16430)
    • Protect internal disaggregated metadata with shared-secret request signing (BREAKING) (#16002)
    • Refactor multiprocess per-request metrics with header/SSE responses and JSONL output; deprecate perf_metrics_max_requests (BREAKING) (#16668)
    • Replace Mamba cache management with KVCacheManagerV2-backed snapshots and move controls under mamba_state_config (BREAKING) (#16598)
    • Move kv_cache_config.block_reuse_policy to block_reuse_config.policy and add max_num_turns (BREAKING) (#16883, #17277)
    • Add TriAttentionKvCacheCompressionConfig for TriAttention KV-cache compression (#16957)
    • Add advanced_sampling_mode controls for one-model MTP sampling (#16561)
    • Expose prototype DSA MTP indexer top-k sharing through index_share_for_mtp_iteration (#15806, #17149)
    • Enable encoder and mixed encoder-decoder CUDA graphs through new LLM arguments (#16706)
    • Add fp8_context_mla_kv_len_cap for context-MLA workspace and KV-capacity tuning (#16399)
    • Add the MEGAMOE_CUTEDSL backend option and disable deprecated WIDEEP selection (#16190, #17119)

  • Feature

    • Introduce C++ KVCacheManagerV2 with paged Vanilla attention and FlashInfer block reuse (#14047, #16714, #17106)
    • Add batched physical compaction for KV-cache compression (#16836)
    • Enable VisualGen tensor parallelism with Attn2D (#16677)
    • Accelerate VisualGen with FLUX.2 fused NVFP4 SwiGLU, Wan layernorm/shift-scale/quant fusion and temporal VAE batching, and Qwen-Image fast paths (#16143, #15762, #17002, #16142)
    • Enable Marlin NVFP4 on Ada Lovelace and W4A16 NVFP4 on SM120 (#16749, #16511)
    • Expand Blackwell kernels with CuTe DSL FP8/FP16 MLA decode attention and optimized fused MHC half-MMA (#15138, #16799)
    • Optimize sparse-attention top-k and indexing with CuTe DSL radix/GVR kernels, selective DSA K-cache allocation, and reduced prefill work (#15756, #16457, #16558, #16789)
    • Optimize MiniMax M3 with fused projections, MoE routing, QK-norm/RoPE/quantization, SwiGLU and AllReduce-residual-RMSNorm, leaner Index-K caching, and deduplicated eager plans (#16904, #16859, #16906, #17093, #16905, [#17091]...
Read more

Contributors

karljang, reasonsolo, and 115 other contributors
Loading

v1.3.0rc23

v1.3.0rc23 Pre-release
Pre-release

Choose a tag to compare

View all tags
@mikeiovine mikeiovine released this 31 Jul 18:55
v1.3.0rc23
d41ab33

  • Known Issues

    • Deepseek-V4-Pro can hang on GB300 disagg setups.
    • DeepSeek-R1 NVFP4 multi-GPU (PP4 + MTP) can crash with an MPI worker exit during execution (seen on GB300).
    • DeepSeek-V3-Lite BF16 + chunked prefill (Python scheduler) can hang / fail to complete.
    • Qwen MoE multi-LoRA (routed experts, varying ranks, eager) can fail during sampling.
    • Gemma3-1B FP8 prequantized with torch.compile can fail during CUDA graph capture (PyTorch CUDA allocator internal assert).
    • Qwen3.5-35B-A3B BF16 (CUTLASS, TP1 on A100) can hit CUDA OOM.

  • Model Support

    • Load DeepSeek V4 mixed-precision NVFP4 checkpoints (#16433)
    • Enable W4A8 checkpoint loading for Gemma4 K=V layers (#16797)

  • VisualGen

    • Implement uneven TP linear for VisualGen models (#14875)

  • API

    • Support multi-process HTTP frontends on the classic IPC executor path in trtllm-serve (#16523)
    • Emit initial KV cache stats at startup for external metric scrapers (#12596)

  • Feature

    • Default GPT-OSS and GLM-5 to the Python KV-cache transceiver (#16479, #16524)
    • Add MiniMax-M3 MSA sparse attention backend and migrate MiniMax M3 to loader v2 for TP8 support (#16291, #16468)
    • Support MiniMax M3 for disaggregated serving (#16017)
    • Support top_p_decay in the PyTorch TorchSampler (#16184)
    • Support GDN MTP replay (#16464)
    • Support Nemotron dynamic-tree MTP decoding (#15582)
    • Support rejection sampling under attention DP (including LM-head TP) (#16544)
    • Support MARLIN MoE with MTP and attention DP + EP (#16597)
    • Support fine-grained context chunk management (#16427)
    • Add TensorRT-LLM runtime integration for KV cache compression (#15697)
    • Add per-conversation KV cache block reuse (#16115)
    • Improve PyTorch encoder-decoder support and performance (#16158)
    • Optimize fused QKNormRoPE kernel (#16633)

  • Fix

    • Account for resume utilization in KV constraints and fix KV cache estimation capacity (#16484, #16545)
    • Make auto host-tier sizing rank-aware in KVCacheManagerV2 (#16106)
    • Fix DSpark all-reduce of draft MoE output under non-attention-DP TP (#16617)
    • Fix DSpark rolling-window slot collision in disaggregated serving (#16772)
    • Move Qwen-VL mRoPE seq-slot delta cache correctly (#16537)
    • Avoid attention workspace resize during CUDA graph capture (#16319)
    • Make FlashInfer sampling op wrappers opaque to Dynamo (#16732)
    • Allow trtllm-gen MoE autotuner when local_num_experts < top_k (#16653)
    • Use attention cache dtype for disaggregated transfer (#16505)
    • Fall back to disagg_request_id in Python NIXL decode receiver (#16720)
    • Fix MPI world heap corruption during teardown (#16550)
    • Stop thinking-budget processor from re-forcing the reasoning end tag (#16785)
    • Fix GPT-OSS router token identity (#16760)
    • Exclude ADP/CUDA-graph dummy requests from speculative-decode acceptance stats (#16571)
    • Use explicit errors instead of asserts for real-dataset input (#16257)

  • Documentation

    • Remove legacy Python relics and refresh docs after the TensorRT backend removal (#16612)

  • Benchmark

    • Support DeepSeek-V4 in layer-wise benchmarks (#16774)

  • Test & Infra

    • Assert MTP acceptance length in ADP + LM-head-TP accuracy tests (#16521)
    • Add DeepSeek R1/V3.2/V3-Lite disaggregated accuracy tests (#16344)
    • Update disaggregated GB200 test configs (#16481)
    • Raise BF16 Laguna DFlash memory skip threshold to prevent OOM (#16627)
    • Remove MiniMax-M2 TP16 multinode eval test case (#16542)
    • Add DeepSeek V4 Pro QA cases and DeepSeek-V4 B200/B300 single-node GPU tests (#16611, #16661)
    • Migrate Kimi disagg tests to Transceiver v2 and trim tests (#16482)
    • Split slow B300 attention unit tests (#16709)
    • Remove legacy TensorRT backend tests, examples, and CI plumbing (#16610)
    • Remove --trt_root and stop installing the TensorRT SDK into images (#16608)
    • Use main as default target branch in L0_MergeRequest_PR (#16480)
    • Preserve pytest progress with S3 capture (#16268)
    • Add support to run NGC container scanning in pre-merge (#15602)
    • Add per-stage option to cap or disable stage-level infra retries (#16551)
    • Require approval for PerfSanity wildcard runs (#16777)
    • Surface SLURM device faults to the failure classifier (#16557)
    • Update flashinfer-python from 0.6.14 to 0.6.15 (#16530)

What's Changed

  • [https://nvbugs/6245862][fix] Fix MoE EP divisibility check for EPLB configurations by @JacobHu-NV in #15410
  • [TRTLLMINF-127][infra] Upgrade dependencies for NGC PyTorch 26.05 stack by @EmmaQiaoCh in #15087
  • [None][test] Waive 7 failed cases for main in QA CI by @trtllm-agent in #16042
  • [None][feat] Add Laguna DFlash drafter support by @joerowell in #15666
  • [None][infra] Enable B300 stages by @yiqingy0 in #16359
  • [https://nvbugs/6428144][test] Unwaive GB300 DeepSeek-R1 gen-only disagg PerfSanity by @chienchunhung in #16267
  • [None][chore] Remove disagg-devs co-ownership of mla.py in CODEOWNERS by @Tabrizian in #16385
  • [TRTLLM-12721][fix] Add gated C++ NIXL in-flight cancellation and safe cleanup by @chienchunhung in #15238
  • [https://nvbugs/6374873][fix] Allow fp4 KV Cache + non-FP4 Mamba State by @VALLIS-NERIA in #16304
  • [https://nvbugs/6435642][fix] detect killed MPI executor workers by @chienchunhung in #16338
  • [None][test] Fix the multinode test case on DGX-Spark(perf skipped and func hang) by @JennyLiu-nv in #16209
  • [TRTLLM-14024][feat] Prune CuTe DSL NVFP4 GEMM autotuner tactics with nvMatmulHeuristics by @peaceh-nv in #15909
  • [TRTLLM-12373][feat] RMSNorm nvfp4 quant fusion for DS V3.2 / Kimi-K2.5 by @JunyiXu-nv in #14848
  • [None][test] Refine CBTS logic for handling -Perf- stages by @yufeiwu-nv in #16328
  • [None][infra] Assign KV cache manager v2 test ownership by @jiaganc in #16372
  • [None][fix] fix missing torch_dtype_to_binding by @bo-nv in #16401
  • [https://nvbugs/6426860][fix] Stabilize compressor BF16 tolerance by @mingyangHao in #16376
  • [https://nvbugs/6403909][test] Remove attention backend test waiver by @yuxianq in #16400
  • [None][chore] Rename AutoDeploy standalone package to Paragraf by @bmarimuthu-nv in #16324
  • [None][test] Split test_moe_backend TRTLLM by quant=None to avoid in-process IMA cascade by @leslie-fang25 in #16068
  • [None][fix] Fix Gemma4 illegal memory access when max_seq_len is at most the sliding window size by @Hudayday in #16099
  • [https://nvbugs/6396422][fix] Disable kv-cache reuse for MiniMax-M2 by @tcherckez-nvidia in #16383
  • [None][test] Waive 1 failed cases for main in QA CI by @trtllm-agent in #16412
  • [None][fix] align KV slice token_range.end with transferred block count by @chuangz0 in #15824
  • [None][feat] Python transceiver support cpp cache manager + offload by @chuangz0 in #15245
  • [None][chore] add disagg-devs as code owner for disaggregated tests by @Shixiaowei02 in #16415
  • [TRTLLM-13212][refactor] Unify sampler ops and clean up dead code in TorchSampler by @zhaoyangwang-nvidia in #16365
  • [https://nvbugs/6260897][fix] Relax line-589 assertion to accept either IN_PROGRESS or SUCCESS (FAILURE… by @tensorrt-cicd in #15121
  • [None][infra] Waive 2 failed cases for main in pre-merge 47854 by @trtllm-agent in #16423
  • [https://nvbugs/6445332][test] Remove test waivers for nvbug 6445332 (code fix merged in #16326) by @trtllm-agent in #16316
  • [None][infra] Tail Slurm job logs when job is no longer active. by @mzweilz in #15455
  • [None][chore] KVCacheManagerV2: Python and test preparation for a C++ backend by @lowsfer in #16218
  • [https://nvbugs/6283537][fix] Unwaive TestQwen3_5_4B::test_bf16 by @amukkara in #16392
  • [None][infra] Waive 1 failed cases for main in pre-merge 47597 by @trtllm-agent in #16361
  • [None][infra] Waive 1 failed cases for main in pre-merge 47847 by @trtllm-agent in #16443
  • [None][infra] Waive 1 failed cases for main in pre-merge 47847 by @trtllm-agent in #16442
  • [https://nvbugs/6269388][fix] Raise error to enforce HMAC encryption in IPC by @yibinl-nvidia in #16249
  • [None][infra] Waive 1 failed cases for main in pre-merge 47953 by @trtllm-agent in #16450
  • [None][infra] Waive 1 failed cases for main in pre-merge 47930 by @trtllm-agent in #16452
  • [TRTLLM-12352][feat] add post-transform capability profi...
Read more

Contributors

reasonsolo, mihai-chiorean, and 92 other contributors
Loading

v1.3.0rc22

v1.3.0rc22 Pre-release
Pre-release

Choose a tag to compare

View all tags
@mikeiovine mikeiovine released this 22 Jul 22:41
v1.3.0rc22
8ba9340

  • Known Issues

    • torch.compile crashes in the PyTorch compilation backend
      • Several multi-GPU accuracy paths fail with KeyError in the remove_copy pass when torch_compile=True (e.g. DeepSeek-V3-Lite bf16/FP8/NVFP4; Llama-3.1-8B-Instruct FP8 with FlashInfer). Workaround: disable torch.compile for affected configs DeepSeek-V3.2 FP8 block-scale OOM on H200.
    • test_fp8_blockscale configs can OOM on DGX H200 under default / disable_skip_indexer settings.
    • Mixtral FP8 MoE + multi-LoRA tactic mismatch
      • Routed-expert FP8 multi-LoRA can hit a Cutlass MoE assertion (GEMM2 finalize-fusion tactic selected while the runner is not configured for it), in both eager and CUDA-graph modes.
    • Kimi K2.5 disagg KV cache transfer can fail on GB300.

  • Model Support

    • Add DeepSeek-V4-Pro curated configurations (#15919)
    • Support mixed image and video modality requests for Qwen3-VL (#15852)

  • VisualGen

    • Add ParallelVAE_TrtllmWan for native Wan VAE support (#16066)
    • Optimize video hashing performance (#16340)
    • Fix Qwen Image CUDA graphs (#16391)

  • API

    • Add automatic per-model transceiver runtime selection, a breaking change (#16164)
    • Add bad_words support to TorchSampler (#16052)
    • Remove C++ modules for the legacy TensorRT backend, a breaking change (#16369)

  • Feature

    • Add Laguna DFlash drafter support (#15666)
    • Add DeepSeek DSpark drafter support (#15808)
    • Allow FP4 KV cache with non-FP4 Mamba state (#16304)
    • Extend rejection sampling to one-model speculative decoding modes (#15775)
    • Add SM121 MLA cache reuse support (#15434)
    • Integrate ModelExpress checkpoint loading (#15641)
    • Add a disaggregated coordinator and multi-process orchestrator fleet (#15905)

  • Fix

    • Fix MoE expert-parallel divisibility checks for EPLB configurations (#15410)
    • Fix Gemma 4 illegal memory access when the maximum sequence length does not exceed the sliding-window size (#16099)
    • Disable KV-cache reuse for MiniMax-M2 (#16383)
    • Avoid loading the multimodal encoder for text-only trtllm-bench runs (#16250)
    • Improve the disaggregated bounce-buffer workflow (#16116)
    • Clamp piecewise CUDA graph captures to the reachable ceiling (#16256)
    • Persist the DSA CuTe DSL paged-MQA-logits output buffer to prevent stale-pointer illegal memory access (#16307)
    • Fix double rotation and GPT-OSS pairing in unfused RoPE for YaRN models (#16378)
    • Honor GenerationResult timeouts (#16453)
    • Fix the greedy-sampling path when use_lm_head_tp_in_adp is enabled (#16440)
    • Fix DeepSeek V3 disaggregated conditional-test configuration (#16528)
    • Optimize Gemma 4 performance (#16509)
    • Derive disaggregated KV-transfer layer offsets from physical slot order (#16429)
    • Make one-model speculative-decoding attention-metadata save and restore exception-safe (#16382)
    • Fix DeepSeek V4 KV-cache warmup and related issues (#16466)
    • Fix intermittent CUDA mapping errors (#16226)
    • Remove unsupported INT8 choices from trtllm-bench --quantization (#16245)
    • Reduce per-step host preparation overhead in the PyTorch executor decode path (#16313)

  • Documentation

    • Add the DeepSeek V4 optimization technical blog (#16539)

  • Benchmark

    • Migrate Kimi performance-sanity tests to Transceiver v2 (#15966)
    • Add DeepSeek-V4-Pro performance-sanity cases on GB300 (#16540)
    • Increase input and output lengths in LLM performance tests (#16538)

  • Test & Infra

    • Upgrade dependencies for the NGC PyTorch 26.05 stack (#15087)
    • Enable B300 CI stages (#16359)
    • Fix the multi-node DGX Spark test (#16209)
    • Stabilize compressor BF16 tolerance checks (#16376)
    • Split unquantized MoE backend tests to avoid in-process IMA cascades (#16068)
    • Prepare Python and test support for the C++ KV cache manager v2 backend (#16218)
    • Improve function-level code coverage for subprocess data (#16229)
    • Upgrade NIXL to v1.3.1 (#16194)
    • Detect worker out-of-memory errors and crashes during disaggregated startup (#16456)
    • Add a --keep-workspace pytest option (#16451)
    • Promote disaggregated performance-sanity tests to pre-merge functional verification (#16351)
    • Remove 1K/1K performance-sanity cases from CI (#16535)

What's Changed

  • [https://nvbugs/6245862][fix] Fix MoE EP divisibility check for EPLB configurations by @JacobHu-NV in #15410
  • [TRTLLMINF-127][infra] Upgrade dependencies for NGC PyTorch 26.05 stack by @EmmaQiaoCh in #15087
  • [None][test] Waive 7 failed cases for main in QA CI by @trtllm-agent in #16042
  • [None][feat] Add Laguna DFlash drafter support by @joerowell in #15666
  • [None][infra] Enable B300 stages by @yiqingy0 in #16359
  • [https://nvbugs/6428144][test] Unwaive GB300 DeepSeek-R1 gen-only disagg PerfSanity by @chienchunhung in #16267
  • [None][chore] Remove disagg-devs co-ownership of mla.py in CODEOWNERS by @Tabrizian in #16385
  • [TRTLLM-12721][fix] Add gated C++ NIXL in-flight cancellation and safe cleanup by @chienchunhung in #15238
  • [https://nvbugs/6374873][fix] Allow fp4 KV Cache + non-FP4 Mamba State by @VALLIS-NERIA in #16304
  • [https://nvbugs/6435642][fix] detect killed MPI executor workers by @chienchunhung in #16338
  • [None][test] Fix the multinode test case on DGX-Spark(perf skipped and func hang) by @JennyLiu-nv in #16209
  • [TRTLLM-14024][feat] Prune CuTe DSL NVFP4 GEMM autotuner tactics with nvMatmulHeuristics by @peaceh-nv in #15909
  • [TRTLLM-12373][feat] RMSNorm nvfp4 quant fusion for DS V3.2 / Kimi-K2.5 by @JunyiXu-nv in #14848
  • [None][test] Refine CBTS logic for handling -Perf- stages by @yufeiwu-nv in #16328
  • [None][infra] Assign KV cache manager v2 test ownership by @jiaganc in #16372
  • [None][fix] fix missing torch_dtype_to_binding by @bo-nv in #16401
  • [https://nvbugs/6426860][fix] Stabilize compressor BF16 tolerance by @mingyangHao in #16376
  • [https://nvbugs/6403909][test] Remove attention backend test waiver by @yuxianq in #16400
  • [None][chore] Rename AutoDeploy standalone package to Paragraf by @bmarimuthu-nv in #16324
  • [None][test] Split test_moe_backend TRTLLM by quant=None to avoid in-process IMA cascade by @leslie-fang25 in #16068
  • [None][fix] Fix Gemma4 illegal memory access when max_seq_len is at most the sliding window size by @Hudayday in #16099
  • [https://nvbugs/6396422][fix] Disable kv-cache reuse for MiniMax-M2 by @tcherckez-nvidia in #16383
  • [None][test] Waive 1 failed cases for main in QA CI by @trtllm-agent in #16412
  • [None][fix] align KV slice token_range.end with transferred block count by @chuangz0 in #15824
  • [None][feat] Python transceiver support cpp cache manager + offload by @chuangz0 in #15245
  • [None][chore] add disagg-devs as code owner for disaggregated tests by @Shixiaowei02 in #16415
  • [TRTLLM-13212][refactor] Unify sampler ops and clean up dead code in TorchSampler by @zhaoyangwang-nvidia in #16365
  • [https://nvbugs/6260897][fix] Relax line-589 assertion to accept either IN_PROGRESS or SUCCESS (FAILURE… by @tensorrt-cicd in #15121
  • [None][infra] Waive 2 failed cases for main in pre-merge 47854 by @trtllm-agent in #16423
  • [https://nvbugs/6445332][test] Remove test waivers for nvbug 6445332 (code fix merged in #16326) by @trtllm-agent in #16316
  • [None][infra] Tail Slurm job logs when job is no longer active. by @mzweilz in #15455
  • [None][chore] KVCacheManagerV2: Python and test preparation for a C++ backend by @lowsfer in #16218
  • [https://nvbugs/6283537][fix] Unwaive TestQwen3_5_4B::test_bf16 by @amukkara in #16392
  • [None][infra] Waive 1 failed cases for main in pre-merge 47597 by @trtllm-agent in #16361
  • [None][infra] Waive 1 failed cases for main in pre-merge 47847 by @trtllm-agent in #16443
  • [None][infra] Waive 1 failed cases for main in pre-merge 47847 by @trtllm-agent in #16442
  • [https://nvbugs/6269388][fix] Raise error to enforce HMAC encryption in IPC by @yibinl-nvidia in #16249
  • [None][infra] Waive 1 failed cases for main in pre-merge 47953 by @trtllm-agent in #16450
  • [None][infra] Waive 1 failed cases for main in pre-merge 47930 by @trtllm-agent in #16452
  • [TRTLLM-12352][feat] add post-transform capability profiles by @chienchunhung in #16238
  • [https://nvbugs/6405760][fix] Do not loa...
Read more

Contributors

reasonsolo, chienchunhung, and 59 other contributors
Loading

v1.3.0rc21

v1.3.0rc21 Pre-release
Pre-release

Choose a tag to compare

View all tags
@mikeiovine mikeiovine released this 15 Jul 22:46
v1.3.0rc21
1662a87
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

  • Deprecation Notices

    • AutoDeploy backend is being deprecated. We are aware that earlier model support is a critical priority for many users of TensorRT LLM and are working on agentic approaches to improve time to functional model support in the PyTorch backend. As an early indicator, this was used to release Minimax M3 functional support within the first week of model release.

  • Known Issues

    • DeepSeek V3.2
      • Host KV cache offload can fail on multi-GPU setups: executor init may hit GPU OOM (H200), or hang when combined with MTP.
      • Multi-GPU NVFP4 configs on B300 show accuracy failures.
    • Disaggregated Serving
      • DeepSeek V3 Lite + Helix (H100/H20): disaggregated generation can produce incorrect output vs. expected strings.
      • DeepSeek V3 Lite auto-dtype paths are failing in disaggregated mode, including:
      • auto_dtype_with_helix + CUDA graphs on B200/B300.
      • auto_dtype with MTP on RTX PRO 6000 Blackwell.
    • DeepSeek V3 / R1 Family (Aggregated PyTorch)
      • DeepSeek V3 Lite with torch.compile + CUDA graphs + overlap scheduler can hit CUDA launch failures during warmup on H100.
      • NVFP4 on RTX PRO 6000 Blackwell can OOM under tested configs.
      • Some multi-GPU parallelism combos (PP4, TP2PP2) with MTP + attention DP remain unstable or unvalidated; several are tracked under open NVBugs (e.g. 6428094, 6428096).
    • Llama 3.x
      • Llama 3.1 8B accuracy tests fail with torch.compile enabled (TRTLLM and FlashInfer backends) — same class of CUDA errors as DeepSeek V3 Lite.
      • Llama 3.1 FP8 PP4 with FlashInfer on GB300 is failing accuracy validation.
      • Llama 3.3 70B NVFP4 TP2PP2 (no torch.compile, no GEMM allreduce fusion) is failing.
    • GPT-OSS
      • W4 4-GPU serving configs (v1 KV cache + DP4/CUTLASS, EP4/TRTLLM FP8) are not passing accuracy checks.
      • EAGLE3 speculative decoding with VSWA reuse on H100 can fail with sampling errors during generation.
    • MiniMax M3
      • MXFP8 + piecewise CUDA graphs at TP8/EP8 on B300 fails due to a KV cache manager API mismatch (num_blocks_per_seq argument).
    • Qwen 3.5
      • An FP4 agg test on 8xB200 is crashing witb a tensor pointer alignment error.

  • Model Support

    • Add DeepSeek V4 (DSv4) model, tokenizer, and integration coverage (#15414)
    • Add Cosmos3 reasoner-only support and Cosmos3 audio output support (#15117, #14827)
    • Support Minimax M3 MXFP8 and NVFP4 checkpoints (#15687, #15857)
    • Add Gemma 4 12B Unified (encoder-free multimodal) support (#15768)
    • Add support for Qwen3.5-VL MoE and Dense variants (#14599, #15249)
    • Add Qwen3.6 NVFP4 checkpoint support (#15703)

  • API

    • BREAKING: Remove Python modules and tests for the legacy TensorRT backend (#15918)
    • BREAKING: Move multimodal-related args and env vars (#15640)
    • BREAKING: Rename server args (#16091)
    • BREAKING: Rename acceptance-rate-based dynamic speculation fields (#12905)
    • Add native /v1/embeddings dynamic batching for encoder-only models (#15424)
    • Add native post-processing hook to trtllm-serve (#15631)
    • Add prefix-aware scheduling config flag to support opt-out (#15526)

  • Feature

    • Add DSv4 sparse MLA attention backend (#15409)
    • Optimize DSv4 follow-ups: autotuner updates, disagg routing, DeepGEMM and MegaMoE, sparse attention and model defaults (#15626, #15625, #15632, #15717)
    • Support inflight weight update (#14815)
    • Expand dynamic speculation to all spec decode algorithms (#12262)
    • Enable CUDA graph execution for PyTorch encoder-decoder models (#15637)
    • Support Tensor Parallelism for PyTorch encoder-decoder models (#15897)
    • Add Piecewise CUDA Graph support for Qwen3.5/Qwen3.6 MoE models (#15900)
    • Add low-latency host task dispatch mode for guided decoding (#15863)
    • Dispatch GDN MTP target-verify to the FlashInfer bf16 kernel (#15975)
    • Optimize trtllm-gen MoE routing (#15656)
    • Allow fp8 per-tensor base weights for MoE LoRA (#15528)
    • Add an opt-in raw-weight cache to the HF weight loader (#16054)
    • Add per-model KV cache manager v2 auto selection (#15823)
    • Add multi-rank sleep/wakeup support to the MPI executor path (#14636)
    • Add proxy fast-death detection and sticky EngineDeadError (#15816)
    • Add a cascade attention implementation up to 4.6x faster than MMHA (#14396)
    • Add fused kernels for Gemma4 serving and improve Gemma4 inference correctness and performance (#16074, #15848)
    • Improve Qwen3-VL preprocessing performance and normalized model_config weight mapping (#15598, #16353)
    • Enable agent serving evaluation via trace-replay on Scaffolding (#14397)
    • Centralize sampling logic and split backends into isolated modules (#15542)
    • Add Cute DSL GVR Top-K short-row optimization that removes cluster sync in run_one_row (#15835)
    • Enable VSA in VisualGen (#14280)
    • Add Wan VAE backend (#15555)
    • Enable CUDA graph capture with torch.compile in VisualGen (#15603)
    • Add LTX-2 tile-parallel VAE decode and parallelize LTX-2 LoRA weight loading (#15753, #13911)
    • Cache LTX2 merged LoRA weight (#14984)
    • Fuse LTX-2 Gate + Residual + Norm + AdaLN modulation (ShiftScale) + Quant kernels (#15102)

  • Fix

    • Fix output distribution correctness for Eagle3 dynamic-tree rejection sampling (#15098)
    • Make EAGLE functional on the v1 KV cache manager (#15708)
    • Remove redundant residual bound check in EAGLE3 hidden-state capture (#16165)
    • Correct speculative XQA attention sinks (#15739)
    • Add spec_metadata=None kwarg to SpecWorkerBase._apply_force_accepted_tokens (#15797)
    • Gate the py_last_draft_tokens snapshot in PyExecutor._prepare_draft_requests (#16121)
    • Fix bugs in Beam Search kernels (#15621)
    • Fix logits post processor for beam search in the PyTorch backend (#16010)
    • Prevent disaggregated KV transfer stalls and fix flaky hang for disagg gen-only (#15737, #16172)
    • Enable disagg partial reuse store for PP>1 (#15655)
    • Fix indexer-k-cache transfer for disagg (#16197)
    • Make cache transceiver transport reporting deterministic (#15893)
    • Pass dtype to AllReduce ctor to enable MNNVL all-reduce (#15547)
    • Use persistent per-stream workspace in cublas_mm for CUDA-graph safety (#15534)
    • Fix FMHA kernels not found for GPT-OSS + SM120 (#15230)
    • Add a runtime guard in FlashInferTrtllmGenAttention.is_supported (#15496)
    • Fix RoPE support in the flashinfer trtllm-gen backend (#15661)
    • Update trtllm-gen FMHA JIT libraries (#15974)
    • Fix TRTLLM-GEN backend multiCtasKv counter clear (#15761)
    • Sync CTA before PDL trigger in quantize_with_block_size (#14668)
    • Add EP assertion to DenseGEMMFusedMoE (#15451)
    • Cap MXFP4 Hopper swizzle transient GPU memory during MoE weight load (#16125)
    • Restrict FP8 rowwise handling for compressed-tensors: resolve config group by name and flatten per-channel weight_scale (#15415)
    • Fix GLM-5.1 NVFP4 fallback to AR-Norm fusion for unquantized dense layers (#15659)
    • Restrict MiniMax M3 dense SDPA backends and enable MiniMax M3 piecewise CUDA graphs (#15937, #15923)
    • Fix NT3 NVFP4 perf regression on Blackwell (#16031)
    • Fix Qwen2-VL Transformers 5 compatibility (#15997)
    • Fix Gemma4 MoE weight loading (#16108)
    • Honor Qwen-Image quant ignore list (#15599)
    • Fix MLA KV cache estimation sizing (#16311)
    • Size DSV4 KV constraint with num_extra_kv_tokens (#16173)
    • Source DSA metadata from sparse params and add a persistent topk-output buffer to avoid CUDA-graph stale-pointer IMA (#16236, #16290)
    • Fix fused mHC output reuse and extend compressor next_n (#16221)
    • Keep SSM cache in weights dtype when materializing (#16065)
    • Correct RocketKV KT cache byte accounting for FP8 KV cache (#16225)
    • Fix VSWA gate in KVCacheManager window-size resolution (#16233)
    • Reserve worst-case SWA slots to avoid single-request deadlock (#15588)
    • Add seq-slot pool overlap headroom with consistent slot-indexed buffer sizing (#16279)
    • Reduce per-request attribute overhead in the model_engine hot path (#15750)

  • Documentation

    • Remove legacy TensorRT docs and add a migration guide (#15767)
    • Add VisualGen engineering criteria (#15225)
    • Add VisualGen MGMN NVL72 tech blog and NVL72 video generation blog to README (#15420, #15858)
    • Clarify dtype='auto' resolution for LLM and KvCacheConfig (#15520)

  • Benchmark

    • Add in-process NeMo-Skills accuracy benchmarks (#15608)
    • Add Kimi-K2.5 disaggregated GSM8K accuracy test (#15617)
    • Add gpt-oss-120b eagle3 accuracy test (#14990)

  • Test & Infra

    • Upgrade dependencies for the dlfw 26.04 stack (#12643)
    • Upgrade NIXL to v1.3.0, aiperf to 0.8.0, ray to 2.55.1, and flashinfer-python to 0.6.14 (#15694, #15769, #15819, #15869)
    • Make FlashInfer a hard dependency for the Torch sampler (#16160)
    • Remove legacy TensorRT examples, TensorRT-backend tests, and the legacy TensorRT-engine Triton backend (#15763, #15810, #15907)
    • Retire Triton backend TRT workflow QA tests (#15814)
    • Add a model-derived PyTorch attention backend test suite (#15536)
    • Add test coverage for Eagle3ForCausalLM.apply_eagle3_fc (#16107)
    • Add KV cache manager v2 (V2 + VSWA) multi-GPU test coverage (#16114)
    • Add DSv4 coverage and import safety checks (#15710)
    • Add gemma and glm disagg python transceiver tests (#15755)
    • Add a CPU-only CI stage (part 1) (#15815)

What's Changed

Read more

Contributors

Superjomn, reasonsolo, and 115 other contributors
Loading

v1.3.0rc20

v1.3.0rc20 Pre-release
Pre-release

Choose a tag to compare

View all tags
@mikeiovine mikeiovine released this 30 Jun 01:11
v1.3.0rc20
c25c23f

This RC version will be the last one supporting the TensorRT backend, in the next version the TensorRT backend will be removed!

  • Known Issues

    • DeepSeek V3/V3.2 can crash with an illegal memory access or hang during warm up.
    • Autotuning for Qwen3-family models can crash with "Assertion failed: Failed to initialize cutlass TMA WS grouped gemm."

  • API

    • Add API to configure TeaCache coefficients (#13170)
    • BREAKING CHANGE: Make request chat_template opt-in (#14646)

  • Feature

    • Add DeepSeek V4 preparation (#15378, #15379, #15381, #15394, #15402, #15222)
    • Add MXFP8 weight format plus CUTLASS W8A8 Linear and MoE (#14962)
    • Add Marlin NVFP4 backend for MoE and Linear on Hopper (#13476)
    • Add CUDA graph wrapper for multimodal encoders (#14829)
    • Support cross-attention with FlashInfer TRT-LLM Gen kernels on Blackwell (#15429)
    • Support post-norm and per-aux fc_norm for Eagle3 draft models (Eagle 3.1) (#14988)
    • Add EPLB support for Qwen3.5 (#15543)
    • Optimize CuteDSL NVFP4 MoE grouped/SwiGLU GEMM accumulation pipeline (#15258)
    • Add CuTe DSL GVR-TopK load-balance optimization (#15304)
    • Enable split-KV heuristic for low-occupancy cross-attention in LTX-2 FA4 (#15399)
    • Fuse MLP up-GEMM + bias + GELU(tanh) + NVFP4-quant into the CuteDSL epilogue for LTX2 and WAN (#15299)
    • Add async mp4 encode and configurable noise latent via env vars in VisualGen (#15229)

  • Fix

    • Harden disagg cache transceiver teardown (#15422)
    • Fix encoder-decoder beam search corruption via per-slot fragmentPointerDevice (#15461)
    • Fix overallocation of draft KV cache (#15017)
    • Disable NCCL window buffers on GB10 (#15559)
    • Fix wrong NCCL fallback in nemotron-h (#15294)
    • Fix CuteDSL NVFP4 EPLB weight layout (#15538)
    • Enable CuTe DSL BF16 kernels for SM100 PP (#14993)
    • Fix Gemma4 multimodal vision TP and xgrammar startup crashes (#15566)
    • Add necessary methods for guided decoding in Kimi K2.5 (#15180)
    • Re-enable Ulysses for LTX-2 v2a cross-attention (#15303)
    • Fix passing scaled timestep to time_embedder in Cosmos3 (#15545)
    • Clarify and align trtllm-bench runtime logging (#15254)

  • Documentation

    • Add deploy guide for Minimax M3 (#15587)
    • Add Qwen Image visual generation examples (#15235)

  • Benchmark

    • Add Qwen-Image-Bench evaluator (#14837)
    • Add modularized perf tests for attention and MoE (discrete/continuous) (#15541)
    • Add Qwen3.5-397B-A17B-NVFP4 B200 aggregated perf-sanity tests (#15650)
    • Add DeepSeek R1 0528 FP4 performance test to llm_perf_core.yml (#15453)

  • Test & Infra

    • Move more test cases to post-merge (#15568)
    • Stabilize perf-sanity tests (#15440)
    • Avoid type checking failures due to pip dependency resolution (#15517)
    • Gate GPT-OSS TRT-LLM Gen MoE tests to SM100/SM103 (#15128)
    • Add GPT-OSS disagg test for transceiver v2 (#15301)
    • Fix Cosmos3 tests after VisualGen config split (#15170)
    • Fix visual gen test leaked issue (#15236)
    • Fix Qwen3-Next bf16 4gpu test (#15206)
    • Clean up Nemotron test cases (#15586)
    • Fix and unwaive step3p7 test cases (#15583)
    • Add test coverage for MiniMax model with multi-node M2.5 checkpoints eval (#15361)
    • Add GLM NVFP4 stress test (#15437)
    • Remove unreferenced accuracy tests and orphaned entries (#15593)
    • Update .gitattributes (#15606)

What's Changed

  • [None][fix] AutoDeploy: Fixed wrong dist_backend AUTO detection when using trtllm-llmapi-launch by @MrGeva in #15423
  • [None][test] Waive 2 failed cases for main in QA CI by @tensorrt-cicd in #15341
  • [TRTLLMINF-81][feat] Avoid failed runners on infra retry by @dpitman-nvda in #15237
  • [https://nvbugs/6179661][fix] Harden disagg cache transceiver teardown by @chienchunhung in #15422
  • [https://nvbugs/6273846][test] gate GPT-OSS TRTLLM Gen MoE tests to SM100/SM103 by @dongfengy in #15128
  • [None][fix] avoid type checking failures due to pip dependency resolution by @ixlmar in #15517
  • [None][feat] VisualGen: async mp4 encode + fixed noise latent via env vars by @wu6u3tw in #15229
  • [https://nvbugs/6337235][test] Fix MX/GMS model loader fixtures by @chienchunhung in #15471
  • [None][test] Un-waive K2.5 Thinking FP4 disagg-NIXL e2e/gen_only tests by @chenfeiz0326 in #15443
  • [None][test] Waive 3 failed cases for main in QA CI by @tensorrt-cicd in #15509
  • [None][test] Waive 11 failed cases for main in QA CI by @tensorrt-cicd in #15506
  • [None][test] Waive 4 failed cases for main in QA CI by @tensorrt-cicd in #15505
  • [TRTLLM-13550][feat] WideEP FT: add MPI signal handler replacement (1d.0) by @chienchunhung in #14160
  • [None][test] Remove 60 closed-bug waive entries for main by @tensorrt-cicd in #15511
  • [#3237][fix] Support negative numbers in MajorityVote digit validation by @nikJ13 in #12294
  • [None][test] Waive 10 failed cases for main in post-merge by @tensorrt-cicd in #15535
  • [None][test] Waive 9 failed cases for main in QA CI by @tensorrt-cicd in #15504
  • [None][test] Waive 1 failed cases for main in QA CI by @tensorrt-cicd in #15499
  • [None][test] Waive 4 failed cases for main in QA CI by @tensorrt-cicd in #15510
  • [None][fix] AutoDeploy: handle torch dist all_gather in multi_stream MLA transform by @MrGeva in #15456
  • [None][feat] Add Gemma-4 NVFP4 quantized models to AutoDeploy registry by @marinayanov in #15382
  • [None][fix] Fix encoder-decoder beam search corruption via per-slot fragmentPointerDevice by @achartier in #15461
  • [https://nvbugs/6306936][test] Re-enable AutoDeploy disagg tests by @govind-ramnarayan in #15325
  • [None][infra] split single-node perf sanity GB200 by @tburt-nv in #15548
  • [None][chore] Bump version to 1.3.0rc20 by @yuanjingx87 in #15551
  • [#10710][fix] clarify and align trtllm-bench runtime logging by @marinayanov in #15254
  • [https://nvbugs/6290345][fix] Fix allreduce benchmark input setup by @nv-lschneider in #15427
  • [None][feat] DSv4 prep: IndexerTopK and TopK primitives by @lfr-0531 in #15381
  • [None][perf] Cutedsl NVF4 MOE: grouped/swiglu GEMM: Fix acc pipeline release arrive threads + FC2 meta stage code clean by @liyuhannnnn in #15258
  • [https://nvbugs/6271740][test] Update llm_perf_core.yml to include new performance test for DeepSeek R1 0528 FP4 model by @yufeiwu-nv in #15453
  • [None][fix] Stabilize perf-sanity tests by @chenfeiz0326 in #15440
  • [None][test] fix Cosmos3 tests after VisualGen config split by @bobboli in #15170
  • [None][feat] DSv4 prep: compressor and mHC primitives by @lfr-0531 in #15379
  • [None][infra] Waive 3 failed cases for main in post-merge 2802 by @ZhanruiSunCh in #15571
  • [https://nvbugs/6264844][fix] Fix wrong NCCL fallback in nemotron-h by @Wanli-Jiang in #15294
  • [None][test] Waive 6 failed cases for main in QA CI by @tensorrt-cicd in #15570
  • [https://nvbugs/6344108][fix] skip TestNemotron3Super120B on pre-blackwell by @bo-nv in #15539
  • [None][fix] Fix passing scaled timestep to time_embedder in Cosmos3 by @bastefaniak in #15545
  • [None][chore] Remove nv-internal-release guardword comments in mega_moe_nvfp4 by @xxi-nv in #15575
  • [None][ci] move more test cases to post merge by @QiJune in #15568
  • [https://nvbugs/6185146][fix] Use mat_a.new_empty([m, n_out//2]) / input_scale.new_empty([sf_size]) in the by @tensorrt-cicd in #14710
  • [TRTLLM-35882][feat] cute dsl gvr-topk load-balance optimization by @limin2021 in #15304
  • [None][test] Waive 2 failed cases for main in QA CI by @tensorrt-cicd in #15579
  • [None][test] waive hang issues by @xinhe-nv in #15576
  • [None][test] waive hang issues by @xinhe-nv in #15581
  • [#14874][feat] AutoDeploy : Perf optimization for gpt-oss-120b for low conc by @taylor-yb-lee in #15531
  • [TRTLLM-12982][perf] reuse multi-item scoring position_ids and params by @ixlmar in #15413
  • [TRTLLM-13599][test] Refine Qwen3.5 test cases by @nv-guomingz in #15544
  • [TRTLLMINF-111][inf...
Read more

Contributors

Thachnh, achartier, and 61 other contributors
Loading

v1.3.0rc19

v1.3.0rc19 Pre-release
Pre-release

Choose a tag to compare

View all tags
@mikeiovine mikeiovine released this 23 Jun 16:49
v1.3.0rc19
a8c5955

  • Known Issues

    • Llama 3.1 8B FP8 can hang during the autotuner warmup on GB200.

  • Model Support

    • Support NVIDIA Wan2.2-T2V quantized checkpoints (#15093)
    • Enable MTP for Step-3.7 NVFP4 and port Step-3.7VL vision tower to TRT-LLM modules (#14926)
    • Support T5 and BART in the PyTorch backend (#13919)
    • Support MiniMax-M3 in the PyTorch backend (#15292)

  • API

    • Align VisualGen serve request schema with VisualGenParams (#14733)
    • Support multi-item scoring in LLM.encode (#14693)
    • Drop legacy --extra_visual_gen_options CLI alias (#15262)

  • Feature

    • Enable TRTLLM MoE backend for Nemotron-H BF16 checkpoint (#14944)
    • Add async Ulysses pipeline (enabled for LTX-2 and WAN) (#13978)
    • Make TrtllmGenAttention the default decode backend on Blackwell+ (#14618)
    • Skip redundant data expand in DeepGemmFusedMoE via fused expand+quant Triton kernel (#14591)
    • Add Prometheus metrics for prompt cache, speculative decoding, perplexity, and batch occupancy (#12636)
    • Add Indexer TopK single-block / multi-pass radix implementation (#14268)
    • Enable gen-only speculative decoding for disagg setups (#14546)
    • Support EAGLE3 dynamic trees on Blackwell (#12958)
    • Add CUDA graph support for per-expert LoRA in Cutlass backend (#14881)
    • Add support for beam search in disaggregated serving (#14876)
    • Add maximal LLMAPI capture in usage telemetry (#14398)
    • Optimize Qwen2.5/3/3.5-VL performance (#11943)
    • Add skip-softmax TMA-load + sync-MMA warp-specialized context FMHA for sm_120/sm_121 (#15163)
    • Enable TRTLLM cross attention backend (#15345)
    • Support per-request mm_processor_kwargs for Qwen3-VL (#14702)
    • Add prefetch_reuse_blocks and configurable prefetch count (#15149)
    • Add MegaMoECuteDsl NVFP4 MoE backend (#14608)
    • Make EAGLE3 honor sampling params by default (#14745)
    • Add multiple FMHA library support to TRTLLM attention backend (#15204)
    • Add checkpointing variant of replay for MTP for mamba models (#14203)

  • Fix

    • Remove redundant TikTokenTokenizer shim from Kimi-K2.5 input processor (#14741)
    • Rename misnamed tunable_fp4_quantize kwarg and add real SF-swizzle control (#15002)
    • Gate FlashInfer GDN kernels to supported configurations (#15094)
    • Count DSA indexer K-cache correctly as UINT8 in KV cache size estimate (#15088)
    • Select CUTLASS MoE backend on non-Blackwell SMs for Qwen3.5-35B-A3B FP8 (#15081)
    • Fix SageAttention kernel regression by using static scheduler (#15047)
    • Fall back to local cache when loading tokenizer for gated models (#12998)
    • Fix PyExecutor FPM iteration timing (#14922)
    • Register multimodal placeholders for Qwen3.5 MoE VLM serving (#15079)
    • Fix and unwaive Nemotron-related bugs (#15085)
    • Guard DSA DSL atom-split against MTP draft next (#14891)
    • Scope disagg-ctx cache-transfer quorum vote to TP instead of WORLD (#15136)
    • Clear workspace in run_mla_generation to avoid illegal memory access (#15173)
    • Fix MAX_UTILIZATION reuse token budget (#15066)
    • Add kv_transfer_timeout_ms to avoid timeout (#15152)
    • Preserve ip:port for trtllm-serve visual-gen (#14355)
    • Fix guided decoding (xgrammar) + EAGLE-3 + draft_len_schedule crash during CUDA graph capture (#15023)
    • Stabilize Mamba replay state update (#14841)
    • Fix max_context_length value for attention workspace sizing (#15156)
    • Fix issue where host KV cache usage would double when speculative decoding is used (#14373)
    • Disable NCCL_SYMMETRIC tactic on GB10 (DGX Spark) (#12902)
    • Fix attentionOp FP8 MLA KV-reuse workspace calculation (#14852)
    • Fix beam search log_probs non-determinism with batch_size > 1 (#15125)
    • Forward secondary_offload_min_priority to KVCacheManager in PyTorch executor (#13768)
    • Enable multi-block mode for XQA HMMA spec-dec (#15312)
    • Fix TinyGEMM barrier bug (#15338)
    • Fix stale sparse attention kwargs (#15460)
    • Fix CppMambaHybridCacheManager to handle dp dummy request (#15054)
    • Fix embedding vocab mask for rejection sampling in Kimi-K2.5 (#15233)

  • Documentation

    • Add FLUX visual generation examples (#14987)
    • Add Qwen3.5 deployment guide doc (#15111)
    • Fix stale --disable_xqa reference in legacy docs (#13395)
    • Add Cache-DiT documentation (#15268)

  • Benchmark

    • Weight trtllm-bench AR/AL averages by output length (#14998)

  • Test & Infra

    • Add accuracy tests for nemotron-v3-ultra (#14808)
    • Remove TestLlama4ScoutInstruct tests (#15144)
    • Require minimum of 4 GPUs in llm_perf_core.yml and add new performance tests (#15090)
    • Add DFlash coverage for Qwen3.5 MoE variant (#15132)
    • Add e2e example tests for flux1/2, ltx2, wan_i2v, and cosmos3 (#15126)
    • Enable disagg cancellation stress test (#15174)
    • Fix periodic-junit in unittest pytest (#14075)
    • Update K2.5 and GLM-5 into CI perf test (#14960)
    • Add Qwen3-32B FP8 disagg stress test (#14278)
    • Sunset old disagg test cases for the QA side (#15290)
    • Add e2e Tensor Parallel LPIPS tests for VisualGen (#15208)
    • Remove TensorRT performance baseline and update to PyTorch only (#15256)
    • Add integration tests for MoE LoRA and bugfixes (#15271)

What's Changed

  • [None][infra] Waive TestQwen3NextInstruct nvfp4 cases by @mzweilz in #15086
  • [https://nvbugs/6248757][fix] Avoid running all reduce in aux stream by @tensorrt-cicd in #14917
  • [https://nvbugs/6221483][fix] AutoDeploy: Fix Eagle metadata host syncs by @govind-ramnarayan in #14714
  • [None][feat] add FLUX visual generation examples by @karljang in #14987
  • [https://nvbugs/6261164][fix] AutoDeploy: Don't allocate speculative caches when speculation is off by @tensorrt-cicd in #15020
  • [https://nvbugs/6211189][fix] Lower the reference to 46.5 (matching cross-GPU empirical mean) and remove the t by @tensorrt-cicd in #14799
  • [None][refactor] split VisualGen pipeline and model configs by @bobboli in #14956
  • [TRTLLM-11457][feat] Async Ulysses pipeline (Enabled for LTX-2 + WAN) by @luyiyun1021 in #13978
  • [TRTLLM-11548][doc] Add Qwen3.5 deployment guide doc by @nv-guomingz in #15111
  • [https://nvbugs/6181383][fix] Build inner text/vision/audio sub-configs as empty PretrainedConfig() then setat by @tensorrt-cicd in #14399
  • [https://nvbugs/6273850][chore] waive TestQwen3_5_4B::test_bf16 for all GPUs by @tburt-nv in #15112
  • [None][doc] Add docs for AutoDeploy transforms by @bmarimuthu-nv in #15122
  • [None][infra] Waive 4 failed cases for main in post-merge 2769 by @ZhanruiSunCh in #15140
  • [https://nvbugs/6227203][fix] Remove redundant TikTokenTokenizer shim from KimiK25InputProcessor by @tianyuxbear in #14741
  • [None][fix] tunable_fp4_quantize: rename misnamed kwarg + add real SF-swizzle control by @luyiyun1021 in #15002
  • [None][test] Fix gen_only missing prev_device_step_time race in perf sanity by @tensorrt-cicd in #15108
  • [None][test] Fix disagg test result dir by @fredricz-20070104 in #14864
  • [TRTLLM-13332][test] Remove TestLlama4ScoutInstruct tests by @QiJune in #15144
  • [https://nvbugs/6266705][fix] Gate FlashInfer GDN kernels to supporte… by @nv-guomingz in #15094
  • [https://nvbugs/6255037][fix] Count DSA indexer K-cache correctly as UINT8 in KV cache size estimate by @eopXD in #15088
  • [https://nvbugs/6194812][test] Update llm_perf_core.yml to require a minimum of 4 GPUs and add new performance tests by @yufeiwu-nv in #15090
  • [TRTLLMINF-112][infra] Reduce the waiting time between check node is online or not by @EmmaQiaoCh in #14819
  • [None][infra] Waive 1 failed cases for main in pre-merge 41821 by @ZhanruiSunCh in #15135
  • [None][infra] CBTS Layer 3: pass test-db via Artifactory instead of env var by @crazydemo in #15142
  • [TRTLLM-13264][feat] Add native bias epilogue to NVFP4 GEMM by @luyiyun1021 in #15053
  • [https://nvbugs/6278380][unwaive] unwaive ad cases by @crazydemo in #15148
  • [https://nvbugs/6244474][fix] AutoDeploy: Remove llama perf test from CI by @MrGeva in #15107
  • [https://nvbugs/6212252][fix] Select CUTLASS MoE backend on non-Blackwell SMs in TestQwen3_5_35B_A3B::test_fp8 by @xxi-nv in #15081
  • [TRTLLM-13302][feat] Register NVIDIA Wan2.2-T2V quantized checkpoints by @zhenhuaw-me in #15093
  • [None][chore] add VisualGen team as the codeowner of the VisualGen Attention by @zhenhuaw-me in #15150
  • [None][feat] Default on FlashInferTrtllmGenAttention by @yihwang-nv in #14618
  • [None][infra] Test DFW with BSL branch by @yuanjingx87 in #14597
  • [TRTLLM-12214][perf] customMoeRoutingKernel: lower BLOCK_SIZE to 128, raise m...
Read more

Contributors

karljang, dcampora, and 84 other contributors
Loading

v1.3.0rc18

v1.3.0rc18 Pre-release
Pre-release

Choose a tag to compare

View all tags
@mikeiovine mikeiovine released this 10 Jun 00:10
v1.3.0rc18
15d06c0
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

  • Known Issues

    • DSV3.2 will crash with an IMA in various long-running perf tests on GB200/GB300 when the CuteDSL MoE backend is used. Work around this issue by using another MoE backend.

  • Model Support

    • Support Nemotron-H NVFP4 checkpoint on Hopper (#14775)
    • Add Qwen image support (#13449)
    • Support Step-3.7-Flash model (#14711)
    • Add Cosmos3-Nano and Cosmos3-Super support (#14824)
    • Add AFMoE Trinity support (#13148)

  • API

    • Add logprobs_simple_format option to return logprobs as a flat list[float] (#13972)
    • trtllm-serve, trtllm-eval, trtllm-bench: Make CLI flags take precedence over --config / --extra_llm_api_options YAML (#14812)

  • Feature

    • Upgrade NIXL to v1.0.1 and UCX to 1.21 (#14436)
    • Refactor DWDP from CUDA IPC to CUDA VMM + MNNVL (#14453)
    • Enable FlashInfer GDN decoding kernel for Qwen3.5 (#13645)
    • Add per-expert LoRA support with Cutlass backend (#14801)
    • Reduce OpenAI stream postprocess overhead (#14708)
    • Add encoder CUDA graph support to llm.encode() (#14326)
    • Use a Triton kernel for C++ mamba hybrid state update (#14869)
    • Fuse masked gather + finalize-scale into one Triton kernel in DeepGemmFusedMoE (#14592)
    • Support KVCacheManagerV2 adjust() in single GPU + agg PyExecutor loop (#14578)
    • Add disk cache config for KVCacheManagerV2 (#14845)
    • Add Wan I2V generation example (#14981)
    • Add LTX-2 visual generation example (#14976)
    • Update flashinfer-python from 0.6.12rc2 to 0.6.12 (#14805)

  • Fix

    • Fix mamba-out-of-block error with ADP + BS=1 + disagg (#14853)
    • Fix XQA IMA for invalid pages with sliding window (#14459)
    • Propagate event loop errors to await_responses callers (#12735)
    • Fix Mamba replay mode accuracy issues (#14509)
    • Fix PyExecutor hang in disagg TP prefill (#14020)
    • Fix stale runtime metadata issues during MLA fallback transitions (#14049)
    • Fix KVCacheManagerV2 block counting correctness issues (#14725)
    • Canonicalize multimodal cache-key serialization to prevent hash collisions (#14800)
    • Fix LTX-2 audio PE padding issues (#14818)
    • Release KVCacheManagerV1 blocks on MAX_UTILIZATION pause (#14723)
    • Fix config sharing issue for Qwen3-VL (#14766)
    • Enforce request and buffer index lifecycle integrity (#14768)
    • Add nemotron-v3 as the proper nemotron-h reasoning parser (#14900)
    • Clamp KV pool window sizes to max_seq_len (#14905)
    • Fix mamba block calculation (#14524)
    • Add trust_remote_code=True to the LLM(...) constructor to fix various model loading issues (#14892)
    • Fix deep EP partial warp sync for GPT-OSS shapes (#14977)
    • Add warmup for trtllm-gen fmha JIT kernels (#14851)

  • Documentation

    • Add VisualGen API walkthrough example and docs page (#14685)
    • Add Nemotron 3 Ultra doc (#14964, #15113)

  • Test & Infra

    • Pipe stderr separately in subprocess calls to improve error reporting in Allure (#14750)
    • Remove obsolete tests (#14995, #14660, #14992, #14952, #14749)
    • Parallelize post stages: Rerun Report, Test Coverage, and AI Failure Analysis (#14528)
    • Relocate tests to right-sized stages (#14684)
    • Move non-default-feature tests to post merge (#15038)

What's Changed

Read more

Contributors

karljang, reasonsolo, and 68 other contributors
Loading

v1.3.0rc17

v1.3.0rc17 Pre-release
Pre-release

Choose a tag to compare

View all tags
@mikeiovine mikeiovine released this 02 Jun 18:50
v1.3.0rc17
a422420
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Highlights

  • Known Issues
    • DeepSeek V3.2 will crash with an illegal memory access during long-running performance tests under various agg/disagg configurations.
  • Model Support
    • Add MoT World Model support (#14012)
    • Enable multi-node tensor parallelism for MiniMax-M2 (#14314)
    • Restore Mistral Large 3 text-only processor (#14248)
    • Support Gemma4 multi-head_dim pools and host-side slicing for SWA Triton kernels (#13745)
    • Add a reasoning parser for Qwen3.5 (#14659)
    • Add LTX-2 Ulysses cross-attention for v2a with audio padding (#14044)
    • Add Poolside Laguna tool parser (#14638)
    • Replace Parakeet audio encoder with native TensorRT-LLM layers (#14474)
    • Set Mamba SSM cache to fp32 for NemotronV2 (#14448)
    • API
    • Allow content: null in CustomChatCompletionMessageParam (#14368)
    • Enforce trust_remote_code flag (#13527)
    • Add thinking token budget control (#14665)
    • Expose host/GPU per-iter time and clarify iter labeling in /metrics (#14127)
    • Make attention backend case-insensitive (#14635)
  • Feature
    • Add FlashInfer NVFP4 MoE backend (SM120/SM121) for Nemotron (#13773)
    • Integrate the FlashInfer GDN prefill kernel for Qwen3.5 (#13644)
    • Add LoRA support to LLMAPI Triton backend (#14079)
    • Log KV cache utilization and context tokens per iteration (#14206)
    • Remove one-warp-per-token policy from MoE A2A kernels (#14550)
    • Support non-divisible expert parallelism in MoE all-to-all and Slurm benchmark (#13888)
    • Add CuTe DSL attention via exported binaries in VisualGen (#13721)
    • Enable NVFP4 KV cache support in trtllm-gen attention (#12544)
    • Add GMS-only weight sharing support (#13926)
    • Add VisualGen tensor parallelism support (#13614)
    • Enable NCCL symmetric zero-copy by default (#14472)
    • Improve disaggregated TTFT (#14719)
  • Fix
    • Restore K2.5 multimodal dep8 accuracy test on Transformers 5.5.x (#14392)
    • Remove sync after FlashInfer attention plan() (#14634)
    • Add a compatibility shim in load_hf_tokenizer for bytes_to_unicode (#14090)
    • Route trtllm-bench and trtllm-serve tokenizer load through TransformersTokenizer (#14452)
    • Fix crash in deep_ep.pyby falling back to the pre-quant dispatch path when hidden_states_sf is missing (#14404)
    • Fix gpt-oss accuracy issue by moving TinyGEMM PDL release after reduction (#14537)
    • Fix Mistral-Large-3 weight loading crash (#14033)
    • Bypass FlashInfer SSD prefill to fix state dtype precision (#14600)
    • Fix qwen3 hang on SM120/121 (#14424)
    • Fix NVFP4 engine size estimation and attention DP batch size in trtllm-bench (#13498)
    • Catch OSError in config_file_lock for NFS compatibility (#11960)
    • Fix MoE DeepGEMM workspace size with attention DP (#13310)
    • Fix inf/NaN issues in Triton Mamba softplus (#14652)
    • Cap per-rank max_num_active_requests by max_num_tokens under attention DP (#14481)
    • Propagate external SWA window to FMHA kernel in V2 KV cache (#13719)
    • Resolve NVML device index mismatch in get_numa_aware_cpu_affinity when CUDA_VISIBLE_DEVICES is set (#12985)
    • Replace fixed disagg fill throttle with slow-start ramp (#14475)
    • Reuse batch_indices_cuda across CUDA graph captures in EAGLE3 (#14381)
    • Make FA4 a proper pip dependency (#13788)
    • Fix GSM8K accuracy tests for LagunaXS on B200/GB200/B300 (#14580)
  • Documentation
    • Add CUTLASS DSL uninstall step to installation guide (#14621)
    • Add deprecation notice to legacy support-matrix.md (#14495)
    • Fix incorrect auto sampler behavior description for beam search (#14487)
    • Add VisualGen context to AGENTS.md (#14732)
  • Test & Infra
    • Update flashinfer-python from 0.6.11.post1 to 0.6.12rc2 (#14512, #14607)
    • Add disagg local one-step run script for CI submit (#14557)
    • Update model path definitions in test_perf.py and clean up waives.txt (#14393)
    • Dedup executor unit tests on H100/B200 (#14556)
    • Add disagg cancellation stress-test harness skeleton (#14375)
    • Add UCX TLS env in disagg-related tests (#14626)
    • Replace ONNX spec with onnx>=1.21.0 in requirements.txt (#14577)
    • Add test lists with multi-GPU tests to CI multi-GPU test trigger files (#14087)
    • Add offline equivalence test for sharding IR (#13963)
    • Enable kv_cache_manager_v2 test for A10 (#12885)
    • Remove two-model EAGLE3 spec-decoding tests (#14735)
    • Add TLLM_SPEC_DECODE_FORCE_NUM_ACCEPTED_TOKENS in spec decoding perf test (#14438)

What's Changed

  • [https://nvbugs/6182617][fix] Restore K2.5 multimodal dep8 accuracy test on transformers 5.5.x by @tianyuxbear in #14392
  • [None][feat] FlashInfer NVFP4 MoE backend (SM120/SM121) for Nemotron … by @farazkh80 in #13773
  • [None][perf] Integrate the flashinfer gdn prefill kernel for qwen3.5 by @nv-guomingz in #13644
  • [None][chore] Update flashinfer-python from 0.6.11.post1 to 0.6.12rc1 by @yihwang-nv in #14512
  • [https://nvbugs/6162328][fix] Add a tiny compat shim in load_hf_tokenizer that, when bytes_to_unicode is m by @tensorrt-cicd in #14090
  • [https://nvbugs/6114610][test] unwaive disagg tests fixed by UCX_TLS setter by @xwang233 in #14440
  • [None][fix] Route trtllm-bench and trtllm-serve tokenizer load through TransformersTokenizer by @dc3671 in #14452
  • [https://nvbugs/6184914][test] Unwaive related tests by @yuxianq in #14523
  • [https://nvbugs/6186880][fix] In deep_ep.py, fall back to the pre-quant dispatch path when hidden_states_sf is by @tensorrt-cicd in #14404
  • [None][infra] Waive 2 failed cases for main in post-merge 2734 by @ZhanruiSunCh in #14526
  • [None][infra] Waive 1 failed cases for main in post-merge 2735 by @ZhanruiSunCh in #14542
  • [#11257][feat] Add LoRA support to llmapi triton backend by @karljang in #14079
  • [None][chore] Include layer_idx in MoE backend fallback warnings by @dc3671 in #13409
  • [None][chore] Add disagg local one-step run script for CI submit by @fredricz-20070104 in #14557
  • [https://nvbugs/5974335][refactor] Update model path definitions in test_perf.py and clean up waives.txt by @yufeiwu-nv in #14393
  • [TRTLLM-12968][ci] Dedup executor unit tests on H100/B200 by @YihuiLu512 in #14556
  • [TRTLLM-12949][refactor] visual_gen: unify fused QK-norm+rope dispatch by @luyiyun1021 in #14529
  • [https://nvbugs/6143579][fix] Allow content: null in CustomChatCompletionMessageParam by @tijyojwad in #14368
  • [None][chore] log KV cache utilization and context tokens per iter by @pcicotti in #14206
  • [https://nvbugs/6168859][fix] move tinygemm PDL release after reduction by @dongfengy in #14537
  • [None][chore] Unwaive test_cp_tp_broadcast_object by @brb-nv in #14328
  • [https://nvbugs/6211185][fix] Fix failed GSM8K accuracy tests for LagunaXS on B200/GB200/B300 by @DomBrown in #14580
  • [TRTLLMINF-106][infra] Use B300 frontend platforms by @mlefeb01 in #14581
  • [None] [refactor] Unify compressed-tensors quant config parsing by @DomBrown in #14468
  • [None][feat] AutoDeploy push the rope buffer to later stage by @nvchenghaoz in #13859
  • [https://nvbugs/6215736][infra] Unwaive test_fp8_blockscale[throughput_mtp] by @bobboli in #14541
  • [https://nvbugs/6175923][test] Revert gpt_oss_20b perf MoE-backend pin by @ruodil in #14612
  • [https://nvbugs/6221621][test] Update trust_remote to nemotron and phi4 models by @yufeiwu-nv in #14570
  • [None][chore] update VisualGen codeowner settings by @zhenhuaw-me in #14530
  • [None][infra] Waive 8 failed cases for main in post-merge 2738 by @ZhanruiSunCh in #14615
  • [None][perf] Fuse FlashInfer GDN prefill state I/O into Triton kernels by @nv-guomingz in #14548
  • [https://nvbugs/6164924][fix] Lower free_gpu_memory_fraction for Exaone tests by @tensorrt-cicd in #14486
  • [https://nvbugs/6163033][fix] Guard q_a_proj.weight dict access behind nvfp4_fused_a; update test to `chec by @tensorrt-cicd in #14033
  • [None][fix] Bypass FlashInfer SSD prefill to fix state dtype precision by @tijyojwad in #14600
  • [None][fix] Exclude Qwen3 VL vision model from quantization by @2ez4bz in #12851
  • [https://nvbugs/6162860][fix] Set free_gpu_memory_fraction=0.6 only when torch_compile=True for test_bfloat16_ by @tensorrt-cicd in #14109
  • [None][chore] Remove one-warp-per-token policy from MoE A2A kernels by @bobboli in #14550
  • [None][test] Waive 7 failed cases for main in QA CI by @xinhe-nv in https://github.c...
Read more

Contributors

karljang, tijyojwad, and 62 other contributors
Loading

v1.3.0rc16

v1.3.0rc16 Pre-release
Pre-release

Choose a tag to compare

View all tags
@VALLIS-NERIA VALLIS-NERIA released this 26 May 08:08
v1.3.0rc16
4517988
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Highlights

  • Model Support

    • Add Gemma4 multimodal support with native vision and audio towers (#14300)
    • Add Qwen3.5 MTP and Qwen3.6-27B-FP8 model support (#12646, #14359)
    • Add EXAONE-4.5 and Laguna model support (#12873, #13559)
    • Switch DeepSeek, NemotronH, Qwen3, and Qwen3.5-MoE to sharding-IR canonical models (#13478)

  • API

    • Refactor the VisualGenArgs API and registry (#14175)
    • Drop sink_token_length from the PyTorch attention surface (#14275)
    • Add OpenAI chat logit bias validation (#13518)
    • Reject incompatible KV connector configurations at construction time (#13577)

  • Feature

    • Add exact multimodal KV block hashing and KV cache reuse probing (#13815, #14333)
    • Add KV cache manager v2 with Python transceiver updates (#12928)
    • Add disaggregated serving support with block reuse enabled for hybrid models (#14060)
    • Add FlashInfer MLA attention backend support and SkipSoftmax sparse attention support for visual generation (#13428, #12947)
    • Add Ring Attention and unified context parallelism for VisualGen (#13821)
    • Add legacy and TensorRT-LLM 1.x modelopt quantization config support (#14088)
    • Add debugging environment variables for mamba modules (#14170)
    • Add single-rank MPI sleep/wakeup support and a rank-0 collective_rpc shim (#14052)
    • Add opentelemetry metrics for disaggregated serving with multiple postprocessing workers (#12637)
    • Support SWA scratch reuse rewind (#14412)
    • Improve FMHA, FlashInfer TRTLLM-Gen, and KV cache buffer calculation paths (#14291, #12525)
    • Improve fused-kernel and attention performance with shared-expert combine fusion, paged MQA logits decode tuning, LTX2 fused RMSNorm/RoPE, EAGLE3 dynamic tree kernel optimizations, and cu_seqlens conversion updates (#14306, #14133, #13985, #13426, #13566)
    • Optimize beam search candidate reconstruction by skipping prompt-prefix copies (#14197)
    • Update cubins to resolve the FMHA PDL issue (#14462)
    • Use CUDA 13 CUTLASS DSL package (#14354)

  • Fix

    • Fix disaggregated benchmark, usage propagation, and worker registration stability issues (#13347, #14177, #14289)
    • Fix DeepSeek-V3 OOM handling and artifacts paths (#14232)
    • Fix missing get_draft_token_length import in py_executor (#14366)
    • Fix Lora load failure handling (#13517)
    • Fix Kimi K2.5 speculative decoding behavior (#14379)
    • Fix Qwen3HybridConfig layer_types derivation and route load_hf_model_config through AutoConfig (#13832, #14410)
    • Fix CppMambaHybridCacheManager functional and performance issues (#14003)
    • Fix MTP disaggregated speculative_config coverage (#14391)
    • Fix KVCacheTransfer divide-by-zero and KV cache grain slot refinement issues (#13618, #14442)
    • Fix memory usage during refit and EPLB config model loading (#14331, #11962)
    • Fix MPI worker allocator configuration and GB300 cluster environment setup (#14152, #14460)
    • Fix profiler runner exception handling with synchronized CUDA cleanup (#13469)
    • Disable mamba replay by default (#14471)

  • Documentation

    • Add a Claude skill for multimodal model onboarding (#13842)
    • Update Gemma 4 entries in supported-models.md (#14463)
    • Fix invalid documentation and deployment guide links (#14337, #14522)

  • Benchmark

    • Add LPIPS scoring for visual generation model regression tests (#13567)
    • Add a bench_moe microbenchmark (#14507)
    • Update visual generation and accuracy thresholds for Wan 2.2, Qwen3.5-4B DFlash, and Nano V3 (#14372, #14411, #14078)
    • Disable ignore-eos when using speculative decoding in performance tests (#14347)

  • Test & Infra

    • Split verl tests into fine-grained per-case wrappers (#14037)
    • Add new stress cases (#14390)
    • Clean outdated test duration entries and remove deprecated disaggregated sampler and spark test cases (#14340, #14335, #14380)
    • Isolate ray tests to avoid GCS timeout in a single pytest session (#14342)
    • Improve L0 retry timeout budgeting and cap infra retry attempts (#14323, #14415)
    • Handle sacct errors when checking Slurm job status (#14367)
    • Fix B300 MegaMoE and MoE test selection (#14362, #14401)
    • Fix container scanning according to the latest security team guidance (#14430)
    • Deduplicate miscellaneous unit tests on B200 (#14525)

What's Changed

Read more

Contributors

Superjomn, karljang, and 61 other contributors
Loading

v1.3.0rc15

v1.3.0rc15 Pre-release
Pre-release

Choose a tag to compare

View all tags
@VALLIS-NERIA VALLIS-NERIA released this 21 May 14:27
v1.3.0rc15
c72d43d
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Highlights

  • Model Support

    • Add Gemma4 multimodal model support with text, vision, audio, and chunked prefill capabilities (#12932, #14134)
    • Add Kimi K2.5 multimodal vision support and reasoning parser integration (#12788, #13801)
    • Add GPT-OSS, Ministral3, Nemotron-H, Nemotron Nano, and DeepSeek model enablement and compatibility updates (#12743, #12884, #13844, #13977)
    • Improve DeepSeek V4 and DeepSeek V3.2 support with new attention kernels, routing updates, tokenizer loading, and AutoConfig registration (#13652, #13186, #14261, #14293)

  • API

    • Add a typed exception hierarchy, shared classifier, retry-consumer migration, and typed Slurm infra failures (#13732, #13780, #13863, #13809, #14147)
    • Add VisualGen public output APIs, serving batch inference, and benchmark timing decomposition (#13635, #12350)
    • Add per-request media_io_kwargs support for chat completions (#13779)
    • Add per-rank iteration statistics and Attention-DP metrics to serving endpoints (#13221, #13649)
    • Add cache_salt_id support to the KV cache v2 manager (#13793)
    • Limit requested sampling logprobs as a breaking API change (#13520)

  • Feature

    • Improve MoE and fused-kernel performance with MegaMoE DeepGEMM, CUTEDSL MoE, shared-expert SwiGLU quantization, GDN fusion, bf16 FlashInfer MoE, and refreshed MoE cubins (#13384, #12884, #11897, #12966, #13689, #12440)
    • Add FP4 and FP8 decode kernels, FP4 DSA indexing, DeepSeek V4 attention kernels, FMHA head_dim 80 cubins, and multi-K and multi-dtype GVR Top-K support (#13929, #13219, #13340, #13652, #13808, #13948)
    • Improve VisualGen and diffusion pipelines with SageAttention for Wan/FLUX, fused cross-head QK Norm plus RoPE for WAN, LTX2 refactoring, and parallel VAE scaling (#13570, #13052, #13285, #13873)
    • Improve KV reuse, disaggregated serving, and transfer paths with transceiver v2 KV reuse, multi-threaded KV transfer, internal TRTLLM-Gen routing, additional conversation headers, and LoRA request-broadcast reduction (#13115, #13075, #13997, #13656, #12959)
    • Improve speculative decoding and hybrid-model execution with fractional synthetic acceptance rates, MTP block reuse, EAGLE3 rejection sampling, MTP max_draft_len decoupling, and mamba SSD prefill optimizations (#13569, #12896, #12588, #12341, #12731)
    • Improve performance tooling and runtime throughput with DFlash optimizations, host-profiler utilities, batch-full benchmark metrics, model-init NVLink caching, scheduling overhead reductions, beam-search overlap scheduling, and FC2 DenseGEMM autotuning (#13996, #11741, #13638, #14070, #13843, #14061, #13833)
    • Add CMake third-party cache support for clean builds (#13942)

  • Fix

    • Fix CUDA graph, profiling, and scheduling correctness issues including YAML CudaGraphConfig validation, profiler scoping, piecewise capture, Eagle3 hidden-state reuse, and guided decoding GIL handling (#13397, #12432, #13574, #13920, #13251)
    • Fix KV cache and scheduler behavior for FlashMLA token block overrides, mamba slot memory, delayed batching page release, adaptive ratio sampling, zero-layer mamba ranks, stale Scheduler V2 state, stale attention metadata, and chunked prefill EVS merging (#13752, #13489, #13805, #13857, #13999, #13592, #13696, #13754)
    • Fix model loading and quantization issues for GPT-OSS MXFP4, dummy weights, Mixtral modelopt export, DeepSeek V3 Lite FP8 MTP weights, composite HF configs, GLM-5 router GEMM, INT4 AWQ on SM120/121, and Qwen3 FP4 CUTLASS MoE OOM (#13708, #13879, #14179, #12530, #14068, #13740, #11561, #13349)
    • Fix serving and benchmark clients with hardened media URL loading, split SSE chunk parsing, aiohttp 3.13 streaming handling, /metrics tee-buffer serving, bounded gRPC payloads, router tokenizer skipping, unset attention_dp_relax handling, and clear GPT-OSS backend errors (#12748, #13686, #13952, #13405, #13519, #14030, #14276, #13166)
    • Fix distributed and disaggregated runtime stability for mamba disaggregation, worker preparation, PP executor shutdown, SM120 all-reduce launch, guided-decoding PP warmup barriers, Torch process-group teardown, Triton MoE memory freeing, and GB300 UCX settings (#13274, #13755, #13267, #13169, #13132, #12993, #14069, #14168)
    • Fix accuracy and memory regressions in DeepSeek, Nemotron, Qwen3, MTP, beam search, FMHA workspace sizing, and FP8 block-scaling autotuner cache growth (#13924, #13968, #13782, #14063, #13799, #13880, #14165)
    • Fix package, license, and compliance issues in llm-c standalone generation, SPDX headers, OSS headers, diffusers pinning, and broken documentation URLs (#14011, #14106, #14193, #14281, #13242, #13422)

  • Documentation

    • Add and update technical blogs for Helix Parallelism, Scaffolding, Gemma4, MoE as Dense GEMM on Blackwell, and VisualGen-related content (#13547, #11841, #13947, #13834, #14171)
    • Add DFlash quickstart updates, custom PyTorch backend kernel integration guidance, Gemma4 usage examples, spec-decoding support matrices, and layer-wise benchmark doc fixes (#13545, #13917, #14303, #14195, #13979)
    • Refresh image links and broken URLs in documentation and blog content (#13838, #13422)

  • Test & Infra

    • Add model and multimodal coverage for Wan 2.2 TI2V, nano v3 omni audio and video, Nemotron Ultra V3, Gemma4 CUDA graph registration, and W4A8_MXFP4_FP8 MoE unit tests (#13739, #13616, #13750, #13883, #13658, #14082, #13401)
    • Add and refresh performance coverage for VisualGen sanity, GB300 disaggregated NIXL, DSR1 disaggregated tests, trtllm-bench metrics, and Kimi K2.5 FP4 RCCA tests (#13144, #13594, #13882, #14178, #14172)
    • Improve change-based testing, CI triggers, GitHub checks, stage splitting, rerun handling, and LFS synchronization (#13382, #13899, #13993, #14022, #14064, #14035, #12406, #13826)
    • Improve build, dependency, and package infrastructure with FlashInfer updates, Transformers 5.x upgrades, compressed cubin archives, SBSA wheel image support, license scanning, and llm-c artifact cleanup (#13746, #13992, #14076, #12829, #13994, #13542, #12635, #13921, #13272)
    • Improve CI coverage organization by moving chunked-prefill cases, splitting long hardware-agnostic tests, adding feature-contract keys, and promoting DeepSeek-V4-Flash to the MoE CI subset (#14083, #13751, #13756, #13933, #13964)
    • Improve developer and CI operations with blossom-ci allowlist updates, skills naming enforcement, pre-commit validation, source-scan cleanup, and NFS temporary-file ignores (#13951, #14132, #14295, #14304, #14285, #13778, #14211)

What's Changed

  • [https://nvbugs/6001694][fix] Add CUDA profiler API scoping for visual gen nsys profiling by @chang-l in #12432
  • [https://nvbugs/6080024][fix] Fix CudaGraphConfig validation conflict from YAML deep merge by @nvchenghaoz in #13397
  • [None][perf] AutoDeploy: reduce C++ dispatch overhead in decode scheduling loop by @nvchenghaoz in #13012
  • [None][doc] Blogpost for Helix Parallelism by @brb-nv in #13547
  • [None][chore] Fix indexing conflict in blogposts by @brb-nv in #13772
  • [#12713][feat] AutoDeploy Model Onboarding Sprint 03/19 - Part 1 (Remove Patches) by @govind-ramnarayan in #13247
  • [https://nvbugs/5911304][fix] Add URL validation and request hardening for media input loading by @yibinl-nvidia in #12748
  • [None][infra] Remove PULSE_REPO_BRANCH when running source code scanning by @yuanjingx87 in #13778
  • [TRTLLMINF-54][feat] Add typed exception hierarchy + unified classifier by @dpitman-nvda in #13732
  • [https://nvbugs/6094072][fix] swizzle GPT-OSS dummy MXFP4 weights by @dongfengy in #13708
  • [https://nvbugs/6094224][fix] Fix mamba disagg issues when conc > mbs by @bo-nv in #13274
  • Add log for raw model weights memory consumption by @HuiGao-NV in #13760
  • [None][perf] Drop cubin and Eliminate ~6s FMHA JIT recompile in eager generation by aligning kernel selection with CUDA graph warmup by @yunruis in #13505
  • [https://nvbugs/5615248][fix] Reduce beam-search prefill->decode handoff cost by @brb-nv in #13748
  • [None][chore] Update flashinfer-python from 0.6.9 to 0.6.10 by @yihwang-nv in #13746
  • [None][feat] Fuse GDN elementwise ops and split/transpose kernels by @Wong4j in #12966
  • [None][infra] Waive 3 failed cases for main in post-merge by @xinhe-nv in #13797
  • [None][chore] Update nvidia-cutlass-dsl version in visual_gen pyproject.toml by @yihwang-nv in #13642
  • [None][infra] Waive 3 failed cases for main in post-merge by @xinhe-nv in #13789
  • [None][feat] Update TRTLLM MoE cubins by @rosenrodt in #12440
  • [None][fix] Fix Autodeploy standalone package builder script tests by @bmarimuthu-nv in #13794
  • [#13320][fix] Propagate FlashMLA tokens_per_block override onto kv_cache_config by @eopXD in #13752
  • [None][test] Unset MPI related Env in local Perf Test Script by @chenfeiz0326 in #13795
  • [https://nvbugs/5615248][fix] Broader capture of piecewise cudagraph by @brb-nv in #13574
  • [...
Read more

Contributors

Superjomn, janbernloehr, and 110 other contributors
Loading