Skip to content

v1.3.0rc22

Pre-release
Pre-release

Choose a tag to compare

View all tags
@mikeiovine mikeiovine released this 22 Jul 22:41
v1.3.0rc22
8ba9340

  • Known Issues

    • torch.compile crashes in the PyTorch compilation backend
      • Several multi-GPU accuracy paths fail with KeyError in the remove_copy pass when torch_compile=True (e.g. DeepSeek-V3-Lite bf16/FP8/NVFP4; Llama-3.1-8B-Instruct FP8 with FlashInfer). Workaround: disable torch.compile for affected configs DeepSeek-V3.2 FP8 block-scale OOM on H200.
    • test_fp8_blockscale configs can OOM on DGX H200 under default / disable_skip_indexer settings.
    • Mixtral FP8 MoE + multi-LoRA tactic mismatch
      • Routed-expert FP8 multi-LoRA can hit a Cutlass MoE assertion (GEMM2 finalize-fusion tactic selected while the runner is not configured for it), in both eager and CUDA-graph modes.
    • Kimi K2.5 disagg KV cache transfer can fail on GB300.

  • Model Support

    • Add DeepSeek-V4-Pro curated configurations (#15919)
    • Support mixed image and video modality requests for Qwen3-VL (#15852)

  • VisualGen

    • Add ParallelVAE_TrtllmWan for native Wan VAE support (#16066)
    • Optimize video hashing performance (#16340)
    • Fix Qwen Image CUDA graphs (#16391)

  • API

    • Add automatic per-model transceiver runtime selection, a breaking change (#16164)
    • Add bad_words support to TorchSampler (#16052)
    • Remove C++ modules for the legacy TensorRT backend, a breaking change (#16369)

  • Feature

    • Add Laguna DFlash drafter support (#15666)
    • Add DeepSeek DSpark drafter support (#15808)
    • Allow FP4 KV cache with non-FP4 Mamba state (#16304)
    • Extend rejection sampling to one-model speculative decoding modes (#15775)
    • Add SM121 MLA cache reuse support (#15434)
    • Integrate ModelExpress checkpoint loading (#15641)
    • Add a disaggregated coordinator and multi-process orchestrator fleet (#15905)

  • Fix

    • Fix MoE expert-parallel divisibility checks for EPLB configurations (#15410)
    • Fix Gemma 4 illegal memory access when the maximum sequence length does not exceed the sliding-window size (#16099)
    • Disable KV-cache reuse for MiniMax-M2 (#16383)
    • Avoid loading the multimodal encoder for text-only trtllm-bench runs (#16250)
    • Improve the disaggregated bounce-buffer workflow (#16116)
    • Clamp piecewise CUDA graph captures to the reachable ceiling (#16256)
    • Persist the DSA CuTe DSL paged-MQA-logits output buffer to prevent stale-pointer illegal memory access (#16307)
    • Fix double rotation and GPT-OSS pairing in unfused RoPE for YaRN models (#16378)
    • Honor GenerationResult timeouts (#16453)
    • Fix the greedy-sampling path when use_lm_head_tp_in_adp is enabled (#16440)
    • Fix DeepSeek V3 disaggregated conditional-test configuration (#16528)
    • Optimize Gemma 4 performance (#16509)
    • Derive disaggregated KV-transfer layer offsets from physical slot order (#16429)
    • Make one-model speculative-decoding attention-metadata save and restore exception-safe (#16382)
    • Fix DeepSeek V4 KV-cache warmup and related issues (#16466)
    • Fix intermittent CUDA mapping errors (#16226)
    • Remove unsupported INT8 choices from trtllm-bench --quantization (#16245)
    • Reduce per-step host preparation overhead in the PyTorch executor decode path (#16313)

  • Documentation

    • Add the DeepSeek V4 optimization technical blog (#16539)

  • Benchmark

    • Migrate Kimi performance-sanity tests to Transceiver v2 (#15966)
    • Add DeepSeek-V4-Pro performance-sanity cases on GB300 (#16540)
    • Increase input and output lengths in LLM performance tests (#16538)

  • Test & Infra

    • Upgrade dependencies for the NGC PyTorch 26.05 stack (#15087)
    • Enable B300 CI stages (#16359)
    • Fix the multi-node DGX Spark test (#16209)
    • Stabilize compressor BF16 tolerance checks (#16376)
    • Split unquantized MoE backend tests to avoid in-process IMA cascades (#16068)
    • Prepare Python and test support for the C++ KV cache manager v2 backend (#16218)
    • Improve function-level code coverage for subprocess data (#16229)
    • Upgrade NIXL to v1.3.1 (#16194)
    • Detect worker out-of-memory errors and crashes during disaggregated startup (#16456)
    • Add a --keep-workspace pytest option (#16451)
    • Promote disaggregated performance-sanity tests to pre-merge functional verification (#16351)
    • Remove 1K/1K performance-sanity cases from CI (#16535)

What's Changed

  • [https://nvbugs/6245862][fix] Fix MoE EP divisibility check for EPLB configurations by @JacobHu-NV in #15410
  • [TRTLLMINF-127][infra] Upgrade dependencies for NGC PyTorch 26.05 stack by @EmmaQiaoCh in #15087
  • [None][test] Waive 7 failed cases for main in QA CI by @trtllm-agent in #16042
  • [None][feat] Add Laguna DFlash drafter support by @joerowell in #15666
  • [None][infra] Enable B300 stages by @yiqingy0 in #16359
  • [https://nvbugs/6428144][test] Unwaive GB300 DeepSeek-R1 gen-only disagg PerfSanity by @chienchunhung in #16267
  • [None][chore] Remove disagg-devs co-ownership of mla.py in CODEOWNERS by @Tabrizian in #16385
  • [TRTLLM-12721][fix] Add gated C++ NIXL in-flight cancellation and safe cleanup by @chienchunhung in #15238
  • [https://nvbugs/6374873][fix] Allow fp4 KV Cache + non-FP4 Mamba State by @VALLIS-NERIA in #16304
  • [https://nvbugs/6435642][fix] detect killed MPI executor workers by @chienchunhung in #16338
  • [None][test] Fix the multinode test case on DGX-Spark(perf skipped and func hang) by @JennyLiu-nv in #16209
  • [TRTLLM-14024][feat] Prune CuTe DSL NVFP4 GEMM autotuner tactics with nvMatmulHeuristics by @peaceh-nv in #15909
  • [TRTLLM-12373][feat] RMSNorm nvfp4 quant fusion for DS V3.2 / Kimi-K2.5 by @JunyiXu-nv in #14848
  • [None][test] Refine CBTS logic for handling -Perf- stages by @yufeiwu-nv in #16328
  • [None][infra] Assign KV cache manager v2 test ownership by @jiaganc in #16372
  • [None][fix] fix missing torch_dtype_to_binding by @bo-nv in #16401
  • [https://nvbugs/6426860][fix] Stabilize compressor BF16 tolerance by @mingyangHao in #16376
  • [https://nvbugs/6403909][test] Remove attention backend test waiver by @yuxianq in #16400
  • [None][chore] Rename AutoDeploy standalone package to Paragraf by @bmarimuthu-nv in #16324
  • [None][test] Split test_moe_backend TRTLLM by quant=None to avoid in-process IMA cascade by @leslie-fang25 in #16068
  • [None][fix] Fix Gemma4 illegal memory access when max_seq_len is at most the sliding window size by @Hudayday in #16099
  • [https://nvbugs/6396422][fix] Disable kv-cache reuse for MiniMax-M2 by @tcherckez-nvidia in #16383
  • [None][test] Waive 1 failed cases for main in QA CI by @trtllm-agent in #16412
  • [None][fix] align KV slice token_range.end with transferred block count by @chuangz0 in #15824
  • [None][feat] Python transceiver support cpp cache manager + offload by @chuangz0 in #15245
  • [None][chore] add disagg-devs as code owner for disaggregated tests by @Shixiaowei02 in #16415
  • [TRTLLM-13212][refactor] Unify sampler ops and clean up dead code in TorchSampler by @zhaoyangwang-nvidia in #16365
  • [https://nvbugs/6260897][fix] Relax line-589 assertion to accept either IN_PROGRESS or SUCCESS (FAILURE… by @tensorrt-cicd in #15121
  • [None][infra] Waive 2 failed cases for main in pre-merge 47854 by @trtllm-agent in #16423
  • [https://nvbugs/6445332][test] Remove test waivers for nvbug 6445332 (code fix merged in #16326) by @trtllm-agent in #16316
  • [None][infra] Tail Slurm job logs when job is no longer active. by @mzweilz in #15455
  • [None][chore] KVCacheManagerV2: Python and test preparation for a C++ backend by @lowsfer in #16218
  • [https://nvbugs/6283537][fix] Unwaive TestQwen3_5_4B::test_bf16 by @amukkara in #16392
  • [None][infra] Waive 1 failed cases for main in pre-merge 47597 by @trtllm-agent in #16361
  • [None][infra] Waive 1 failed cases for main in pre-merge 47847 by @trtllm-agent in #16443
  • [None][infra] Waive 1 failed cases for main in pre-merge 47847 by @trtllm-agent in #16442
  • [https://nvbugs/6269388][fix] Raise error to enforce HMAC encryption in IPC by @yibinl-nvidia in #16249
  • [None][infra] Waive 1 failed cases for main in pre-merge 47953 by @trtllm-agent in #16450
  • [None][infra] Waive 1 failed cases for main in pre-merge 47930 by @trtllm-agent in #16452
  • [TRTLLM-12352][feat] add post-transform capability profiles by @chienchunhung in #16238
  • [https://nvbugs/6405760][fix] Do not load the multimodal encoder for text-only trtllm-bench runs by @thorjohnsen in #16250
  • [None][perf] Optimize Video Hashing Speed by @aswinvisva in #16340
  • [None][chore] Waive failing torch-compile test by @brb-nv in #16461
  • [https://nvbugs/6414762][fix] Minor fixes for helix tests by @brb-nv in #15999
  • [TRTLLM-14123][feat] add ParallelVAE_TrtllmWan for native Wan VAE by @taianz-nv in #16066
  • [None][feat] BREAKING: add per-model transceiver runtime auto selection by @nv-xtf in #16164
  • [None][fix] Skip commented-out lines in Groovy stage parsing by @yufeiwu-nv in #16414
  • [None][infra] Waive 14 failed cases for main in post-merge 2839 by @trtllm-agent in #16465
  • [None][perf] fuse gdn post-conv split, QK norm and gating and RMSNorm with FP8 quantization by @nv-guomingz in #16299
  • [TRTLLM-12838][infra] enhance function-level code coverage for catching subprocess data by @crazydemo in #16229
  • [None][infra] Waive 1 failed cases for main in pre-merge 48013 by @trtllm-agent in #16470
  • [TRTLLM-6905][feat] MM encoder cache by @2ez4bz in #15734
  • [TRTLLM-13614][fix] Enhance the dis-agg bounce buffer workflow by @Shixiaowei02 in #16116
  • [None][infra] Auto-label PRs by component from CODEOWNERS by @zhenhuaw-me in #16368
  • [None][feat] Add DeepSeek-V4-Pro curated configs by @lfr-0531 in #15919
  • [https://nvbugs/6327143][chore] Unwaive test cases by @marinayanov in #16367
  • [https://nvbugs/6412108][fix] AutoDeploy:Shard shared Qwen3.5 experts by @greg-kwasniewski1 in #16286
  • [TRTLLM-13639][perf] Migrate Kimi perf-sanity tests to Transceiver v2 by @Shixiaowei02 in #15966
  • [None][infra] Upgrade NIXL to v1.3.1 by @zackyoray in #16194
  • [TRTLLM-13321][feat] Extend rejection sampling to one-model speculative decoding (MTP/DRAFT_TARGET/PARD/DFLASH) by @zhaoyangwang-nvidia in #15775
  • [None][infra] Waive 3 failed cases for main in pre-merge 48086 by @trtllm-agent in #16495
  • [https://nvbugs/6435642][fix] handle session reuse worker registration by @chienchunhung in #16444
  • [TRTLLMINF-99][infra] Add SLURM frontend failover to L0 by @dpitman-nvda in #15674
  • [None][infra] Handle SBSA image tags for x86 CI agents by @chzblych in #16503
  • [https://nvbugs/6403920][fix] Move allreduce strategy test to smoke, use full-warp hidden_size by @MrGeva in #16389
  • [None][fix] Fix Qwen image CUDA graph with CFG by @yibinl-nvidia in #16391
  • [None][chore] Update API update missed in Dynamo test by @brb-nv in #16512
  • [None][test] disagg startup: detect and fail fast on worker OOM/crash by @brnguyen2 in #16456
  • [https://nvbugs/6422339][fix] unwaive disagg tests by @bo-nv in #16474
  • [None][test] add --keep-workspace pytest option to skip workspace cleanup by @brnguyen2 in #16451
  • [None][infra] Waive 1 failed cases for main in pre-merge 48139 by @trtllm-agent in #16519
  • [TRTLLM-14054][perf] Reduce per-step host preparation overhead in the PyTorch executor decode path by @kaiyux in #16313
  • [https://nvbugs/6404567][fix] Clamp piecewise cudagraph captures to the reachable ceiling instead of force-appending it by @dongfengy in #16256
  • [https://nvbugs/6272673][chore] Unwaive DeepSeek V3 Lite NVFP4 test by @yuxianq in #16398
  • [None][fix] DSA CuteDSL paged-MQA-logits: persistent output buffer (CUDA-graph stale-pointer IMA) by @Tabrizian in #16307
  • [None][fix] Fix unfused RoPE for yarn models: double rotation and GPT-OSS pairing by @Hudayday in #16378
  • [None][test] Update LLM performance test cases to use increased input/output lengths by @yufeiwu-nv in #16538
  • [https://nvbugs/6336747][fix] Honor GenerationResult timeout by @2ez4bz in #16453
  • [https://nvbugs/6460072][fix] Split the caller — for use_lm_head_tp_in_adp and is_all_greedy_sample, keep… by @trtllm-agent in #16440
  • [None][infra] Add blossom-ci authorized users by @yiqingy0 in #16541
  • [None][doc] Add DeepSeek-V4 optimization tech blog by @lfr-0531 in #16539
  • [https://nvbugs/6329052][fix] Change the config for DSV3 disagg conditional test by @Shixiaowei02 in #16528
  • [None][test] Promote disagg perf sanity tests to pre-merge for functional verification by @chenfeiz0326 in #16351
  • [None][test] Remove all 1k1k perf-sanity cases from CI by @chenfeiz0326 in #16535
  • [None][perf] Prewarm DeepGEMM paged_mqa_logits_metadata JIT buckets by @chenfeiz0326 in #16178
  • [https://nvbugs/6272644][fix] Stabilize and unwaive multi-GPU VisualGen LPIPS tests by @chang-l in #15730
  • [None][feat] Qwen3-VL: support mixed image+video modality requests by @aswinvisva in #15852
  • [#15344][fix] Add SM121 MLA cache reuse support by @peter941221 in #15434
  • [None][perf] Various Gemma4 related perf fixes by @2ez4bz in #16509
  • [None][feat] Add DeepSeek DSpark speculative decoding by @longlee0622 in #15808
  • [TRTLLM-12352][feat] integrate ModelExpress checkpoint loading by @chienchunhung in #15641
  • [TRTLLM-14509][feat] Disagg coordinator + multi-process orchestrator fleet by @reasonsolo in #15905
  • [None][chore] update allowlist 2026-07-17 by @tburt-nv in #16547
  • [None][feat] Append git commit hash to version for editable installs by @dc3671 in #16395
  • [None][fix] disagg: derive KV transfer layer offset from physical slot order by @lowsfer in #16429
  • [None][fix] Copy all AutoDeploy tests to Paragraf by @bmarimuthu-nv in #16408
  • [https://nvbugs/6476233][test] waive DeepSeek V3.2 tests on DGX H200 by @chzblych in #16573
  • [None][test] Consolidate test coverage for helix by @brb-nv in #16570
  • [https://nvbugs/6442074][fix] Make one-model spec-dec attn-metadata save/restore exception-safe by @dongfengy in #16382
  • [TRTLLM-14026][feat] BREAKING: Remove C++ modules for legacy TRT backend by @Wanli-Jiang in #16369
  • [TRTLLMINF-188][infra] Require approval for broad post-merge bot runs by @chzblych in #16318
  • [TRTLLM-5311][infra] Periodically query OpenSearch for test duration and push to main automatically by @EmmaQiaoCh in #13804
  • [https://nvbugs/6440089][test] Rework disagg mixed-stress accuracy metric and harden the test by @brnguyen2 in #16445
  • [TRTLLM-13235][feat] Support bad_words in TorchSampler by @zhaoyangwang-nvidia in #16052
  • [None][test] Add opt-in background prefetch of test MPI sessions and model page cache by @sunnyqgg in #15908
  • [None][fix] Fix DeepSeek V4 KV cache warmup handling and serveral other issues by @liji-nv in #16466
  • [None][infra] Waive 1 failed cases for main in pre-merge 48569 by @trtllm-agent in #16582
  • [None][test] Add DeepSeek-V4-Pro perf sanity cases on GB300 by @chenfeiz0326 in #16540
  • [https://nvbugs/6442074][fix] Unbreak main: DSparkWorker forward rename + test stub _forward_impl by @dongfengy in #16579
  • [TRTLLM-13409][fix] make proxy shutdown non-blocking when the engine is dead by @JunyiXu-nv in #16312
  • [TRTQA-3274][test] Waive 3 failed cases for main in QA CI by @trtllm-agent in #16585
  • [TRTQA-3273][test] Waive 4 failed cases for main in QA CI by @trtllm-agent in #16584
  • [https://nvbugs/6426852][fix] Fix intermittent cuda mapping error by @shuyixiong in #16226
  • [None][infra] Waive 5 failed cases for main in post-merge 2844 by @trtllm-agent in #16587
  • [None][fix] remove unsupported INT8 from trtllm-bench --quantization choices by @lonexreb in #16245
  • [https://nvbugs/6316983][chore] Unwaive TestQwen2_5_VL_7B::test_auto_dtype by @yihwang-nv in #16534
  • [None][infra] Waive B300 Nemotron SuperV3 MTP and Mistral Large3 timeout cases by @jieli-matrix in #16588

New Contributors

Full Changelog: v1.3.0rc21...v1.3.0rc22

Contributors

reasonsolo, chienchunhung, and 59 other contributors
Assets 2
Loading