Skip to content

v1.1.0

Choose a tag to compare

View all tags
@HuiGao-NV HuiGao-NV released this 19 Dec 13:55
v1.1.0
48b7b5d
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Know Issue

  • If users create project with tensorrt-llm==1.1.0 in pyproject.toml file as dependency as below:

    dependencies = [
  "tensorrt-llm==1.1.0",
]

    when users install project dependencies with command uv sync, error will happend with message:

    No solution found when resolving dependencies for split (markers: python_full_version >= '3.13' and sys_platform == 'darwin'):
╰─▶ Because patchelf==0.18.0.0 was yanked (reason: https://github.com/mayeut/patchelf-pypi/issues/87) and tensorrt-llm==1.1.0 depends on patchelf==0.18.0, we can
    conclude that tensorrt-llm==1.1.0 cannot be used.
    And because your project depends on tensorrt-llm==1.1.0, we can conclude that your project's requirements are unsatisfiable.".

    That's because patchelf 0.18.0 was yanked by author.

    A valid work around for this issue is to add block in pyproject.toml:

    [tool.uv]
 override-dependencies = [
 "patchelf==0.17.2.4",
]

Key Features and Enhancements

  • Model Support
    • Add model supports of GPT-OSS, Hunyuan-Dense (contribution from @sorenwu), Hunyuan-MoE (contribution from @qianbiaoxiang), Seed-OSS (contribution from @Nekofish-Li).
  • Features
    • Connector API: Introduced a new KV Cache Connector API for state transfer in disaggregated serving.
    • Reuse & Offloading: Enabled KV cache reuse for MLA (Multi-Head Latent Attention) and added examples for host offloading.
    • Salting: Implemented KV cache salting for secure cache reuse.
    • Guided Decoding Integration: Enabled guided decoding to work in conjunction with speculative decoding (including 2-model and draft model chunked prefill).
    • Eagle: Added multi-layer Eagle support and optimizations.
    • CuteDSL: Integrated CuteDSL NVFP4 grouped GEMM for Blackwell.
    • B300/GB300: Added support for B300/GB300.
  • Documentation
    • Deployment Guides: Added comprehensive deployment guides for GPT-OSS, DeepSeek-R1, and VDR 1.0.
    • Feature Documentation: Created new documentation for KV Cache Connector, LoRA feature usage, and AutoDeploy.
    • Tech Blogs: Published blogs on “Combining Guided Decoding and Speculative Decoding” and “ADP Balance Strategy”.
    • Quick Start: Refined Quick Start guides with new links to ModelOpt checkpoints and updated installation steps (Linux/Windows).
    • API Reference: Enhanced LLM API documentation by explicitly labeling stable vs. unstable APIs.
    • Performance: Updated online benchmarking documentation and performance overview pages.
    • Examples: Refined Slurm examples and added K2 tool calling examples.
  • Infrastructure Changes
    • The base Docker image for TensorRT-LLM: nvcr.io/nvidia/pytorch:25.10-py3.
    • The base Docker image for TensorRT-LLM Backend: nvcr.io/nvidia/tritonserver:25.10-py3.
    • The dependent public PyTorch version: 2.9.0.
    • The dependent NVIDIA ModelOpt version: 0.37.
    • The dependent xgrammar version: 0.1.25.
    • The dependent transformers version: 4.56.0.
    • The dependent NIXL version: 0.5.0.
  • API Changes
    • Breaking Change: The C++ TRTLLM sampler is now enabled by default, replacing the legacy implementation.
    • KV Cache Connector API: Introduced a new KV Cache Connector API.
    • Standardized topk logprob returns across TRT and PyTorch backends.
    • Added stable labels to arguments in the LLM class to better indicate API stability.
    • Wait and Cancel API: Added tests and support for handling non-existent and completed request cancellations in the executor.
  • Fixed multiple Issues
  • illegal memory access, weight loading issues for DeepSeek-R1 W4A8, CUDA graph warmup issues with speculative decoding, memory leaks
  • Known Issues
    • GB300 Multi-Node: Support for GB300 in multi-node configurations is currently in beta and not fully validated in this release. GB300 multi-node configurations have been validated in 1.2.0rc4+.

What's Changed

New Contributors

Full Changelog: v1.0.0...v1.1.0

Contributors

WilliamTambellini, Superjomn, and 166 other contributors
Assets 2
Loading