Skip to content

Torch-TensorRT v2.13.0

Choose a tag to compare

View all tags
@zewenli98 zewenli98 released this 28 Jul 21:52
v2.13.0
64f0ed4
This commit was signed with the committer’s verified signature.
lanluo-nvidia 岚岚
GPG key ID: FB85EB5ED4A410FE
Verified
Learn about vigilant mode.

Torch-TensorRT 2.13.0 Linux x86-64 and Windows targets

PyTorch 2.13, CUDA 13.0/13.2, TensorRT 11.0, Python 3.10~3.13

Torch-TensorRT Wheels are available:

x86-64 Linux and Windows:
CUDA 13.0/13.2 + Python 3.10-3.13 is Available via PyPI

aarch64 SBSA Linux and Jetson Thor and Orin:
CUDA 13.0/13.2 + Python 3.10–3.13 + Torch 2.13 + TensorRT 11.0

Torch-TensorRT-RTX 2.13.0 Linux x86-64 and Windows targets

PyTorch 2.13, CUDA 13.0/13.2, TensorRT-RTX 1.5, Python 3.10~3.13

Torch-TensorRT-RTX Wheels are available:

x86-64 Linux and Windows:
CUDA 13.0/13.2 + Python 3.10-3.13 is Available via PyPI

CUDA 13.0/13.2 + Python 3.10-3.13 is also Available via Pytorch Index

New Features:

GQA/MQA and Decode-Phase Attention Support

Extends Group Query Attention (GQA) and Multi-Query Attention (MQA) alongside traditional Multi-Head Attention (MHA), enabling efficient inference for modern large language models. This release relaxes SDPA validator constraints to allow variable head-count configurations, adds native support for decode-phase attention where query and key sequences differ, and introduces comprehensive HLO-level test coverage across all three attention kernel variants (standard, flash, efficient) with multiple precision levels (fp16, bf16, fp32). These changes unlock optimized inference patterns for LLMs using GQA/MQA while maintaining full backward compatibility with existing MHA-based models.

FP8 Attention Quantization Support

Supports FP8 quantization of the attention in both TensorRT's IAttention path and decomposed attention path (decompose_attention=True). Working with NVIDIA ModelOpt, it detects when attention inputs (Q, K, V) are quantized to FP8 and configures TensorRT's native FP8 MHA kernel for optimal performance on FP8-capable hardware (Hopper and newer GPUs). A ViT quantization example demonstrates both IAttention and decomposed attention paths.

Caller-Selected CUDA Streams and Green Context Support

Enables TensorRT inference on caller-selected CUDA streams across both Torch-TensorRT and ExecuTorch. The PyTorch runtime now respects active non-default streams, while ExecuTorch introduces a scoped CudaStreamGuard API. This supports CUDA Green Contexts, NCCL streams, and custom GPU scheduling without changing default-stream behavior. CUDA Graphs are recaptured when streams change, and completion events safely protect asynchronous execution-context reuse.

TensorRT 11 Support

Torch-TensorRT 2.13 adds support for NVIDIA TensorRT 11, with release packages updated to TensorRT 11.0. The integration has been updated for TensorRT 11 APIs across compilation, runtime execution, packaging, and distributed inference.
Highlights:

  • TensorRT 11.0 support for Python and C++ workflows.
  • Updated CUDA 12 and CUDA 13 package dependencies for TensorRT 11.
  • Updated distributed and NCCL-based operators for TensorRT 11.
  • Improved scaled dot-product attention accuracy using TensorRT 11 fixes.
  • Updated build configurations, CI coverage, and prebuilt release artifacts for TensorRT 11.
    TensorRT 11 is the default supported TensorRT version for Torch-TensorRT 2.13 release packages.

ExecuTorch Support

This release delivers a pure ExecuTorch TensorRT backend for deploying TensorRT-accelerated models in ExecuTorch .pte format. The native runtime path no longer depends on Torch or LibTorch, enabling lightweight C++ deployment using only ExecuTorch and TensorRT runtime components.
Users can continue exporting TensorRT-compiled ExportedProgram and FX models with:
torch_tensorrt.save(model, "model.pte", output_format="executorch")
Highlights:

  • Pure ExecuTorch native backend with no Torch or LibTorch runtime dependency.
  • Support for TensorRT output allocators and data-dependent output shapes.
  • Support for embedding TensorRT engine payloads larger than 2 GiB in .pte files.
  • Target-device selection during ExecuTorch export.
  • Continued support for static-shape and TensorRT profile-based dynamic-shape models.
  • Updated C++ TensorRT ExecuTorch backend and reference runner for Python-free inference.

ExecuTorch Target-Device Metadata

Adds target_device metadata to TensorRT-delegated ExecuTorch partitions so device placement propagates into exported .pte files. Single-engine exports derive the CUDA index from the engine itself, while explicit caller overrides remain supported. This prepares exported programs for CUDA-aware memory planning and direct device-resident delegate buffers.

Caller-Selected CUDA Streams and Green Contexts

Introduces CudaStreamGuard, allowing the ExecuTorch TensorRT delegate to execute on a caller-selected CUDA stream. This enables CUDA Green Contexts, custom GPU scheduling, and asynchronous device-resident execution while preserving existing default behavior.

Large TensorRT Engine Export for ExecuTorch

Supports exporting TensorRT engines larger than 2 GB to ExecuTorch .pte files. Serialized engines are now stored as CPU tensor attributes instead of inline base64 Python string literals, avoiding CPython’s string-size limit during FX graph recompilation. This enables deployment of large models through ExecuTorch without changing the underlying TensorRT engine format.

Unified Python and C++ Runtime with Cross-Runtime Serialization

Unifies the Python and C++ TensorRT runtimes behind TorchTensorRTModule and a common execution operation. Torch-TensorRT now automatically uses the C++ runtime when available and falls back to the pure-Python implementation otherwise (Python-only build). Engine constants are normalized during serialization so models saved under either runtime can be loaded and executed in Python-only or C++-enabled runtime.

Simplified Refit Pipeline

Removes the heuristic cached weight-name mapping previously stored in compiled modules and engine-cache entries. Refit now reconstructs the TensorRT-to-PyTorch weight mapping directly from the new graph on every operation, eliminating stale or incorrect cache matches and the associated fallback logic. Weight refitting and engine caching remain supported, and existing cache entries continue to load with their obsolete mapping metadata ignored.

Multi-Device TensorRT (MD-TRT) Distributed Inference

Adds native multi-device inference built on TensorRT 11's multi-device (MD) engines, enabling TensorRT-accelerated collectives across GPUs driven through standard torch.distributed APIs. This release adds converters for All-to-All, Scatter, and Gather (alongside existing all-reduce, all-gather, and reduce-scatter), and integrates TensorRT's NCCL path with PyTorch's process groups via a distributed_context API that routes each collective to the correct communicator across default and non-default (subgroup) process groups. NCCL library discovery now shares the same libnccl instance PyTorch loads — on TensorRT 11.0+ the runtime loads libnccl.so.2 directly, matching PyTorch's library and removing the need for manual LD_LIBRARY_PATH or symlink workarounds — so distributed, tensor-parallel, and TP-subgroup inference works out of the box.

Shared Dynamic Dimensions Across Inputs

Adds a shared_dims argument to torch_tensorrt.Input for models whose dynamic axes must be equal across multiple inputs (for example, a batch dimension shared by input_ids and attention_mask). Axes given the same name via Input(shared_dims={axis: name}) are exported as a single torch.export.Dim, so the equality constraint is satisfied automatically and torch.export no longer raises a ConstraintViolationError from treating each input's dynamic axis as independent. New shared_dynamic_dims_example documents the end-to-end usage.

What's Changed

New Contributors

Full Changelog: v2.12.1...v2.13.0

Contributors

shoumikhin, narendasan, and 13 other contributors
Assets 28
Loading