Repository navigation
We are excited to announce the release 0.12.0 of TensorRT Edge-LLM!
TensorRT Edge-LLM 0.12.0 introduces initial Safety Runtime support for InternVL3, compatible with QNX Standard. This release also expands single-device in-flight batching, adds Qwen3.8-Flash-Next and Nemotron-3.5-Bolt-74B-A9B support, and broadens speculative decoding with EAGLE-4, Domino, LiLiCorr, and JetSpec.
Key features
- Bringup of the first Safety Runtime for InternVL3, compatible with QNX Standard
- Expanded opt-in, single-device in-flight batching with compact mixed prefill/decode execution, scheduling policies, per-request sampling and generation limits, and speculative decoding
- Added Qwen3.8-Flash-Next support with cached per-layer embeddings and Nemotron-3.5-Bolt-74B-A9B support with IQ2 MoE and Q8_0 Linear
- Added EAGLE-4 and greedy-only Domino speculative decoding, LiLiCorr drafts for Nemotron-3.5-Bolt-74B-A9B, and JetSpec support for selected MoE models
Models and generation
- Expanded compatible speculative-draft support across ONNX export and checkpoint-direct builds, and broadened direct-builder support for pre-quantized GPTQ, AutoRound, and compressed-tensors checkpoints
- Added FP16 MiniMax-H3 text-to-video generation
- Added GR00T N1.7 FP16 policy inference for the LIBERO checkpoint and libero_sim embodiment
- Added a checkpoint-direct pi0.5 engine builder and expanded Cosmos3 policy deployment guidance
- Added MTP support for dense Qwen3 checkpoints and DFlash2 tree decoding
- Added jump-forward decoding for guided generation
- Added Gemma 4 Flax checkpoint conversion example
Runtime and performance
- Expanded context reuse to hybrid DFlash, JetSpec, DSpark, and Domino deployments, speculative requests with image or audio input
- Added Alpamayo paged action inference at TP=1
- Added FP8 KV-cache storage support for Ampere, including Jetson Orin, with attention computed in FP16
- Added separate prefill and decode scale tables for skip-softmax attention, plus deployment and calibration guidance
- Added NVFP4 token-embedding quantization and native lookup and FP8 per-layer embedding quantization. Thanks @mihai-chiorean contribution #249.
- Refactored Omni runtime into staged execution
- Added TensorRT refit-based weightless engine
Installation and workflows
- Migrated Python bindings to nanobind ABI3 to support Python 3.10 and later
- Replaced the previous Python extras with capability-specific runtime, builder, onnx, quantize, and vocab extras; moved the builder and server into tensorrt_edgellm
- Updated PyTorch from 2.13.0 to 2.14.0 and Transformers from 5.14.1 to 5.17.0
- Added the OpenAI-compatible /v1/completions endpoint and an HTTP benchmark CLI
- Added PhysicalAI-AV request preparation guide for Alpamayo-R1
Compatibility and fixes
- Fixed Gemma 4 image-token budgets and calibrated input-scale loading for packed NVFP4 experts
- Fixed checkpoint-direct builds for ModelOpt mixed-precision checkpoints containing FP8 embeddings and DSpark drafts with nested metadata
- Removed the legacy system-prompt KV cache; use context-cache reuse for repeated prefixes
- Fixed calibrated NVFP4 precision selection, quantized speculative-draft export, and Muse-Glimmer checkpoint loading
- Fixed sliding-window context reuse by selecting full KV storage
- Fixed assistant tool-call argument rendering and sampling for requests without an explicit seed
- Fixed Qwen3-Omni FP8 KV-cache option validation and TensorRT-RTX engine targeting for the current GPU
NVIDIA Contributors
@nvamberl @duofant @ever-wong @Jasper-NV @jaybajajnvidia @jhalabi-nv @JCalafato @kevinfu1 @chemudupatiks @levichen-nvidia @nvluxiaoz @mahu888 @nvmbreughe @nv-samcheng @MadFunMaker @willg-nv @xiangg-nv @nvxingkaiz @xinrzhang-star @duanyaqi @yen-shi @nvyocox @yongzhiz @yuanyao-nv @wyw1267 @zhazhang-nv @zhaoyuanh-nvidia @zhijial-nvidia