Skip to content

v0.12.0

Latest

Choose a tag to compare

@jhalabi-nv jhalabi-nv released this 13 Nov 23:42
513dcd1

We are excited to announce the release 0.12.0 of TensorRT Edge-LLM!

TensorRT Edge-LLM 0.12.0 introduces initial Safety Runtime support for InternVL3, compatible with QNX Standard. This release also expands single-device in-flight batching, adds Qwen3.8-Flash-Next and Nemotron-3.5-Bolt-74B-A9B support, and broadens speculative decoding with EAGLE-4, Domino, LiLiCorr, and JetSpec.

Key features

  • Bringup of the first Safety Runtime for InternVL3, compatible with QNX Standard
  • Expanded opt-in, single-device in-flight batching with compact mixed prefill/decode execution, scheduling policies, per-request sampling and generation limits, and speculative decoding
  • Added Qwen3.8-Flash-Next support with cached per-layer embeddings and Nemotron-3.5-Bolt-74B-A9B support with IQ2 MoE and Q8_0 Linear
  • Added EAGLE-4 and greedy-only Domino speculative decoding, LiLiCorr drafts for Nemotron-3.5-Bolt-74B-A9B, and JetSpec support for selected MoE models

Models and generation

  • Expanded compatible speculative-draft support across ONNX export and checkpoint-direct builds, and broadened direct-builder support for pre-quantized GPTQ, AutoRound, and compressed-tensors checkpoints
  • Added FP16 MiniMax-H3 text-to-video generation
  • Added GR00T N1.7 FP16 policy inference for the LIBERO checkpoint and libero_sim embodiment
  • Added a checkpoint-direct pi0.5 engine builder and expanded Cosmos3 policy deployment guidance
  • Added MTP support for dense Qwen3 checkpoints and DFlash2 tree decoding
  • Added jump-forward decoding for guided generation
  • Added Gemma 4 Flax checkpoint conversion example

Runtime and performance

  • Expanded context reuse to hybrid DFlash, JetSpec, DSpark, and Domino deployments, speculative requests with image or audio input
  • Added Alpamayo paged action inference at TP=1
  • Added FP8 KV-cache storage support for Ampere, including Jetson Orin, with attention computed in FP16
  • Added separate prefill and decode scale tables for skip-softmax attention, plus deployment and calibration guidance
  • Added NVFP4 token-embedding quantization and native lookup and FP8 per-layer embedding quantization. Thanks @mihai-chiorean contribution #249.
  • Refactored Omni runtime into staged execution
  • Added TensorRT refit-based weightless engine

Installation and workflows

  • Migrated Python bindings to nanobind ABI3 to support Python 3.10 and later
  • Replaced the previous Python extras with capability-specific runtime, builder, onnx, quantize, and vocab extras; moved the builder and server into tensorrt_edgellm
  • Updated PyTorch from 2.13.0 to 2.14.0 and Transformers from 5.14.1 to 5.17.0
  • Added the OpenAI-compatible /v1/completions endpoint and an HTTP benchmark CLI
  • Added PhysicalAI-AV request preparation guide for Alpamayo-R1

Compatibility and fixes

  • Fixed Gemma 4 image-token budgets and calibrated input-scale loading for packed NVFP4 experts
  • Fixed checkpoint-direct builds for ModelOpt mixed-precision checkpoints containing FP8 embeddings and DSpark drafts with nested metadata
  • Removed the legacy system-prompt KV cache; use context-cache reuse for repeated prefixes
  • Fixed calibrated NVFP4 precision selection, quantized speculative-draft export, and Muse-Glimmer checkpoint loading
  • Fixed sliding-window context reuse by selecting full KV storage
  • Fixed assistant tool-call argument rendering and sampling for requests without an explicit seed
  • Fixed Qwen3-Omni FP8 KV-cache option validation and TensorRT-RTX engine targeting for the current GPU

NVIDIA Contributors

@nvamberl @duofant @ever-wong @Jasper-NV @jaybajajnvidia @jhalabi-nv @JCalafato @kevinfu1 @chemudupatiks @levichen-nvidia @nvluxiaoz @mahu888 @nvmbreughe @nv-samcheng @MadFunMaker @willg-nv @xiangg-nv @nvxingkaiz @xinrzhang-star @duanyaqi @yen-shi @nvyocox @yongzhiz @yuanyao-nv @wyw1267 @zhazhang-nv @zhaoyuanh-nvidia @zhijial-nvidia