v0.9.1

[nvluxiaoz]

TensorRT Edge-LLM 0.9.1 Release 07/23/2026

We are excited to announce the release 0.9.1 of TensorRT Edge-LLM!

TensorRT Edge-LLM 0.9.1 expands Gemma 4 multimodal support, DFlash, paged KV cache, and OpenAI-compatible generation controls while improving runtime performance and stability.

Key Features

• Added Gemma 4 E2B, E4B, 12B, 26B-A4B (nvfp4), 31B full modality support with MTP.
• Expanded DFlash with DDTree for Qwen3 and Qwen3.5 hybrid models and draft vocabulary reduction.
• Added paged KV-cache prefill and paged XQA decode support.
• Added online GPU filter-bank preprocessing for Whisper/Qwen3 audio models and Parakeet/Nemotron-Omni, plus Qwen vision video preprocessing.
• Added logprobs across vanilla and speculative decoders and OpenAI-compatible logit_bias requests.

Other Important Features

Runtime and Performance

• Added INT4-FP16 CuTe DSL GEMM/GEMV kernels for SM80+.
• Added CuTe DSL support for the FP16 MoE plugin, including 128/256-expert configurations.
• Improved FMHA for short sequences, partial prefill, and head dimension 256.
• Improved attention configuration validation to fail unsupported configurations cleanly.
• Fixed implicit stream synchronization, streamed engine deserialization, logger teardown, duplicate symbols, and FP8 long-context accuracy.

Export and Quantization

• Added named, per-modality calibration dataset selection
• Added Qwen3-Omni/Qwen3-TTS CodePredictor FP8 quantization.

Server and API

• Added speculative-decoding engine-directory routing through the Python LLM API.
• Fixed greedy sampling fallback handling for speculative decoding.

Examples

• Fixed some llm_bench issues.

NVIDIA Contributors

@nvluxiaoz @nvamberl @xiangg-nv @willg-nv @mahu888 @nv-samcheng @duofant @Caohanwen0 @JCalafato @zhazhang-nv @zhaoyuanh-nvidia @zhijial-nvidia @ever-wong @Jasper-NV @wanghr323 @jhalabi-nv @levichen-nvidia @sunghyunp-nvidia @nvyocox @fans-nv @ruocheng-nv @yuanyao-nv @duanyaqi

---

This discussion was created from the release v0.9.1.