v0.9.0

TensorRT Edge-LLM 0.9.0 Release 2026-07-03

We are excited to announce the release 0.9.0 of TensorRT Edge-LLM!

TensorRT Edge-LLM 0.9.0 officially supports DGX Spark, expands speculative decoding with DFlash, adds Qwen3-Omni-30B-A3B nvfp4 end-to-end support, gemma4 e2b and e4b text inference, C++ audio preprocessing and server audio input for supported audio models. It also fixes a decode performance regression introduced in 0.8.0.

Breaking Changes

• Audio preprocessing has moved into C++ runtime/server paths for supported audio models. The Python tensorrt-edgellm-preprocess-audio package entry point is no longer present.
• NVFP4 MoE backend selection no longer uses --nvfp4-moe-backend or QuantConfig.nvfp4_moe_backend. Export target selection is now handled by EDGELLM_NVFP4_MOE_TARGET.
• Speculative decoding build artifacts now use generic names such as spec_base.engine and spec_draft.engine. Use --specBase and --specDraft; --eagleBase and --eagleDraft remain deprecated aliases.
• Speculative decoding runtime flags now prefer --specVerifySize; --specVerifyTreeSize remains available as a compatibility alias.
• Occasionally, FP8 ViT for some models might have NaN issues. If encountered, falling back to FP16 ViT will resolve the issue.

Key Features

• Officially supports NVIDIA DGX Spark.
• Added Gemma 4 E2B/E4B text inference support.
• Added DFlash speculative decoding support, including export, runtime decode integration, target KV materialization kernels, DFlash draft quantization, and Qwen3/Qwen3.5 validation coverage.
• Added Qwen3-Omni-30B-A3B NVFP4 support for Thinker and Talker quantization/export.
• Fixed a decode performance regression introduced in 0.8.0.
• Expanded Nemotron3 NVFP4 support for Jetson Thor.
• Added C++ audio decoding and mel preprocessing for multimodal audio models and server audio input.
• Improved NVFP4 MoE support for SM100/101/110, including E=256 experts, SM110 split FC1/FC2 CuTeDSL kernels, and SM12x fused MoE plugin coverage.
• Added XQA sliding-window support and head-dim 512 kernels across supported SM targets.

Other Important Features

Runtime and Performance

• Improved CuTeDSL attention performance and added FFPA-style Ampere-floor FMHA CuTeDSL kernels.
• Added Qwen3-Omni TTS standalone streaming and unified Talker/CodePredictor runtime handling.
• Fixed MTP speculative decoding configuration validation.
• Fixed an EAGLE/MTP accept-kernel race on nextTokenIdx.
• Fixed SSD prefill accuracy and SSD TMA bounds issues.
• Migrated llm_bench from legacy runners to EngineExecutor and removed dead legacy runtime code.

Export and Quantization

• Forced ONNX exports to use external data files.
• Fixed Qwen3.5 MoE GPTQ export and symmetric GPTQ checkpoints without zero points.
• Added Qwen3.6-35B-A3B NVFP4 export.
• Fixed InternVL3.5 visual weight-key prefix handling for quantized checkpoints.

Server and API

• Added experimental server audio input handling for OpenAI-compatible input_audio, audio_url, and local audio content forms.
• Added server response metadata fields such as created, model, and token usage placeholders.
• Expanded speculative decoding server wording and handling beyond EAGLE/MTP to include DFlash.
• Added TTS streaming latency reporting for time-to-first-codec and time-to-first-playable-audio.

Documentation

• Added DFlash speculative decoding documentation and supported draft-model mappings.
• Added a new Qwen3-Omni workflow covering quantization, export, engine build, text/audio/image input, and speech output.
• Updated MoE documentation for Nemotron3 NVFP4 and NVFP4 MoE export behavior.
• Updated ASR and server documentation for C++ audio preprocessing.
• Updated 0.8.0 performance data

NVIDIA Contributors

@fans-nv @nvluxiaoz @nvamberl @ruocheng-nv @poweiw @Jasper-NV @charllll @mahu888 @xinrzhang-star @JCalafato @ever-wong @zhijial-nvidia @willg-nv @levichen-nvidia @xiangg-nv @nv-samcheng @jhalabi-nv @nvmbreughe