TensorRT Optimization

[nityanandmathur]

Is there any TensorRT optimised inference available?

[78Alpha]

No

[UmerrAhsan]

StyleTTS2 consists of several models and components that work together to generate audio. To optimize it using TensorRT, you first need to convert each model separately from PyTorch to ONNX.

Once converted, you can either:

• Run the ONNX models using ONNX Runtime with the TensorRT execution provider, or
• Convert the ONNX models directly into TensorRT format and perform inference using TensorRT's Python or C++ API.

From my experience with ablation studies, the decoder is the most resource-intensive component in StyleTTS2. If you aim for partial optimization, converting just the decoder from PyTorch to ONNX and running it in TensorRT can provide significant speed improvements. Alternatively, converting all models to ONNX and running them in ONNX Runtime with the TensorRT execution provider will also yield noticeable performance gains. This approach is feasible and I have tested it.

[nityanandmathur]

HI @UmerrAhsan. Could you please share the overall latencies of your ONNX model?

[UmerrAhsan]

Hi @nityanandmathur. I have ran the decoder model and predictor.text encoder model in tensorrt. It decreases my latency by over 50%. Also I have cached the style vectors from diffusion and style encoder before. After that, a single short sentence runs in under 100ms.

[moesaeed]

@UmerrAhsan Are you open to do a contract job about optimize StyleTTS2 using TensorRT? If so, Could we connect over email? shank187 at gmail dot com

[anadeprado4]

@UmerrAhsan - Could you (or anyone else) share an example of how to load and run the ONNX model using C++? Thank you :)

[Amarnath1906]

@UmerrAhsan Can you guide how did u convert onnx model to tensorrt? I am facing a lot of blockers with HIFIGAN decoder.

[Amarnath1906]

One of the issues,
[11/20/2025-16:07:46] [V] [TRT] Applying generic optimizations to the graph for inference.
[11/20/2025-16:07:46] [E] Error[4]: kOPT values for profile 0 violate shape constraints: /generator/Add_1: dimensions not compatible for elementwise. Condition '==' violated: 1279 != 639.
[11/20/2025-16:07:46] [E] Error[4]: [shapeCompiler.cpp::evaluateShapeChecks::1276] Error Code 4: Internal Error (kOPT values for profile 0 violate shape constraints: /generator/Add_1: dimensions not compatible for elementwise. Condition '==' violated: 1279 != 639.)

I am facing this issue in forward function of generator,

        x = self.ups[i](x)
        x = x + x_source

Can you please help with converting decoder to tensorRT?