[Question] Extremely long build time (>50 hours) when compiling Transformer/SNN models with TensorRT

[pengjh0111]

I am a beginner with TensorRT, and I am currently trying to optimize some classic SNN (Spiking Neural Network) models using TensorRT.

When I previously used TensorRT for CNN-type networks, the scheduling/compilation process was quite fast — I could generate an .engine file within minutes.

However, when I try to import networks with transformer structures, such as BERT, the scheduling time becomes extremely long. For example, in one model I tested, the process has already taken more than 50 hours and is still not finished. Is this expected behavior?

I also noticed a similar issue mentioned in a 2020 paper, where the authors reported that scheduling LSTM networks with TensorRT also took an abnormally long time.

Since SNNs also have temporal characteristics similar to LSTMs, I wonder:
• Is this a known limitation or issue in TensorRT when dealing with such models?
• Or could it be that my scheduling code has some problem that causes this excessive compilation time?

[SumitdevelopAI]

Enable Builder Flags to Limit Search

Use builder.setFlag(trt.BuilderFlag.FP16) or INT8 (if calibrated). This reduces the number of tactics TensorRT tries.

Use builder.max_workspace_size wisely (don’t make it unnecessarily huge).

Use Heuristic Tactic Sources
TensorRT ≥ 8.0 lets you control tactic sources. For example:

config.set_tactic_sources([trt.TacticSource.CUBLAS, trt.TacticSource.CUBLAS_LT])

This avoids TensorRT exploring all kernels (which is what causes 50h builds).

Limit Timing Iterations

config.set_timing_cache(timing_cache, ignore_mismatch=True)
config.max_timing_iterations = 2
config.min_timing_iterations = 1

By default TensorRT may do hundreds of timing iterations per layer, which explodes build time.

Use Prebuilt / Pre-Optimized Plugins
For BERT/Transformer-like models, NVIDIA provides TensorRT OSS and FasterTransformer. These include hand-optimized fused kernels that skip the massive scheduling step.

If you use their plugins, build times drop from hours to minutes.

Profile the Graph
You can dump the TensorRT network to see which subgraph is consuming build time. Often it’s MatMul + Softmax in transformers, or RNNCell in LSTMs/SNNs.