engine build problem

[qiumeng6]

I am using TensorRT 10.8 on the Thor-U. When building an engine with trtexec and performing inference, I noticed that running the same command with the same parameters multiple times can produce two different engine graphs, which leads to different inference results.

Is it expected that engine builds are inconsistent even when using identical commands and parameters?

[Saad-Mallebhari]

@qiumeng6
Non-deterministic engine builds are not uncommon in TensorRT , but whether this is expected or worth investigating depends on a couple of things.

Why builds can differ across runs:

• Tactic selection: TRT benchmarks multiple kernel implementations per layer at build time. Timing noise between runs can cause different tactics to win, producing different graph structures.
• No shared timing cache: Without one, each build re-benchmarks independently. Sharing a cache across runs tends to make tactic selection more stable:

  trtexec --onnx=model.onnx --saveEngine=model.engine --timingCacheFile=timing.cache

• Builder optimization level: Higher levels explore more tactics, which can increase variability. Try pinning it:

  --builderOptimizationLevel=3

When it's worth investigating further:

Different graph structures causing small numerical drift is generally acceptable. But if outputs differ significantly, it's worth checking:

• What precision are you using (FP32 / FP16 / FP8)? FP16 and FP8 builds are more sensitive to tactic variation.
• How large is the output difference , cosine similarity or max absolute diff would help quantify it.
• Are you on a fixed driver/CUDA version? Tactic availability can shift across driver updates.

A few other knobs worth trying for more stable builds:

• --noBuilderCache to disable the internal builder cache and force fresh benchmarking
• --tacticSources to restrict which tactic sources TRT considers

Note: timing cache reduces variability but isn't a strict determinism guarantee , hardware timing jitter can still occasionally flip tactic selection.

[qiumeng6]

@Saad-Mallebhari
Thank you very much for your response.

1. Since we observed a metric gap with the FP16 model (we have not completed the root cause analysis yet), we temporarily rolled back to FP32.
2. However, we found that even with FP32, two different engines may be generated during the build process. The cosine similarity between their outputs is 0.99747. According to our evaluation metrics, one engine produces results very close to the PyTorch model, while the other shows a noticeable accuracy drop.
3. Both cases use the same driver and CUDA versions.

In addition, I have a question: during the engine build phase, if there are other processes running on the Thor platform that also utilize the GPU, could this affect the engine building process?

[Saad-Mallebhari]

@qiumeng6
Thanks for the follow-up

On GPU contention during builds:
Yes, other processes using the GPU during engine build can affect tactic selection. TRT benchmarks kernel timings at build time, and if another process is competing for GPU resources, timing measurements become less reliable , which can cause a different tactic to win. Building on an idle GPU is worth trying to rule this out.

On the FP32 accuracy gap:
A cosine similarity of 0.99747 with one engine matching PyTorch closely and the other showing a noticeable drop suggests a specific layer or tactic may be behaving inconsistently , though cosine similarity alone doesn't always tell the full story. A few things worth trying to isolate it:

• Use --timingCacheFile=timing.cache across both builds , if the same cache produces consistent results, tactic timing noise is likely the cause
• Try --builderOptimizationLevel=3, which can sometimes make tactic selection more consistent, though it's not a strict guarantee
• Enable --profilingVerbosity=detailed and compare the two engine plans with trtexec --dumpLayerInfo , look for layers where different tactics were selected between the two builds

On identifying the problematic tactic:
If you can reproduce both engines reliably, polygraphy's debug build mode can help bisect which layer is responsible for the accuracy drop:

polygraphy debug build model.onnx --save-tactics bad_tactics.json

Whether this is a tactic stability issue, a precision edge case, or something deeper in the build path is worth keeping open until the differing layer is identified.