platforms vs. GPU architectures vs. GPU one specific model

[leimeng86]

For Q1, there is no guarantee the serialized engine will work on different platforms, as the optimized graph uses kernels specific to the GPU. If you are running on different machines with the Same GPU architecture/OS it should work. You can support multiple GPU's as backends, with multiple saved serialized engines with TRT-IS (Inference Server) if you are trying to support a larger collection of GPU's as one suggestion.

Originally posted by @CMagWheels in #70 (comment)

[leimeng86]

Another quick question:

1. Do platforms mean one specific GPU architecture (e.g. Ampere architecture) or a specific GPU model (e.g. GeForce RTX 3090)?
2. For example, GeForce RTX 3070 and GeForce RTX 3090 are build with the same Ampere architecture but are difference models. Can I reuse the serialized TensorRT inference engine compiled on 3090 on a different GPU model 3070?

Regards

[MamaShip]

I'm concerned about this issue too.

My team is trying to make our software compatible with all GPUs on Ampere architecture and later. But we have to re-generate all engines on different GPUs again and again. Such work is too overload for us. Not to mention that we are unable to obtain the latest GPU model immediately.

Can't we just release a general compatible version with some performance penalty, and enhance it later with specifically adapted one?

[zerollzeng]

• Do platforms mean one specific GPU architecture (e.g. Ampere architecture) or a specific GPU model (e.g. GeForce RTX 3090)?

specific gpu model, tensorrt will test all the available kernels on the GPU when you build the engine, and pick the best kernel that produce best performance. so it's GPU model targeted, even more, let's said that you have a A100 with 40GB memory, the engine built on it may not be optimal in A100 with 80GB memory.

• For example, GeForce RTX 3070 and GeForce RTX 3090 are build with the same Ampere architecture but are difference models. Can I reuse the serialized TensorRT inference engine compiled on 3090 on a different GPU model 3070?

no, don't do that. you won't get the best performance.

[ttyio]

Can't we just release a general compatible version with some performance penalty, and enhance it later with specifically adapted one?

Thanks @MamaShip , could you elaborate more on your use case here? Can you generate the engine on-the-fly when your customers first start your product? There is one possible brainstorm solution for the fallback path you asked, e.g, TRT provide some heuristic without engine build and you can always ask for a good enough tactic set work on your environment , this set of tactic is not global optimal because we have not run through the engine build process, but always works.

[leimeng86]

• Do platforms mean one specific GPU architecture (e.g. Ampere architecture) or a specific GPU model (e.g. GeForce RTX 3090)?

specific gpu model, tensorrt will test all the available kernels on the GPU when you build the engine, and pick the best kernel that produce best performance. so it's GPU model targeted, even more, let's said that you have a A100 with 40GB memory, the engine built on it may not be optimal in A100 with 80GB memory.

• For example, GeForce RTX 3070 and GeForce RTX 3090 are build with the same Ampere architecture but are difference models. Can I reuse the serialized TensorRT inference engine compiled on 3090 on a different GPU model 3070?

no, don't do that. you won't get the best performance.

Indeed, it makes sense.
Thanks!

[MamaShip]

TRT provide some heuristic without engine build and you can always ask for a good enough tactic set work on your environment , this set of tactic is not global optimal because we have not run through the engine build process, but always works.

Thanks for your kind reply. If TRT provide this kind of function, it would be great help for us!

could you elaborate more on your use case here? Can you generate the engine on-the-fly when your customers first start your product?

Actually we are developing Gene Sequencing Instrument. And our product contains a PC (with A4000 GPU) to perform the neural network inference process. We choose TensorRT to unleash the maximum performance of A4000. And expect our customers to buy a complete set of sequencing equipment including that A4000 PC.

When our sales face customers, we find that many customers want to use their own PC & GPU instead of our A4000 PC. So we have to start adapting to all GPU models.

We also considered generating the engine in customer's environment on first boot. But our neural network models are quite large and each generation process cost more than 1 hour. Since our users need real-time sequencing, especially in some medical scenarios, this solution seems unacceptable.
In addition, our product is in a frequent iterative process, we may update our neural network models every month. And we provide different type of models for user to choose, which make things more complicated.

[ttyio]

Thanks @MamaShip for the detail.

our neural network models are quite large and each generation process cost more than 1 hour.

Have you enabled global timing cache? They are shared between models and speed up the engine build time. For more detail, please check https://docs.nvidia.com/deeplearning/tensorrt/developer-guide/index.html#builder-layer-timing

I have created RFE internally for your case. Before we have this feature, one WAR you can try is that: Find out GPU with the lowest memory for each SM arch, and generate engine for this arch. Then after your product is installed, For A4000 PC, you still use the best engine that have tuned. For other GPUs, you can use the lowest engine as default for all SKU in the same SM arch. Notice this is not official workflow, so you need test on your side for the robustness.

[MamaShip]

@ttyio Thanks for your advice!
We will try global timing cache. And we will test the lowest engine's robustness.

[MamaShip]

@ttyio Excuse me, can you estimate the official release time of this feature?

The lowest engine solution works fine. But we wonder how long this will take.