TensorRT Lean Best Practices

[joekale-pp]

TensorRT Core libraries contain all of the functions to build, fine-tune, and compile model engine files. This is unnecessary bulk to the target filesystem for edge devices with no need to compile engines. To combat this Nvidia provides the "dispatch" and "lean" libraries where the "lean" allows a smaller footprint on the device.

Builds fail when you try to DEPEND on tensorrt-core-lean so the solution I used to allow building was:

1. Add tensorrt-core to DEPENDS
2. Remove tensorrt-core from RDEPENDS
3. Add tensorrt-core-lean to the RDEPENDS

My current understanding is, the application only needs to link against libnvinfer_lean for runtime usage of that lean_runtime, or the dispatch can be used to allow multpile runtimes to be deployed for engines not compiled for the exact lean_runtime.

The engine also needs to be compiled on the same hardware, with the same set of libraries (full tensorrt-core required) but with the options --versionCompatible --excludeLeanRuntime with the latter keeping the runtime from being built into the engine.

Is this the best way to achieve this? By adding tensortt-core to the DEPENDS all libraries end up in the sysroots which is good, but you have to manually remove the largest package manually from the RDEPENDS and add the smaller one to achieve the 750MB size reduction.

[joekale-pp]

I added a Draft PR to potentially address some of the packaging issues based on my understanding of the packages. #2357