Problem
Relax TensorRT BYOC builds the TensorRT engine during the first runtime initialization or inference. For YOLO-style models, I observed a cold-start delay of approximately 20–80 seconds, depending on the model complexity and the number of BYOC subgraphs. The model currently produces a single TensorRT subgraph, but other models may be partitioned into multiple subgraphs.
This is consistent with TensorRT documentation, which states that engine building can take significant time and is normally treated as an offline step:
https://docs.nvidia.com/deeplearning/tensorrt/latest/inference-library/c-api-docs.html#the-build-phase
TVM_TENSORRT_CACHE_DIR avoids rebuilding on subsequent runs, but it is not very convenient for deployment:
- The first process still pays the complete build cost.
- A writable persistent cache directory is required.
.plan and .meta files must be managed separately.- Cache configuration is process-wide rather than model-local.
For comparison, I tested the same YOLO model with TVM native CUDA. Compilation took about 24.2 seconds and export took 3.0 seconds, but these costs occurred offline. In a new process, loading the .so took about 319 ms and VM initialization took about 143 ms. There was no additional runtime build delay comparable to TensorRT BYOC.
Is there a supported way to perform the expensive TensorRT engine-building work during compilation/export instead of the first runtime initialization?
TensorRT plans are device- and version-dependent, but even a target-machine export workflow would be easier to manage than an implicit build during application startup.
Problem
Relax TensorRT BYOC builds the TensorRT engine during the first runtime initialization or inference. For YOLO-style models, I observed a cold-start delay of approximately 20–80 seconds, depending on the model complexity and the number of BYOC subgraphs. The model currently produces a single TensorRT subgraph, but other models may be partitioned into multiple subgraphs.
This is consistent with TensorRT documentation, which states that engine building can take significant time and is normally treated as an offline step:
https://docs.nvidia.com/deeplearning/tensorrt/latest/inference-library/c-api-docs.html#the-build-phase
TVM_TENSORRT_CACHE_DIRavoids rebuilding on subsequent runs, but it is not very convenient for deployment:.planand.metafiles must be managed separately.For comparison, I tested the same YOLO model with TVM native CUDA. Compilation took about 24.2 seconds and export took 3.0 seconds, but these costs occurred offline. In a new process, loading the
.sotook about 319 ms and VM initialization took about 143 ms. There was no additional runtime build delay comparable to TensorRT BYOC.Is there a supported way to perform the expensive TensorRT engine-building work during compilation/export instead of the first runtime initialization?
TensorRT plans are device- and version-dependent, but even a target-machine export workflow would be easier to manage than an implicit build during application startup.