The Triton backend for TensorRT. You can learn more about Triton backends in the backend repo. Ask questions or report problems on the issues page. This backend is designed to run a serialized TensorRT engine models using the TensorRT C++ API.
Where can I ask general questions about Triton and Triton backends? Be sure to read all the information below as well as the general Triton documentation available in the main server repo. If you don't find your answer there you can ask questions on the main Triton issues page.
The command-line options configure properties of the TensorRT backend that are then applied to all models that use the backend.
--backend-config=tensorrt,coalesce-request-input=<boolean>,plugins="/path/plugin1.so;/path2/plugin2.so"
The coalesce-request-input flag instructs TensorRT to consider the requests' inputs with the same name as one contiguous buffer if their memory addresses align with each other. This option should only be enabled if all requests' input tensors are allocated from the same memory region. Default value is false.
The plugins flag provides a way to load any custom TensorRT plugins that your models rely on. If you have multiple plugins to load, use a semicolon as the delimiter.
Appropriate version of TensorRT must be installed on the system. Check the support matrix to find the correct version of TensorRT to be installed.
$ mkdir build
$ cd build
$ cmake -DCMAKE_INSTALL_PREFIX:PATH=`pwd`/install ..
$ make install
The following required Triton repositories will be pulled and used in the build. By default the "main" branch/tag will be used for each repo but the listed CMake argument can be used to override.
- triton-inference-server/backend: -DTRITON_BACKEND_REPO_TAG=[tag]
- triton-inference-server/core: -DTRITON_CORE_REPO_TAG=[tag]
- triton-inference-server/common: -DTRITON_COMMON_REPO_TAG=[tag]