ONNX-TensorRT: TensorRT backend for ONNX
Clone or download
Caenorst and yinghai Add basic support for data type inputs other than float32 (#92)
* add basic fp16 inputs support

* correct syntax error
Latest commit 1e209e5 Jan 4, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
onnx_tensorrt TRT4: Import pycuda.autoinit to fix context issue Jun 26, 2018
third_party ONNX changed to use int32_t to store data type (#86) Dec 8, 2018
.gitignore Initial code commit Apr 30, 2018
.gitmodules Initial code commit Apr 30, 2018
CMakeLists.txt Implemented support for Cmake versions >= 3.10 (#81) Nov 28, 2018
Dockerfile Minor whitespace cleanup in Dockerfile May 26, 2018
FancyActivation.cu Initial code commit Apr 30, 2018
FancyActivation.hpp Revert "Fix compilation with TensorRT 5 RC (#55)" (#69) Nov 11, 2018
ImporterContext.hpp Fix crash due to missing opset field in old models (#41) Aug 11, 2018
InstanceNormalization.cpp Enable compiler warnings and fix some May 24, 2018
InstanceNormalization.hpp Revert "Fix compilation with TensorRT 5 RC (#55)" (#69) Nov 11, 2018
LICENSE Initial code commit Apr 30, 2018
ModelImporter.cpp Update onnx (#42) Aug 14, 2018
ModelImporter.hpp Update onnx (#42) Aug 14, 2018
NvOnnxParser.cpp Initial code commit Apr 30, 2018
NvOnnxParser.h Update onnx (#42) Aug 14, 2018
NvOnnxParserRuntime.cpp Initial code commit Apr 30, 2018
NvOnnxParserRuntime.h Initial code commit Apr 30, 2018
OnnxAttrs.cpp Support vector-of-float attribute type May 12, 2018
OnnxAttrs.hpp Update onnx submodule and fix compiler warnings (#70) Nov 11, 2018
PluginFactory.cpp Initial code commit Apr 30, 2018
PluginFactory.hpp Initial code commit Apr 30, 2018
README.md Add info about -v flag support in tests (#51) Sep 11, 2018
ResizeNearest.cu Initial code commit Apr 30, 2018
ResizeNearest.hpp Revert "Fix compilation with TensorRT 5 RC (#55)" (#69) Nov 11, 2018
ShapedWeights.cpp Enable compiler warnings and fix some May 24, 2018
ShapedWeights.hpp ONNX changed to use int32_t to store data type (#86) Dec 8, 2018
Split.cu Initial code commit Apr 30, 2018
Split.hpp Revert "Fix compilation with TensorRT 5 RC (#55)" (#69) Nov 11, 2018
Status.hpp Initial code commit Apr 30, 2018
TensorOrWeights.hpp Refactor + improve handling of elwise broadcasting (#53) Oct 9, 2018
builtin_op_importers.cpp Add support for Upsample with opset_ver >= 9 (#80) Nov 28, 2018
builtin_op_importers.hpp Initial code commit Apr 30, 2018
builtin_plugins.cpp Fix plugins not being registered in Release build May 2, 2018
builtin_plugins.hpp Initial code commit Apr 30, 2018
libnvonnxparser.version Initial code commit Apr 30, 2018
libnvonnxparser_runtime.version Initial code commit Apr 30, 2018
main.cpp Fix some opset 7 issues (#47) Sep 18, 2018
nv_onnx_parser_bindings.i Initial code commit Apr 30, 2018
nv_onnx_runtime_bindings.i Initial code commit Apr 30, 2018
onnx2trt.hpp Update onnx submodule and fix compiler warnings (#70) Nov 11, 2018
onnx2trt_common.hpp Fix some issues with plugin type serialization May 12, 2018
onnx2trt_runtime.hpp Initial code commit Apr 30, 2018
onnx2trt_utils.cpp Fix some opset 7 issues (#47) Sep 18, 2018
onnx2trt_utils.hpp Add basic support for data type inputs other than float32 (#92) Jan 4, 2019
onnx_backend_test.py TRT4: Adjust backend test exclusions Jun 26, 2018
onnx_trt_backend.cpp Update onnx submodule and fix compiler warnings (#70) Nov 11, 2018
onnx_utils.hpp Update onnx submodule and fix compiler warnings (#70) Nov 11, 2018
plugin.cpp WAR an issue with some NvPlugins in TRT3 May 26, 2018
plugin.hpp Revert "Fix compilation with TensorRT 5 RC (#55)" (#69) Nov 11, 2018
plugin_common.hpp Initial code commit Apr 30, 2018
serialize.hpp Initial code commit Apr 30, 2018
setup.py Add pycuda dependency to setup.py (#49) Sep 11, 2018
toposort.hpp Initial code commit Apr 30, 2018
trt_utils.hpp Fix some opset 7 issues (#47) Sep 18, 2018
utils.hpp Initial code commit Apr 30, 2018

README.md

TensorRT backend for ONNX

Parses ONNX models for execution with TensorRT.

See also the TensorRT documentation.

ONNX Python backend usage

The TensorRT backend for ONNX can be used in Python as follows:

import onnx
import onnx_tensorrt.backend as backend
import numpy as np

model = onnx.load("/path/to/model.onnx")
engine = backend.prepare(model, device='CUDA:1')
input_data = np.random.random(size=(32, 3, 224, 224)).astype(np.float32)
output_data = engine.run(input_data)[0]
print(output_data)
print(output_data.shape)

Executable usage

ONNX models can be converted to serialized TensorRT engines using the onnx2trt executable:

onnx2trt my_model.onnx -o my_engine.trt

ONNX models can also be converted to human-readable text:

onnx2trt my_model.onnx -t my_model.onnx.txt

See more usage information by running:

onnx2trt -h

C++ library usage

The model parser library, libnvonnxparser.so, has a C++ API declared in this header:

NvOnnxParser.h

TensorRT engines built using this parser must use the plugin factory provided in libnvonnxparser_runtime.so, which has a C++ API declared in this header:

NvOnnxParserRuntime.h

Installation

Dependencies

Download the code

Clone the code from GitHub.

git clone --recursive https://github.com/onnx/onnx-tensorrt.git

Executable and libraries

Suppose your TensorRT library is located at /opt/tensorrt. Build the onnx2trt executable and the libnvonnxparser* libraries using CMake:

mkdir build
cd build
cmake .. -DTENSORRT_ROOT=/opt/tensorrt
make -j8
sudo make install

Python modules

Build the Python wrappers and modules by running:

python setup.py build
sudo python setup.py install

Docker image

Build the onnx_tensorrt Docker image by running:

cp /path/to/TensorRT-3.0.*.tar.gz .
docker build -t onnx_tensorrt .

Tests

After installation (or inside the Docker container), ONNX backend tests can be run as follows:

Real model tests only:

python onnx_backend_test.py OnnxBackendRealModelTest

All tests:

python onnx_backend_test.py

You can use -v flag to make output more verbose.

Pre-trained models

Pre-trained Caffe2 models in ONNX format can be found at https://github.com/onnx/models