Tensorflow V2 object detection onnx to tensorrt

[Source82]

Hello,

I have been trying to convert mobilenet v2 SSD following the suggestions given here unfortunatley I am stuck with the error shown below. Basically I have tensorflow v2 model that has dynamic shapes, how do I go about making them static since one cannot freeze model on tensorflow v2.

/usr/src/tensorrt/bin/trtexec --onnx=modele2-nms.onnx

[07/28/2021-10:36:44] [I] === Model Options ===
[07/28/2021-10:36:44] [I] Format: ONNX
[07/28/2021-10:36:44] [I] Model: modele2-nms.onnx
[07/28/2021-10:36:44] [I] Output:
[07/28/2021-10:36:44] [I] === Build Options ===
[07/28/2021-10:36:44] [I] Max batch: 1
[07/28/2021-10:36:44] [I] Workspace: 16 MB
[07/28/2021-10:36:44] [I] minTiming: 1
[07/28/2021-10:36:44] [I] avgTiming: 8
[07/28/2021-10:36:44] [I] Precision: FP32
[07/28/2021-10:36:44] [I] Calibration:
[07/28/2021-10:36:44] [I] Safe mode: Disabled
[07/28/2021-10:36:44] [I] Save engine:
[07/28/2021-10:36:44] [I] Load engine:
[07/28/2021-10:36:44] [I] Builder Cache: Enabled
[07/28/2021-10:36:44] [I] NVTX verbosity: 0
[07/28/2021-10:36:44] [I] Inputs format: fp32:CHW
[07/28/2021-10:36:44] [I] Outputs format: fp32:CHW
[07/28/2021-10:36:44] [I] Input build shapes: model
[07/28/2021-10:36:44] [I] Input calibration shapes: model
[07/28/2021-10:36:44] [I] === System Options ===
[07/28/2021-10:36:44] [I] Device: 0
[07/28/2021-10:36:44] [I] DLACore:
[07/28/2021-10:36:44] [I] Plugins:
[07/28/2021-10:36:44] [I] === Inference Options ===
[07/28/2021-10:36:44] [I] Batch: 1
[07/28/2021-10:36:44] [I] Input inference shapes: model
[07/28/2021-10:36:44] [I] Iterations: 10
[07/28/2021-10:36:44] [I] Duration: 3s (+ 200ms warm up)
[07/28/2021-10:36:44] [I] Sleep time: 0ms
[07/28/2021-10:36:44] [I] Streams: 1
[07/28/2021-10:36:44] [I] ExposeDMA: Disabled
[07/28/2021-10:36:44] [I] Spin-wait: Disabled
[07/28/2021-10:36:44] [I] Multithreading: Disabled
[07/28/2021-10:36:44] [I] CUDA Graph: Disabled
[07/28/2021-10:36:44] [I] Skip inference: Disabled
[07/28/2021-10:36:44] [I] Inputs:
[07/28/2021-10:36:44] [I] === Reporting Options ===
[07/28/2021-10:36:44] [I] Verbose: Disabled
[07/28/2021-10:36:44] [I] Averages: 10 inferences
[07/28/2021-10:36:44] [I] Percentile: 99
[07/28/2021-10:36:44] [I] Dump output: Disabled
[07/28/2021-10:36:44] [I] Profile: Disabled
[07/28/2021-10:36:44] [I] Export timing to JSON file:
[07/28/2021-10:36:44] [I] Export output to JSON file:
[07/28/2021-10:36:44] [I] Export profile to JSON file:
[07/28/2021-10:36:44] [I]

Input filename: modele2-nms.onnx
ONNX IR version: 0.0.6
Opset version: 11
Producer name:
Producer version:
Domain:
Model version: 0
Doc string:
----------------------------------------------------------------
[07/28/2021-10:36:48] [W] [TRT] onnx2trt_utils.cpp:220: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[07/28/2021-10:36:48] [W] [TRT] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[07/28/2021-10:36:48] [I] [TRT] ModelImporter.cpp:135: No importer registered for op: BatchedNMS_TRT. Attempting to import as plugin.
[07/28/2021-10:36:48] [I] [TRT] builtin_op_importers.cpp:3659: Searching for plugin: BatchedNMS_TRT, plugin_version: 1, plugin_namespace:
[07/28/2021-10:36:48] [I] [TRT] builtin_op_importers.cpp:3676: Successfully created plugin: BatchedNMS_TRT
[07/28/2021-10:36:48] [E] [TRT] onnx_graphsurgeon_node_0: PluginV2Layer must be V2DynamicExt when there are runtime input dimensions.
[07/28/2021-10:36:48] [E] [TRT] onnx_graphsurgeon_node_0: PluginV2Layer must be V2DynamicExt when there are runtime input dimensions.
[07/28/2021-10:36:48] [E] [TRT] onnx_graphsurgeon_node_0: PluginV2Layer must be V2DynamicExt when there are runtime input dimensions.
ERROR: builtin_op_importers.cpp:3040 In function importSlice:
[4] Assertion failed: -r <= axis && axis < r
[07/28/2021-10:36:48] [E] Failed to parse onnx file
[07/28/2021-10:36:48] [E] Parsing model failed
[07/28/2021-10:36:48] [E] Engine creation failed
[07/28/2021-10:36:48] [E] Engine set up failed
&&&& FAILED TensorRT.trtexec # /usr/src/tensorrt/bin/trtexec --onnx=modele2-nms.onnx

[Source82]

@pranavm-nvidia please help with suggestions

[pranavm-nvidia]

You could try using Polygraphy to freeze the input shapes:

polygraphy surgeon sanitize modele2-nms.onnx \
    --override-input-shapes input_name:[1,3,224,224] \
    -o modele2-nms-static-shape.onnx

[Source82]

You could try using Polygraphy to freeze the input shapes:

polygraphy surgeon sanitize modele2-nms.onnx \
    --override-input-shapes input_name:[1,3,224,224] \
    -o modele2-nms-static-shape.onnx

ok thanks, i tried, it still failed at trtexec

[azhurkevich]

@Source82 can you try a sample we've posted in fall. Please follow the instructions closely. This is for pretrained TFOD models from the zoo. If you have a custom model we cannot guarantee flawless conversion but it can certainly be done, if you are willing to read through code. I made sure there are a lot of comments that will help.

[nvpohanh]

This is because BatchedDynamicNMS_TRT instead of BatchedNMS_TRT should be used. @kevinch-nv @rajeevsrao Which one does ONNX parser use?

[rajeevsrao]

I think @Source82 explicitly added the BatchedNMS_TRT plugin using onnx-gs as a workaround for missing NMS support in the onnx parser prior to TRT 8.0.

Since TensorRT 8.x, our ONNX parser supports NonMaxSuppression Op and maps it to the new EfficientNMS_ONNX_TRT plugin.

@Source82 please refer to the instructions from @azhurkevich above regarding support for TFOD models.

[ttyio]

closing due to no activity for more than 3 weeks, please reopen if you still have question, thanks!