Error in : Doing inference using the engine file using ./trtexec [Tensorrt 8.0 OSS with CUDA 11.3]

[nanditam1]

I need to build Tensorrt 8.0 OSS with CUDA 11.3, and the build files were generated without errors. However on running inference using engine file, I see an error /Tensort8.0/TensorRT/plugin/instanceNormalizationPlugin/instanceNormalizationPlugin.cu (286) - Cuda Error in enqueue: 906 (operation would make the legacy stream depend on a capturing blocking stream)

terminate called after throwing an instance of 'nvinfer1::plugin::CudaError'

what(): std::exception

Aborted (core dumped)

Environment

TensorRT Version: 8.0
NVIDIA GPU: QUADRO RTX 5000
NVIDIA Driver Version: 465.19.01
CUDA Version: 11.3
CUDNN Version: 8.2
Operating System: Ubuntu
Python Version (if applicable): 3.8
Tensorflow Version (if applicable):
PyTorch Version (if applicable): 1.9
Baremetal or Container (if so, version): Baremetal

Command:
./trtexec --loadEngine=/Tensort8.0/TensorRT/build/out/model.engine --useCudaGraph --iterations=1 --avgRuns=1 --plugins=/Tensort8.0/TensorRT/build/out/libnvinfer_plugin.so --verbose

Error:
[05/05/2022-13:02:46] [I] Created input binding for input1 with dimensions 1x3x376x1224

[05/05/2022-13:02:46] [I] Created input binding for input2 with dimensions 1x3x376x1224

[05/05/2022-13:02:46] [I] Created output binding for _up with dimensions 1x2x376x1224

[05/05/2022-13:02:46] [I] Created output binding for _low with dimensions 1x2x47x153

[05/05/2022-13:02:46] [I] Starting inference

[05/05/2022-13:02:46] [E] [TRT] /Tensort8.0/TensorRT/plugin/instanceNormalizationPlugin/instanceNormalizationPlugin.cu (286) - Cuda Error in enqueue: 906 (operation would make the legacy stream depend on a capturing blocking stream)

terminate called after throwing an instance of 'nvinfer1::plugin::CudaError'

what(): std::exception

Aborted (core dumped)

How can I resolve this? Is there a cuda version mismatch? However as per the github page The recommended versions are cuda-11.3.1 + cuDNN-8.2. Kindly help debug this?

[ttyio]

@nanditam1 , have you tried remove --useCudaGraph? the errors seems related to cuda graph capture, and this plugin using cudnn, maybe there is some cudnn api not compatible with cuda graph.

[nanditam1]

@ttyio Thanks for your response! Yes removing --useCudaGraph solved the problem of this error. Would you suggest I reinstall cudnn to avoid this incompatibility?

[ttyio]

@nanditam1 , not sure if reinstall the same cudnn version helps. I would suggest upgrade to latest TRT which use a newer cudnn, thanks!

[nvpohanh]

This is not related to CUDNN. It is because our InstanceNorm plugin used to have synchronous CUDA API calls within enqueue() function: https://gitlab-master.nvidia.com/TensorRT/Public/oss/-/blob/release/8.0/plugin/instanceNormalizationPlugin/instanceNormalizationPlugin.cu#L281

These have been fixed in the latest (8.2 and 8.4) plugin. Please update your plugin code to release/8.2 or release/8.4 branch and see if the issue is fixed. Thanks

[nanditam1]

ok I shall try with upgraded plugin and update once I have the result. Thank you.

[nvpohanh]

I am going to close this issue for now. Please feel free to reopen if the issue still exists. Thanks