TensorRT 8 Support

[Michelvl92]

I have tried to build yolov4 with the tutorial, but instead of using Nvidia tensorrt docker version 21.03 version, I used 21.09 (with TensorRT 8.0.3) which resulted in the following errors:

liermlv@v2119:/data/liermlv/gitlab/external$ docker run --gpus all -it --rm -v /data/liermlv/^Cvcr.io/nvidia/tensorrt:21.09-py3
liermlv@v2119:/data/liermlv/gitlab/external$ tmux new -s tensorrt-conv
root@786e221fbf7e:/workspace# cd /external/yolov4-triton-tensorrt/                                                                                                        [66/66]
root@786e221fbf7e:/external/yolov4-triton-tensorrt# mkdir build
root@786e221fbf7e:/external/yolov4-triton-tensorrt# cd build
root@786e221fbf7e:/external/yolov4-triton-tensorrt/build# cmake ..
-- The CXX compiler identification is GNU 9.3.0
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Looking for C++ include pthread.h
-- Looking for C++ include pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Found CUDA: /usr/local/cuda (found version "11.4")
embed_platform off
-- Configuring done
-- Generating done
-- Build files have been written to: /external/yolov4-triton-tensorrt/build
root@786e221fbf7e:/external/yolov4-triton-tensorrt/build# make
[ 25%] Building NVCC (Device) object CMakeFiles/layerplugin.dir/layers/layerplugin_generated_yololayer.cu.o
/external/yolov4-triton-tensorrt/layers/yololayer.h(59): error: member function declared with "override" does not override a base class member

/external/yolov4-triton-tensorrt/layers/yololayer.h(108): error: function "nvinfer1::IPluginV2IOExt::configurePlugin(const nvinfer1::Dims *, int32_t, const nvinfer1::Dims *, int
32_t, const nvinfer1::DataType *, const nvinfer1::DataType *, const __nv_bool *, const __nv_bool *, nvinfer1::PluginFormat, int32_t)"
/usr/include/x86_64-linux-gnu/NvInferRuntimeCommon.h(836): here is inaccessible

/external/yolov4-triton-tensorrt/layers/yololayer.h(59): warning: function "nvinfer1::IPluginV2::enqueue(int32_t, const void *const *, void *const *, void *, cudaStream_t)" is h
idden by "nvinfer1::YoloLayerPlugin::enqueue" -- virtual function override intended?

/external/yolov4-triton-tensorrt/layers/yololayer.h(89): warning: function "nvinfer1::IPluginV2Ext::configurePlugin(const nvinfer1::Dims *, int32_t, const nvinfer1::Dims *, int3
2_t, const nvinfer1::DataType *, const nvinfer1::DataType *, const __nv_bool *, const __nv_bool *, nvinfer1::PluginFormat, int32_t)" is hidden by "nvinfer1::YoloLayerPlugin::con
figurePlugin" -- virtual function override intended?

/external/yolov4-triton-tensorrt/layers/yololayer.h(91): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::detachFromContext" is incompatible with
that of overridden function "nvinfer1::IPluginV2Ext::detachFromContext"

/external/yolov4-triton-tensorrt/layers/yololayer.h(89): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::configurePlugin" is incompatible with th
at of overridden function "nvinfer1::IPluginV2IOExt::configurePlugin(const nvinfer1::PluginTensorDesc *, int32_t, const nvinfer1::PluginTensorDesc *, int32_t)"

/external/yolov4-triton-tensorrt/layers/yololayer.h(87): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::attachToContext" is incompatible with th
at of overridden function "nvinfer1::IPluginV2Ext::attachToContext"

/external/yolov4-triton-tensorrt/layers/yololayer.h(85): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::canBroadcastInputAcrossBatch" is incompa
tible with that of overridden function "nvinfer1::IPluginV2Ext::canBroadcastInputAcrossBatch"

/external/yolov4-triton-tensorrt/layers/yololayer.h(83): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::isOutputBroadcastAcrossBatch" is incompa
tible with that of overridden function "nvinfer1::IPluginV2Ext::isOutputBroadcastAcrossBatch"

/external/yolov4-triton-tensorrt/layers/yololayer.h(81): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::getOutputDataType" is incompatible with
that of overridden function "nvinfer1::IPluginV2Ext::getOutputDataType"

/external/yolov4-triton-tensorrt/layers/yololayer.h(79): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::getPluginNamespace" is incompatible with
 that of overridden function "nvinfer1::IPluginV2::getPluginNamespace"

/external/yolov4-triton-tensorrt/layers/yololayer.h(77): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::setPluginNamespace" is incompatible with
 that of overridden function "nvinfer1::IPluginV2::setPluginNamespace"

/external/yolov4-triton-tensorrt/layers/yololayer.h(75): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::clone" is incompatible with that of over
ridden function "nvinfer1::IPluginV2Ext::clone"

/external/yolov4-triton-tensorrt/layers/yololayer.h(75): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::clone" is incompatible with that of overridden function "nvinfer1::IPluginV2::clone"

/external/yolov4-triton-tensorrt/layers/yololayer.h(73): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::destroy" is incompatible with that of overridden function "nvinfer1::IPluginV2::destroy"

/external/yolov4-triton-tensorrt/layers/yololayer.h(71): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::getPluginVersion" is incompatible with that of overridden function "nvinfer1::IPluginV2::getPluginVersion"

/external/yolov4-triton-tensorrt/layers/yololayer.h(69): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::getPluginType" is incompatible with that of overridden function "nvinfer1::IPluginV2::getPluginType


/external/yolov4-triton-tensorrt/layers/yololayer.h(65): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::supportsFormatCombination" is incompatible with that of overridden function "nvinfer1::IPluginV2IOExt::supportsFormatCombination"

/external/yolov4-triton-tensorrt/layers/yololayer.h(63): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::serialize" is incompatible with that of overridden function "nvinfer1::IPluginV2::serialize"

/external/yolov4-triton-tensorrt/layers/yololayer.h(61): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::getSerializationSize" is incompatible with that of overridden function "nvinfer1::IPluginV2::getSerializationSize"

/external/yolov4-triton-tensorrt/layers/yololayer.h(57): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::getWorkspaceSize" is incompatible with that of overridden function "nvinfer1::IPluginV2::getWorkspaceSize"

/external/yolov4-triton-tensorrt/layers/yololayer.h(55): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::terminate" is incompatible with that of overridden function "nvinfer1::IPluginV2::terminate"

/external/yolov4-triton-tensorrt/layers/yololayer.h(53): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::initialize" is incompatible with that of overridden function "nvinfer1::IPluginV2::initialize"

/external/yolov4-triton-tensorrt/layers/yololayer.h(51): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::getOutputDimensions" is incompatible wit
h that of overridden function "nvinfer1::IPluginV2::getOutputDimensions"

/external/yolov4-triton-tensorrt/layers/yololayer.h(46): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::getNbOutputs" is incompatible with that
of overridden function "nvinfer1::IPluginV2::getNbOutputs"

/external/yolov4-triton-tensorrt/layers/yololayer.h(133): error: exception specification for virtual function "nvinfer1::YoloPluginCreator::getPluginNamespace" is incompatible w
ith that of overridden function "nvinfer1::IPluginCreator::getPluginNamespace"

/external/yolov4-triton-tensorrt/layers/yololayer.h(128): error: exception specification for virtual function "nvinfer1::YoloPluginCreator::setPluginNamespace" is incompatible w
ith that of overridden function "nvinfer1::IPluginCreator::setPluginNamespace"

/external/yolov4-triton-tensorrt/layers/yololayer.h(126): error: exception specification for virtual function "nvinfer1::YoloPluginCreator::deserializePlugin" is incompatible wi
th that of overridden function "nvinfer1::IPluginCreator::deserializePlugin"

/external/yolov4-triton-tensorrt/layers/yololayer.h(124): error: exception specification for virtual function "nvinfer1::YoloPluginCreator::createPlugin" is incompatible with th
at of overridden function "nvinfer1::IPluginCreator::createPlugin"

/external/yolov4-triton-tensorrt/layers/yololayer.h(122): error: exception specification for virtual function "nvinfer1::YoloPluginCreator::getFieldNames" is incompatible with t
hat of overridden function "nvinfer1::IPluginCreator::getFieldNames"

/external/yolov4-triton-tensorrt/layers/yololayer.h(120): error: exception specification for virtual function "nvinfer1::YoloPluginCreator::getPluginVersion" is incompatible wit
h that of overridden function "nvinfer1::IPluginCreator::getPluginVersion"

/external/yolov4-triton-tensorrt/layers/yololayer.h(118): error: exception specification for virtual function "nvinfer1::YoloPluginCreator::getPluginName" is incompatible with t
hat of overridden function "nvinfer1::IPluginCreator::getPluginName"

/external/yolov4-triton-tensorrt/layers/yololayer.cu(173): error: object of abstract class type "nvinfer1::YoloLayerPlugin" is not allowed:
            pure virtual function "nvinfer1::IPluginV2::enqueue" has no overrider

/external/yolov4-triton-tensorrt/layers/yololayer.cu(412): error: object of abstract class type "nvinfer1::YoloLayerPlugin" is not allowed:
            pure virtual function "nvinfer1::IPluginV2::enqueue" has no overrider

/external/yolov4-triton-tensorrt/layers/yololayer.cu(419): error: object of abstract class type "nvinfer1::YoloLayerPlugin" is not allowed:
            pure virtual function "nvinfer1::IPluginV2::enqueue" has no overrider

33 errors detected in the compilation of "/external/yolov4-triton-tensorrt/layers/yololayer.cu".
CMake Error at layerplugin_generated_yololayer.cu.o.cmake:279 (message):
  Error generating file
  /external/yolov4-triton-tensorrt/build/CMakeFiles/layerplugin.dir/layers/./layerplugin_generated_yololayer.cu.o


make[2]: *** [CMakeFiles/layerplugin.dir/build.make:65: CMakeFiles/layerplugin.dir/layers/layerplugin_generated_yololayer.cu.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:109: CMakeFiles/layerplugin.dir/all] Error 2
make: *** [Makefile:84: all] Error 2

[philipp-schmidt]

TensorRT 8 changed the way plugins work. I did not have time to look into it yet. Contributions are very welcomed!

[philipp-schmidt]

Usually the computational part of the plugin stays the same and only the interface around it changes, so it should not be too big of a change.

[revanb88]

I am also facing same issue, could you please help me. I used tensorrt:21.08 got this error

/yolov4-triton-tensorrt/layers/yololayer.h(59): error: member function declared with "override" does not override a base class member

/yolov4-triton-tensorrt/layers/yololayer.h(108): error: function "nvinfer1::IPluginV2IOExt::configurePlugin(const nvinfer1::Dims *, int32_t, const nvinfer1::Dims *, int32_t, const nvinfer1::DataType *, const nvinfer1::DataType *, const __nv_bool *, const __nv_bool *, nvinfer1::PluginFormat, int32_t)"
/usr/include/x86_64-linux-gnu/NvInferRuntimeCommon.h(836): here is inaccessible

/yolov4-triton-tensorrt/layers/yololayer.h(59): warning: function "nvinfer1::IPluginV2::enqueue(int32_t, const void *const *, void *const *, void *, cudaStream_t)" is hidden by "nvinfer1::YoloLayerPlugin::enqueue" -- virtual function override intended?

/yolov4-triton-tensorrt/layers/yololayer.h(89): warning: function "nvinfer1::IPluginV2Ext::configurePlugin(const nvinfer1::Dims *, int32_t, const nvinfer1::Dims *, int32_t, const nvinfer1::DataType *, const nvinfer1::DataType *, const __nv_bool *, const __nv_bool *, nvinfer1::PluginFormat, int32_t)" is hidden by "nvinfer1::YoloLayerPlugin::configurePlugin" -- virtual function override intended?

/yolov4-triton-tensorrt/layers/yololayer.h(91): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::detachFromContext" is incompatible with that of overridden function "nvinfer1::IPluginV2Ext::detachFromContext"

/yolov4-triton-tensorrt/layers/yololayer.h(89): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::configurePlugin" is incompatible with that of overridden function "nvinfer1::IPluginV2IOExt::configurePlugin(const nvinfer1::PluginTensorDesc *, int32_t, const nvinfer1::PluginTensorDesc *, int32_t)"

/yolov4-triton-tensorrt/layers/yololayer.h(87): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::attachToContext" is incompatible with that of overridden function "nvinfer1::IPluginV2Ext::attachToContext"

/yolov4-triton-tensorrt/layers/yololayer.h(85): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::canBroadcastInputAcrossBatch" is incompatible with that of overridden function "nvinfer1::IPluginV2Ext::canBroadcastInputAcrossBatch"

/yolov4-triton-tensorrt/layers/yololayer.h(83): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::isOutputBroadcastAcrossBatch" is incompatible with that of overridden function "nvinfer1::IPluginV2Ext::isOutputBroadcastAcrossBatch"

/yolov4-triton-tensorrt/layers/yololayer.h(81): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::getOutputDataType" is incompatible with that of overridden function "nvinfer1::IPluginV2Ext::getOutputDataType"

/yolov4-triton-tensorrt/layers/yololayer.h(79): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::getPluginNamespace" is incompatible with that of overridden function "nvinfer1::IPluginV2::getPluginNamespace"

/yolov4-triton-tensorrt/layers/yololayer.h(77): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::setPluginNamespace" is incompatible with that of overridden function "nvinfer1::IPluginV2::setPluginNamespace"

/yolov4-triton-tensorrt/layers/yololayer.h(75): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::clone" is incompatible with that of overridden function "nvinfer1::IPluginV2Ext::clone"

/yolov4-triton-tensorrt/layers/yololayer.h(75): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::clone" is incompatible with that of overridden function "nvinfer1::IPluginV2::clone"

/yolov4-triton-tensorrt/layers/yololayer.h(73): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::destroy" is incompatible with that of overridden function "nvinfer1::IPluginV2::destroy"

/yolov4-triton-tensorrt/layers/yololayer.h(71): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::getPluginVersion" is incompatible with that of overridden function "nvinfer1::IPluginV2::getPluginVersion"

/yolov4-triton-tensorrt/layers/yololayer.h(69): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::getPluginType" is incompatible with that of overridden function "nvinfer1::IPluginV2::getPluginType"

/yolov4-triton-tensorrt/layers/yololayer.h(65): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::supportsFormatCombination" is incompatible with that of overridden function "nvinfer1::IPluginV2IOExt::supportsFormatCombination"

/yolov4-triton-tensorrt/layers/yololayer.h(63): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::serialize" is incompatible with that of overridden function "nvinfer1::IPluginV2::serialize"

/yolov4-triton-tensorrt/layers/yololayer.h(61): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::getSerializationSize" is incompatible with that of overridden function "nvinfer1::IPluginV2::getSerializationSize"

/yolov4-triton-tensorrt/layers/yololayer.h(57): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::getWorkspaceSize" is incompatible with that of overridden function "nvinfer1::IPluginV2::getWorkspaceSize"

/yolov4-triton-tensorrt/layers/yololayer.h(55): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::terminate" is incompatible with that of overridden function "nvinfer1::IPluginV2::terminate"

/yolov4-triton-tensorrt/layers/yololayer.h(53): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::initialize" is incompatible with that of overridden function "nvinfer1::IPluginV2::initialize"

/yolov4-triton-tensorrt/layers/yololayer.h(51): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::getOutputDimensions" is incompatible with that of overridden function "nvinfer1::IPluginV2::getOutputDimensions"

/yolov4-triton-tensorrt/layers/yololayer.h(46): error: exception specification for virtual function "nvinfer1::YoloLayerPlugin::getNbOutputs" is incompatible with that of overridden function "nvinfer1::IPluginV2::getNbOutputs"

/yolov4-triton-tensorrt/layers/yololayer.h(133): error: exception specification for virtual function "nvinfer1::YoloPluginCreator::getPluginNamespace" is incompatible with that of overridden function "nvinfer1::IPluginCreator::getPluginNamespace"

/yolov4-triton-tensorrt/layers/yololayer.h(128): error: exception specification for virtual function "nvinfer1::YoloPluginCreator::setPluginNamespace" is incompatible with that of overridden function "nvinfer1::IPluginCreator::setPluginNamespace"

/yolov4-triton-tensorrt/layers/yololayer.h(126): error: exception specification for virtual function "nvinfer1::YoloPluginCreator::deserializePlugin" is incompatible with that of overridden function "nvinfer1::IPluginCreator::deserializePlugin"

/yolov4-triton-tensorrt/layers/yololayer.h(124): error: exception specification for virtual function "nvinfer1::YoloPluginCreator::createPlugin" is incompatible with that of overridden function "nvinfer1::IPluginCreator::createPlugin"

/yolov4-triton-tensorrt/layers/yololayer.h(122): error: exception specification for virtual function "nvinfer1::YoloPluginCreator::getFieldNames" is incompatible with that of overridden function "nvinfer1::IPluginCreator::getFieldNames"

/yolov4-triton-tensorrt/layers/yololayer.h(120): error: exception specification for virtual function "nvinfer1::YoloPluginCreator::getPluginVersion" is incompatible with that of overridden function "nvinfer1::IPluginCreator::getPluginVersion"

/yolov4-triton-tensorrt/layers/yololayer.h(118): error: exception specification for virtual function "nvinfer1::YoloPluginCreator::getPluginName" is incompatible with that of overridden function "nvinfer1::IPluginCreator::getPluginName"

/yolov4-triton-tensorrt/layers/yololayer.cu(173): error: object of abstract class type "nvinfer1::YoloLayerPlugin" is not allowed:
pure virtual function "nvinfer1::IPluginV2::enqueue" has no overrider

/yolov4-triton-tensorrt/layers/yololayer.cu(412): error: object of abstract class type "nvinfer1::YoloLayerPlugin" is not allowed:
pure virtual function "nvinfer1::IPluginV2::enqueue" has no overrider

/yolov4-triton-tensorrt/layers/yololayer.cu(419): error: object of abstract class type "nvinfer1::YoloLayerPlugin" is not allowed:
pure virtual function "nvinfer1::IPluginV2::enqueue" has no overrider

33 errors detected in the compilation of "/yolov4-triton-tensorrt/layers/yololayer.cu".
CMake Error at layerplugin_generated_yololayer.cu.o.cmake:279 (message):
Error generating file
/yolov4-triton-tensorrt/build/CMakeFiles/layerplugin.dir/layers/./layerplugin_generated_yololayer.cu.o

make[2]: *** [CMakeFiles/layerplugin.dir/build.make:65: CMakeFiles/layerplugin.dir/layers/layerplugin_generated_yololayer.cu.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:109: CMakeFiles/layerplugin.dir/all] Error 2
make: *** [Makefile:84: all] Error 2

[chull434]

yeah ran into the same issue when I update to jetson jetpack 4.6 from 4.5 and tensorRT upgraded to 8.0.1

[chull434]

I followed this pr's changes and got it to at least build now wang-xinyu/tensorrtx#540

Also note need to update the docker image to a TRT8 one, such as nvcr.io/nvidia/tensorrt:21.09-py3

[revanb88]

Thanks a lot. I will go through it.

…

On Wed, Nov 3, 2021 at 9:00 PM chull434 ***@***.***> wrote: I followed this pr's changes and got it to at least build now wang-xinyu/tensorrtx#540 <wang-xinyu/tensorrtx#540> Also note need to update the docker image to a TRT8 one, such as nvcr.io/nvidia/tensorrt:21.09-py3 — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#55 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AWDAXTIJB4DDQHIBALD76KLUKFIRHANCNFSM5F5DGZ3Q> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

-- Thanks and Regards Dr. Revanasiddappa M Biradar Data Scientist Bengaluru

[ohlr]

@revanb88, @chull434 - would you like to make this a PR?

[ohlr]

@Michelvl92 did you find a way to support TRT8?

[jenfredwell]

I've also ended up here with the same errors trying to use the 21.08 version. Curious about the status?

[philipp-schmidt]

No support yet. We will update this to TensorRT 8 at some point, no roadmap though. PR from someone else will probably be faster. Should only be a matter of applying necessary changes from the patch notes from nvidia to the new plugin interface.

[philipp-schmidt]

Newest v1.4.0 release comes with support for TRT 8. Tested with NGC TensorRT 21.10 and Triton 21.10 with 470 driver. Performance gain compared to previous 1.3.0 release is at about 19.5%.

Did not have the time to run and check the python client, so feel free to report any issues with that.