trt.Builder failure of TensorRT 8.5.1.7 when running python3.9.18 and python3.8.18 (miniconda) on GPU Tesla T4

[PigletOS]

Description

I tried to run tensorrt 8.5.1.7 (python3.9.18, python3.8.18, miniconda envrioment)，but it fails with the error below :

[TRT] [I] [MemUsageChange] Init CUDA: CPU +15, GPU +0, now: CPU 29, GPU 105 (MiB)
[TRT] [V] Trying to load shared library libnvinfer_builder_resource.so.8.5.1
[TRT] [V] Loaded shared library libnvinfer_builder_resource.so.8.5.1
[TRT] [E] 2: [builder.cpp::createCaskKernelLibraryImpl::157] Error Code 2: Internal Error (Assertion validateCaskKLibSize(buffer.size) failed. )
TypeError: pybind11::init(): factory function returned nullptr

It can run on python3.6 (without the miniconda environment).

Environment

TensorRT Version: 8.5.1.7

NVIDIA GPU: Tesla T4

NVIDIA Driver Version: 535.161.07

CUDA Version: 10.2

CUDNN Version: 8.6.0 (which I found in /usr/include/cudnn_version.h)
#define CUDNN_MAJOR 8
#define CUDNN_MINOR 6
#define CUDNN_PATCHLEVEL 0
#define CUDNN_VERSION (CUDNN_MAJOR * 1000 + CUDNN_MINOR * 100 + CUDNN_PATCHLEVEL)

Operating System: Ubuntu 20.04

Python Version (if applicable): python3.9.18, python3.8.18 (miniconda)

Relevant Files

Model link:

Steps To Reproduce

Commands or scripts:

import tensorrt as trt
trt_logger = trt.Logger(trt.Logger.VERBOSE)
builder = trt.Builder(trt_logger)

[lix19937]

It seems version not match.

[PigletOS]

After I run "apt remove -y libnvinfer8", the problem was solved.
It seems the "libnv*.so" files are already in "~/miniconda/lib/python3.9/site-packages/tensorrt/" and "/usr/lib/x86_64-linux-gnu/libnv*.so" are no longer needed.

[lix19937]

So your env has multiple tensorrt related so.