TensorRT 8.4.0.6 fallback to 8.2.5 when using trtexec

[deephog]

It is a weird thing that even though I have installed a TensorRT of version 8.4.0.6, sometimes it falls back to 8.2.5 when compiling certain models. I don't know if it relates to the opset version I used for my ONNX model, or is it related to certain layers or operations I used in the model. It will be great if someone can explain the general rules to me. Thanks!

[pranavm-nvidia]

trtexec just uses the OS's dynamic loader, so it doesn't choose which version of TensorRT to load.
Have you set your LD_LIBRARY_PATH correctly?

[deephog]

Here is everything I have when I do "pip list", almost the same when I do "conda list", basically 8.4.0.6 is the only one I have.

absl-py 1.0.0
addict 2.4.0
alabaster 0.7.12
apex 0.1
appdirs 1.4.4
argon2-cffi 21.3.0
argon2-cffi-bindings 21.2.0
asttokens 2.0.5
attrs 21.4.0
audioread 2.1.9
Babel 2.10.1
backcall 0.2.0
backports.functools-lru-cache 1.6.4
beautifulsoup4 4.11.1
bleach 5.0.0
blis 0.7.7
brotlipy 0.7.0
cachetools 5.0.0
catalogue 2.0.6
certifi 2021.10.8
cffi 1.15.0
chardet 4.0.0
charset-normalizer 2.0.12
click 8.0.4
cloudpickle 2.0.0
codecov 2.1.12
colorama 0.4.4
commonmark 0.9.1
conda 4.12.0
conda-build 3.21.8
conda-package-handling 1.8.1
correlation 0.0.0
coverage 6.3.2
cryptography 36.0.2
cuda-python 11.6.1
cudf 22.4.0a0+306.g0cb75a4913
cugraph 22.4.0a0+102.g4106a188
cuml 22.4.0a0+108.g2be11269d
cupy-cuda115 9.6.0
cycler 0.11.0
cymem 2.0.6
Cython 0.29.28
dask 2022.3.0
dask-cuda 22.4.0
dask-cudf 22.4.0a0+306.g0cb75a4913
dataclasses 0.8
debugpy 1.6.0
decorator 5.1.1
defusedxml 0.7.1
distributed 2022.3.0
docker-pycreds 0.4.0
docutils 0.17.1
einops 0.4.1
entrypoints 0.4
executing 0.8.3
expecttest 0.1.3
fastjsonschema 2.15.3
fastrlock 0.8
filelock 3.6.0
Flask 2.1.2
flatbuffers 2.0
fonttools 4.33.3
fsspec 2022.3.0
future 0.18.2
gitdb 4.0.9
GitPython 3.1.27
glob2 0.7
google-auth 2.6.6
google-auth-oauthlib 0.4.6
graphsurgeon 0.4.5
grpcio 1.46.0
HeapDict 1.0.1
hypothesis 4.50.8
idna 3.3
imageio 2.19.3
imagesize 1.3.0
importlib-metadata 4.11.3
importlib-resources 5.7.1
iniconfig 1.1.1
ipykernel 6.13.0
ipython 8.3.0
ipython-genutils 0.2.0
itsdangerous 2.1.2
jedi 0.18.1
Jinja2 3.1.2
joblib 1.1.0
json5 0.9.6
jsonschema 4.5.1
jupyter-client 7.3.0
jupyter-core 4.10.0
jupyter-tensorboard 0.2.0
jupyterlab 2.3.2
jupyterlab-pygments 0.2.2
jupyterlab-server 1.2.0
jupytext 1.13.8
kiwisolver 1.4.2
kornia 0.6.5
langcodes 3.3.0
libarchive-c 4.0
librosa 0.8.1
llvmlite 0.36.0
lmdb 1.3.0
locket 1.0.0
Markdown 3.3.7
markdown-it-py 2.1.0
MarkupSafe 2.1.1
matplotlib 3.5.2
matplotlib-inline 0.1.3
mdit-py-plugins 0.3.0
mdurl 0.1.1
mistune 0.8.4
mmcv-full 1.5.3
mock 4.0.3
msgpack 1.0.3
murmurhash 1.0.7
nbclient 0.6.2
nbconvert 6.5.0
nbformat 5.4.0
nest-asyncio 1.5.5
networkx 2.6.3
nltk 3.7
notebook 6.4.1
numba 0.53.1
numpy 1.22.3
nvidia-cublas 11.5.1.101
nvidia-cublas-cu11 2022.4.8
nvidia-cublas-cu117 11.10.1.25
nvidia-cuda-nvrtc 11.3.58
nvidia-cuda-runtime 11.3.58
nvidia-cuda-runtime-cu11 2022.4.25
nvidia-cuda-runtime-cu117 11.7.60
nvidia-cudnn 8.2.0.51
nvidia-cudnn-cu11 2022.5.19
nvidia-cudnn-cu116 8.4.0.27
nvidia-dali-cuda110 1.13.0
nvidia-pyindex 1.0.9
nvidia-tensorrt 8.4.0.6
nvtx 0.2.4
oauthlib 3.2.0
onnx 1.11.0
onnx-simplifier 0.3.10
onnxoptimizer 0.2.7
onnxruntime 1.11.1
onnxruntime-gpu 1.11.1
opencv-python 4.5.5.64
packaging 21.3
pandas 1.3.5
pandocfilters 1.5.0
parameterized 0.8.1
parso 0.8.3
partd 1.2.0
pathtools 0.1.2
pathy 0.6.1
pexpect 4.8.0
pickleshare 0.7.5
Pillow 9.0.1
pip 22.1.2
pkginfo 1.8.2
pluggy 1.0.0
polygraphy 0.33.0
pooch 1.6.0
preshed 3.0.6
prettytable 3.3.0
prometheus-client 0.14.1
promise 2.3
prompt-toolkit 3.0.29
protobuf 3.20.1
psutil 5.9.0
ptyprocess 0.7.0
pure-eval 0.2.2
py 1.11.0
pyarrow 6.0.1
pyasn1 0.4.8
pyasn1-modules 0.2.8
pybind11 2.9.2
pycocotools 2.0+nv0.6.0
pycosat 0.6.3
pycparser 2.21
pydantic 1.8.2
pydot 1.4.2
Pygments 2.12.0
pynvml 11.4.1
pyOpenSSL 22.0.0
pyparsing 3.0.8
pyrsistent 0.18.1
PySocks 1.7.1
pytest 6.2.5
pytest-cov 3.0.0
pytest-pythonpath 0.7.4
python-dateutil 2.8.2
python-hostlist 1.21
python-nvd3 0.15.0
python-slugify 6.1.2
pytorch-quantization 2.1.2
pytz 2022.1
PyWavelets 1.3.0
PyYAML 6.0
pyzmq 22.3.0
raft 22.4.0a0+113.gf5d2627
regex 2022.4.24
requests 2.27.1
requests-oauthlib 1.3.1
resampy 0.2.2
revtok 0.0.3
rich 12.4.4
rmm 22.4.0a0+50.gf82d458
rsa 4.8
ruamel-yaml-conda 0.15.80
sacremoses 0.0.53
scikit-image 0.19.3
scikit-learn 0.24.2
scipy 1.6.3
Send2Trash 1.8.0
sentry-sdk 1.6.0
setproctitle 1.2.3
setuptools 62.6.0
shellingham 1.4.0
shortuuid 1.0.9
six 1.16.0
smart-open 5.2.1
smmap 5.0.0
snowballstemmer 2.2.0
sortedcontainers 2.4.0
SoundFile 0.10.3.post1
soupsieve 2.3.1
spacy 3.3.0
spacy-legacy 3.0.9
spacy-loggers 1.0.2
Sphinx 4.5.0
sphinx-glpi-theme 0.3
sphinx-rtd-theme 1.0.0
sphinxcontrib-applehelp 1.0.2
sphinxcontrib-devhelp 1.0.2
sphinxcontrib-htmlhelp 2.0.0
sphinxcontrib-jsmath 1.0.1
sphinxcontrib-qthelp 1.0.3
sphinxcontrib-serializinghtml 1.1.5
srsly 2.4.3
stack-data 0.2.0
tabulate 0.8.9
tblib 1.7.0
tensorboard 2.9.0
tensorboard-data-server 0.6.1
tensorboard-plugin-wit 1.8.1
terminado 0.13.3
text-unidecode 1.3
thinc 8.0.15
threadpoolctl 3.1.0
tifffile 2022.5.4
timm 0.5.4
tinycss2 1.1.1
toml 0.10.2
tomli 2.0.1
toolz 0.11.2
torch 1.13.0.dev20220629+cu116
torch-tensorrt 1.1.0a0
torchaudio 0.13.0.dev20220629+cu116
torchtext 0.13.0a0
torchvision 0.14.0.dev20220629+cu116
tornado 6.1
tqdm 4.64.0
traitlets 5.1.1
treelite 2.3.0
treelite-runtime 2.3.0
typer 0.4.1
typing_extensions 4.2.0
ucx-py 0.25.0a0+13.ga16f8a2
uff 0.6.9
urllib3 1.26.9
wandb 0.12.20
wasabi 0.9.1
wcwidth 0.2.5
webencodings 0.5.1
Werkzeug 2.1.2
wheel 0.37.1
xgboost 1.5.2
yapf 0.32.0
zict 2.2.0
zipp 3.8.0

I used the newest Pytorch docker image from NGC, which initially contains a tensorrt of version 8.2.5 . Later I uninstalled it, and installed the 8.4.0.6 via pip (which shows up as nvidia-tensorrt, I still don't know what is the difference, but it runs ok).

Here is when I import tensorrt in the python and print out its version:

Python 3.8.13 | packaged by conda-forge | (default, Mar 25 2022, 06:04:10)
[GCC 10.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

import tensorrt
tensorrt.version
'8.4.0.6'

then I run:

trtexec --onnx=onnx_dir --saveEngine=engine_dir --workspace=4096 --fp16 --noTF32 --verbose --noDataTransfers --separateProfileRun --dumpProfile --useCudaGraph > log.txt

to compile the engine. But in the log file, it shows:

[06/30/2022-10:17:33] [I] TensorRT version: 8.2.5

here is the full log: link

Maybe I didn't cleanly uninstall 8.2.5? But how could a half uninstalled package still be working?

[deephog]

Ok I know where I am wrong. I need to rebuild the trtexec after I manually install the tensorrt.

[PrajwalCogniac]

I am facing the same issue .... What do you mean rebuild the trtexec ? @deephog @pranavm-nvidia

[deephog]

I am facing the same issue .... What do you mean rebuild the trtexec ? @deephog @pranavm-nvidia

I was using the docker image released by Nvidia, basically they have trtexec pre-built in the docker image, when you just downgrade your tensorrt, and when you run trtexec, it is still running the pre-built trtexec which is another version. Basically you need to re-build the trtexec following their documentation.