Error when building TensorRT Engine. #76
Comments
|
It seems that this issue is related to issue #30. I apologize for opening duplicate issues. |
|
I discovered another issue that might be related to this problem. When I installed import tensorrt
print(tensorrt.__version__)
assert tensorrt.Builder(tensorrt.Logger())
8.6.0
[05/06/2023-17:34:17] [TRT] [W] Unable to determine GPU memory usage
[05/06/2023-17:34:17] [TRT] [W] Unable to determine GPU memory usage
[05/06/2023-17:34:17] [TRT] [W] CUDA initialization failure with error: 35. Please check your CUDA installation: http://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.htmlHowever, when I used import tensorrt
print(tensorrt.__version__)
assert tensorrt.Builder(tensorrt.Logger())
8.5.3.1
[05/06/2023-17:38:45] [TRT] [W] CUDA lazy loading is not enabled. Enabling it can significantly reduce device memory usage. See `CUDA_MODULE_LOADING` in https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#env-varsBased on the NVIDIA TensorRT Install Guide, this error is likely due to an incorrect NVIDIA Driver installation. However, since I am using a managed Docker image, it may not be compatible with |
|
TensorRT has recommended CUDA 12.x, try to update the drivers if you have the option |
Yes, this must be the main problem. Unfortunately, I don't have the option to upgrade the CUDA version. It appears that Radiata TensorRT is incompatible with |
|
I seem to have this issue as well. The cuda version is 12.1. Does anyone have fixes? |
|
Is cudnn properly installed on the machine being used? |
|
I never downloaded it specifically. I just did the set up steps for Linux on a Ubuntu WSL distribution. |
Yes, it is likely installed properly. I use a managed Docker image, and I am able to generate normal images using the default diffusers. $: nvidia-smi
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.161.03 Driver Version: 470.161.03 CUDA Version: 11.4 |
|-------------------------------+----------------------+----------------------+$: nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Mon_Apr__3_17:16:06_PDT_2023
Cuda compilation tools, release 12.1, V12.1.105
Build cuda_12.1.r12.1/compiler.32688072_0
Tue May 9 14:34:12 2023 import tensorflow as tf
sys_details = tf.sysconfig.get_build_info()
cuda_version = sys_details["cuda_version"]
cudnn_version = sys_details["cudnn_version"]
print(f'cuda: {cuda_version}', f'cudnn: {cudnn_version}')
$: cuda: 11.2 cudnn: 8I'm not sure why the CUDA version from TensorFlow is mismatched with |
|
I am able to generate normal image as well, but the error shows up when I use --tensorrt flag. I should say I am using anaconda with Ubuntu WSL. Should I just reinstall everything from scratch? |
|
0zl, you mentioned a docker image. Is it the one in the tensorrt 8.5 version? |
The public PyTorch docker image I am using does not come with TensorRT pre-installed. For convenience, you can use the TensorRT pre-installed docker container provided by NVIDIA. |
|
that sounds like a good idea, thanks. i'll try it out. |
Have you solved it? i met the same trouble,but i didnt receive the detail of bug, in addition: |
have you solved it? @0zl |
There seems to be a version mismatch everywhere. That will cause some problems. This usually happens with poorly installed NVIDIA drivers and CUDA. As for how to fix this stuff, if you are on a Linux host, then it's not exactly simple. You are usually better off just purging everything with Can you tell me what exact environment are you running in (OS, Docker Image, GPU...) |
Describe the bug
When attempting to build TensorRT Engine, an error message is displayed indicating that cuDNN is not initialized (
CUDNN_STATUS_NOT_INITIALIZED).Error Log
Reproduction
launch.shwith--shareand--tensorrtcommand argumentsunet.onnx.Expected behavior
It should be exported and without throwing any errors.
System Info
launch.py)launch.py)Additional context
No response
Validations
The text was updated successfully, but these errors were encountered: