Current version change to 3.6 Object detect issue

[hagsac]

Preliminary Checks

• This issue is not a duplicate. Before opening a new issue, please search existing issues.
• This issue is not a question, feature request, or anything other than a bug report directly related to this project.

Description

In the previous CUDA 11.1, 3.5.X SDK is normally used,
After installing CUDA 11.4, and 3.6 SDK,
There is a problem in the dialog history of the ZED SDK and Graphics Card.

After changing to CUDA 10.2 and installing 3.6SDK, the dialog is normal as shown in the figure below, but Object Detect does not work properly.

Steps to Reproduce

1. Install cuda 11.4
2. Install sdk 3.6
3. object detection example failed
4. zed sdk, graphics card problem occurred in diagnostic
5. Reinstall multiple times;
6. Failure
7. Install cuda 10.2
8. Install sdk 3.6
9. diagnostic normal
10. object detection example failed

Expected Result

Normal operation.

Actual Result

ZED Camera model

ZED2i

Environment

win 10
Processor Intel(R) Core(TM) i7-8700 CPU @ 3.20 GHz 3.19 GHz
16.0 GB of installed RAM
System type 64-bit operating system, x64-based processor
rtx 2070

1.cuda 11.4 sdk 3.6,
2.cuda 11.2 sdk 3.6

Anything else?

No response

[harishkool]

I am also facing the same issue with the newer version of SDK, cuda version Iam using is 10.2. Check the below message:

[ZED][Object Detection] Please wait while the AI model is being optimized for your graphics card This operation will be run only once and may take a few minutes \Segmentation fault (core dumped)

[adujardin]

Could you try to uninstall CUDA then completely remove the CUDA folder and reinstall it ? I suspect it's a conflict of version with TensorRT and/or cuDNN installed by 3.5.

[hagsac]

Could you try to uninstall CUDA then completely remove the CUDA folder and reinstall it ? I suspect it's a conflict of version with TensorRT and/or cuDNN installed by 3.5.

Apparently there is a problem with cuDNN. I just tried installing cuda toolkit several times but it didn't solve the problem.
I have installed cuDNN separately in the past.
I would like to know which version of cuDNN is required for zed sdk 3.6.

[adujardin]

The 3.6 is currently using cuDNN 8.2 (3.6.0 installs 8.2.1). Installing it separately is not something we recommend to avoid issues.

[hagsac]

The 3.6 is currently using cuDNN 8.2 (3.6.0 installs 8.2.1). Installing it separately is not something we recommend to avoid issues.

I also have no experience with separately installing cunn while using zed. But I checked the current cuda toolkit folder,
cuda 10.2 had cudnn inside. (sdk works normally)
But from cuda 11.x, cudnn is not installed in the internal folder.
Obviously, after deleting all folders and reinstalling, cudnn was not installed. So I'm trying it out separately.

Another question is, is there any difference in object detection performance between cuda 10 version and 11?

[adujardin]

Ok we'll investigate this issue, cuDNN should be installed with cuda 11. We didn't notice any major performance differences between cuda 10.2 and 11 for the object detection.

[hagsac]

Ok we'll investigate this issue, cuDNN should be installed with cuda 11. We didn't notice any major performance differences between cuda 10.2 and 11 for the object detection.

I just reinstalled and checked.
After installing cuda11.4, I checked that cudnn*.dll is not in the bin, include folders.

Even in the cudnn 8.x version of the manual, it is recommended to copy the .dll file to the cuda11 related folder when installing windows.

My computer may be the problem, but please review.

[harishkool]

Could you try to uninstall CUDA then completely remove the CUDA folder and reinstall it ? I suspect it's a conflict of version with TensorRT and/or cuDNN installed by 3.5.

I am currently using Cuda-10.2 and CUDNN 7.6.5, do SDK 3.6 doesn't work this version of cuda and cudnn?, do I have to install cuda-11 and cuDNN 8.2 to work with SDK 3.6 ?

[adujardin]

@hagsac There seems to be an issue, you should indeed have the cudnn files in the cuda 11 folder. We'll do some testing to try to reproduce this.

@harishkool If you need specifically cudnn 7.6.5 for other libraries then no, you should not upgrade. 3.6 is compatible with cuda 10.2 and 11.x and installs it's the required version of cudnn and tensorrt automatically.

[adujardin]

@hagsac I checked and on Windows we install cudnn and tensorrt in the ZED SDK installation folder.
If possible you should try using cuda 11.3 instead of 11.4. Tensorrt could have a compatibility issue with 11.4 for 3.6.0. We'll try to fix this in a 3.6.1

[hagsac]

@hagsac I checked and on Windows we install cudnn and tensorrt in the ZED SDK installation folder. If possible you should try using cuda 11.3 instead of 11.4. Tensorrt could have a compatibility issue with 11.4 for 3.6.0. We'll try to fix this in a 3.6.1

Thanks for your help.
Because of this problem, I spent three days.
Please do a detailed test from next time.

In fact, this problem existed in version 3.5.x.Due to some of my actions, it became a normal operation, and although it has been used so far, an error occurred in the sdk check of the actual [zed 3.5.x Diagnostic app].
However, I raised the issue now because there was a communication problem.

And I found a hardware problem that occurs in zed2, zed2i that I learned a few months ago, where should I submit this problem?
This problem is related to static electricity.

[Myzhar]

@hagsac regarding the hardware problem please write an email to support@stereolabs.com
Please provide as many details as possible

[harishkool]

@Myzhar @adujardin I have uninstalled cuda-10.2 and installed cuda-11.3 and cudnn-8.2, I am still facing this issue.

[ZED][Object Detection] Please wait while the AI model is being optimized for your graphics card This operation will be run only once and may take a few minutes |Segmentation fault (core dumped)

[adujardin]

Ok, so it's not about the version of CUDA then.

• Do you have this issue with all models?

• Could you try running the Diagnostic tool with the new option:

First clean all remaining model related files:

/usr/local/zed/tools/ZED_Diagnostic -aic

Then try to optimize them:

/usr/local/zed/tools/ZED_Diagnostic -aio

• Could you run the sample in gdb to have more information about the stack (especially where it's crashing)?

• Could you post the output of:

ldd /usr/local/zed/sl_ai.so

• Could you post your diagnostic file report?

[adujardin]

@hagsac The 3.6.1 has been released and should fix your issue with CUDA 11.4, if it doesn't please follow the procedure above, thank you

[harishkool]

Ok, so it's not about the version of CUDA then.

• Do you have this issue with all models?
• Could you try running the Diagnostic tool with the new option:

First clean all remaining model related files:

/usr/local/zed/tools/ZED_Diagnostic -aic

Then try to optimize them:

/usr/local/zed/tools/ZED_Diagnostic -aio

• Could you run the sample in gdb to have more information about the stack (especially where it's crashing)?
• Could you post the output of:

ldd /usr/local/zed/sl_ai.so

• Could you post your diagnostic file report?

I don't understand, I have installed cuda-10.2 again and tried the newer SDK 3.6.1 released, I am not facing segmentation faults now. Will work more with the object detection API and get back to you If I face any issues.

[Murdism]

Ok, so it's not about the version of CUDA then.

• Do you have this issue with all models?
• Could you try running the Diagnostic tool with the new option:

First clean all remaining model related files:

/usr/local/zed/tools/ZED_Diagnostic -aic

Then try to optimize them:

/usr/local/zed/tools/ZED_Diagnostic -aio

• Could you run the sample in gdb to have more information about the stack (especially where it's crashing)?
• Could you post the output of:

ldd /usr/local/zed/sl_ai.so

• Could you post your diagnostic file report?

Hello i have the same issue,

i am using

ubuntu 18.04 with cuda 10.2 and ZED sdk 3.7.0 GPU RTX 2080 with 12GB

object detection example causes segmentation fault.

-> here is a result /usr/local/zed/tools/ZED_Diagnostic -aio

Optimizing all AI models
Optimizing: MULTI CLASS DETECTION...
[ZED][INFO] AI model not found, downloading
[ZED][INFO] Done
[ZED][INFO] Done
|[ZED][INFO] Please wait while the AI model is being optimized for your graphics card
This operation will be run only once and may take a few minutes
|Stack trace (most recent call last):
#11 Object "[0xffffffffffffffff]", at 0xffffffffffffffff, in
#10 Object "/usr/local/zed/tools/ZED_Diagnostic", at 0x432ef9, in
#9 Object "/lib/x86_64-linux-gnu/libc.so.6", at 0x7f2805237c86, in __libc_start_main
#8 Object "/usr/local/zed/tools/ZED_Diagnostic", at 0x41bc8d, in
#7 Object "/usr/local/zed/tools/ZED_Diagnostic", at 0x434b35, in
#6 Object "/usr/local/zed/tools/ZED_Diagnostic", at 0x46d3f2, in
#5 Object "/usr/local/zed/tools/ZED_Diagnostic", at 0x46c9ca, in
#4 Object "/usr/local/zed/lib/libsl_ai.so", at 0x7f27febfa827, in
#3 Object "/usr/local/zed/lib/libsl_ai.so", at 0x7f27febfdf30, in
#2 Object "/usr/local/zed/lib/libsl_ai.so", at 0x7f27fec3166e, in
#1 Object "/usr/local/zed/lib/libsl_ai.so", at 0x7f27fec2dc2a, in
#0 Object "/usr/local/zed/lib/libsl_ai.so", at 0x7f27fec29a34, in
Segmentation fault (Address not mapped to object [0x8])
Segmentation fault (core dumped)

Running ldd /usr/local/zed/sl_ai.so shows:
ldd: /usr/local/zed/sl_ai.so: No such file or directory

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment otherwise it will be automatically closed in 5 days