Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can't Import cv2 after Jetson Update #41

Closed
ben-mcdonnell-hacksmith opened this issue Jul 20, 2020 · 21 comments
Closed

Can't Import cv2 after Jetson Update #41

ben-mcdonnell-hacksmith opened this issue Jul 20, 2020 · 21 comments

Comments

@ben-mcdonnell-hacksmith

I am running an Xavier NX with Jetpack 4.2. After an OTA update cv2 won't import (before or after building with this script). To on my system I can build OpenCV running sudo ./build_opencv.sh master and trying to import cv2 in python 3.6.9
image

My OpenCV build information is:
General configuration for OpenCV 4.4.0-dev =====================================
Version control: e65e23b

Extra modules:
Location (extra): /tmp/build_opencv/opencv_contrib/modules
Version control (extra): 4.4.0

Platform:
Timestamp: 2020-07-20T13:29:52Z
Host: Linux 4.9.140-tegra aarch64
CMake: 3.17.3
CMake generator: Unix Makefiles
CMake build tool: /usr/bin/make
Configuration: RELEASE

CPU/HW features:
Baseline: NEON FP16
required: NEON
disabled: VFPV3

C/C++:
Built as dynamic libs?: YES
C++ standard: 11
C++ Compiler: /usr/bin/c++ (ver 7.5.0)
C++ flags (Release): -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Winit-self -Wsuggest-override -Wno-delete-non-virtual-dtor -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections -fvisibility=hidden -fvisibility-inlines-hidden -O3 -DNDEBUG -DNDEBUG
C++ flags (Debug): -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Winit-self -Wsuggest-override -Wno-delete-non-virtual-dtor -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections -fvisibility=hidden -fvisibility-inlines-hidden -g -O0 -DDEBUG -D_DEBUG
C Compiler: /usr/bin/cc
C flags (Release): -fsigned-char -W -Wall -Werror=return-type -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wshadow -Wuninitialized -Winit-self -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections -fvisibility=hidden -O3 -DNDEBUG -DNDEBUG
C flags (Debug): -fsigned-char -W -Wall -Werror=return-type -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wshadow -Wuninitialized -Winit-self -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections -fvisibility=hidden -g -O0 -DDEBUG -D_DEBUG
Linker flags (Release): -Wl,--gc-sections -Wl,--as-needed
Linker flags (Debug): -Wl,--gc-sections -Wl,--as-needed
ccache: NO
Precompiled headers: NO
Extra dependencies: m pthread cudart_static dl rt nppc nppial nppicc nppicom nppidei nppif nppig nppim nppist nppisu nppitc npps cublas cudnn cufft -L/usr/local/cuda/lib64 -L/usr/lib/aarch64-linux-gnu
3rdparty dependencies:

OpenCV modules:
To be built: alphamat aruco bgsegm bioinspired calib3d ccalib core cudaarithm cudabgsegm cudacodec cudafeatures2d cudafilters cudaimgproc cudalegacy cudaobjdetect cudaoptflow cudastereo cudawarping cudev datasets dnn dnn_objdetect dnn_superres dpm face features2d flann freetype fuzzy gapi hfs highgui img_hash imgcodecs imgproc intensity_transform line_descriptor ml objdetect optflow phase_unwrapping photo plot python2 python3 quality rapid reg rgbd saliency shape stereo stitching structured_light superres surface_matching text tracking video videoio videostab xfeatures2d ximgproc xobjdetect xphoto
Disabled: world
Disabled by dependency: -
Unavailable: cnn_3dobj cvv hdf java js julia matlab ovis sfm ts viz
Applications: apps
Documentation: NO
Non-free algorithms: YES

GUI:
GTK+: YES (ver 3.22.30)
GThread : YES (ver 2.56.4)
GtkGlExt: NO
OpenGL support: NO
VTK support: NO

Media I/O:
ZLib: /usr/lib/aarch64-linux-gnu/libz.so (ver 1.2.11)
JPEG: /usr/lib/aarch64-linux-gnu/libjpeg.so (ver 80)
WEBP: build (ver encoder: 0x020f)
PNG: /usr/lib/aarch64-linux-gnu/libpng.so (ver 1.6.34)
TIFF: /usr/lib/aarch64-linux-gnu/libtiff.so (ver 42 / 4.0.9)
JPEG 2000: build Jasper (ver 1.900.1)
OpenEXR: build (ver 2.3.0)
HDR: YES
SUNRASTER: YES
PXM: YES
PFM: YES

Video I/O:
DC1394: YES (2.2.5)
FFMPEG: YES
avcodec: YES (57.107.100)
avformat: YES (57.83.100)
avutil: YES (55.78.100)
swscale: YES (4.8.100)
avresample: YES (3.7.0)
GStreamer: YES (1.14.5)
v4l/v4l2: YES (linux/videodev2.h)

Parallel framework: pthreads

Trace: YES (with Intel ITT)

Other third-party libraries:
Lapack: YES (/usr/lib/aarch64-linux-gnu/liblapack.so /usr/lib/aarch64-linux-gnu/libcblas.so /usr/lib/aarch64-linux-gnu/libatlas.so)
Eigen: YES (ver 3.3.4)
Custom HAL: YES (carotene (ver 0.0.1))
Protobuf: build (3.5.1)

NVIDIA CUDA: YES (ver 10.2, CUFFT CUBLAS FAST_MATH)
NVIDIA GPU arch: 53 62 72
NVIDIA PTX archs:

cuDNN: YES (ver 8.0)

OpenCL: YES (no extra features)
Include path: /tmp/build_opencv/opencv/3rdparty/include/opencl/1.2
Link libraries: Dynamic load

Python 2:
Interpreter: /usr/bin/python2.7 (ver 2.7.17)
Libraries: /usr/lib/aarch64-linux-gnu/libpython2.7.so (ver 2.7.17)
numpy: /usr/lib/python2.7/dist-packages/numpy/core/include (ver 1.13.3)
install path: lib/python2.7/dist-packages/cv2/python-2.7

Python 3:
Interpreter: /usr/bin/python3 (ver 3.6.9)
Libraries: /usr/lib/aarch64-linux-gnu/libpython3.6m.so (ver 3.6.9)
numpy: /usr/lib/python3/dist-packages/numpy/core/include (ver 1.13.3)
install path: lib/python3.6/dist-packages/cv2/python-3.6

Python (for build): /usr/bin/python2.7

Java:
ant: NO
JNI: NO
Java wrappers: NO
Java tests: NO

Install to: /usr/local

@mdegans
Copy link
Owner

mdegans commented Jul 20, 2020

Are you sure your JetPack version is 4.2 on NX, @ben-mcdonnell-hacksmith ?

@ben-mcdonnell-hacksmith
Copy link
Author

@mdegans the version I installed was r32.4.2. Is there something else I should check?

@mdegans
Copy link
Owner

mdegans commented Jul 20, 2020

@ben-mcdonnell-hacksmith

Gotcha. That's JetPack 4.4. It's super confusing since the r32.xx versions do not align with the JetPack version.

I am running a test build now on an NX of the same release. I will let you know if I hit the same error.

@mdegans
Copy link
Owner

mdegans commented Jul 20, 2020

So, I can't replicate. Here are my steps to a working install. My starting point is a more or less unmodified, fully-updated JetPack 4.4.

To build and install, I did:

./build_opencv.sh 4.4.0

My output from opencv_version --verbose is:

 $ opencv_version --verbose

General configuration for OpenCV 4.4.0 =====================================
  Version control:               4.4.0

  Extra modules:
    Location (extra):            /tmp/build_opencv/opencv_contrib/modules
    Version control (extra):     4.4.0

  Platform:
    Timestamp:                   2020-07-20T20:21:24Z
    Host:                        Linux 4.9.140-tegra aarch64
    CMake:                       3.10.2
    CMake generator:             Unix Makefiles
    CMake build tool:            /usr/bin/make
    Configuration:               RELEASE

  CPU/HW features:
    Baseline:                    NEON FP16
      required:                  NEON
      disabled:                  VFPV3

  C/C++:
    Built as dynamic libs?:      YES
    C++ standard:                11
    C++ Compiler:                /usr/bin/c++  (ver 7.5.0)
    C++ flags (Release):         -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Winit-self -Wsuggest-override -Wno-delete-non-virtual-dtor -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections    -fvisibility=hidden -fvisibility-inlines-hidden -O3 -DNDEBUG  -DNDEBUG
    C++ flags (Debug):           -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Winit-self -Wsuggest-override -Wno-delete-non-virtual-dtor -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections    -fvisibility=hidden -fvisibility-inlines-hidden -g  -O0 -DDEBUG -D_DEBUG
    C Compiler:                  /usr/bin/cc
    C flags (Release):           -fsigned-char -W -Wall -Werror=return-type -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wshadow -Wuninitialized -Winit-self -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections    -fvisibility=hidden -O3 -DNDEBUG  -DNDEBUG
    C flags (Debug):             -fsigned-char -W -Wall -Werror=return-type -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wshadow -Wuninitialized -Winit-self -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections    -fvisibility=hidden -g  -O0 -DDEBUG -D_DEBUG
    Linker flags (Release):      -Wl,--gc-sections -Wl,--as-needed  
    Linker flags (Debug):        -Wl,--gc-sections -Wl,--as-needed  
    ccache:                      NO
    Precompiled headers:         NO
    Extra dependencies:          m pthread cudart_static -lpthread dl rt nppc nppial nppicc nppicom nppidei nppif nppig nppim nppist nppisu nppitc npps cublas cudnn cufft -L/usr/local/cuda/lib64 -L/usr/lib/aarch64-linux-gnu
    3rdparty dependencies:

  OpenCV modules:
    To be built:                 alphamat aruco bgsegm bioinspired calib3d ccalib core cudaarithm cudabgsegm cudacodec cudafeatures2d cudafilters cudaimgproc cudalegacy cudaobjdetect cudaoptflow cudastereo cudawarping cudev datasets dnn dnn_objdetect dnn_superres dpm face features2d flann freetype fuzzy gapi hfs highgui img_hash imgcodecs imgproc intensity_transform line_descriptor ml objdetect optflow phase_unwrapping photo plot python2 python3 quality rapid reg rgbd saliency shape stereo stitching structured_light superres surface_matching text tracking video videoio videostab xfeatures2d ximgproc xobjdetect xphoto
    Disabled:                    world
    Disabled by dependency:      -
    Unavailable:                 cnn_3dobj cvv hdf java js julia matlab ovis sfm ts viz
    Applications:                apps
    Documentation:               NO
    Non-free algorithms:         YES

  GUI: 
    GTK+:                        YES (ver 3.22.30)
      GThread :                  YES (ver 2.56.4)
      GtkGlExt:                  NO
    OpenGL support:              NO
    VTK support:                 NO

  Media I/O: 
    ZLib:                        /usr/lib/aarch64-linux-gnu/libz.so (ver 1.2.11)
    JPEG:                        /usr/lib/aarch64-linux-gnu/libjpeg.so (ver 80)
    WEBP:                        build (ver encoder: 0x020f)
    PNG:                         /usr/lib/aarch64-linux-gnu/libpng.so (ver 1.6.34)
    TIFF:                        /usr/lib/aarch64-linux-gnu/libtiff.so (ver 42 / 4.0.9)
    JPEG 2000:                   build Jasper (ver 1.900.1)
    OpenEXR:                     build (ver 2.3.0)
    HDR:                         YES
    SUNRASTER:                   YES
    PXM:                         YES
    PFM:                         YES

  Video I/O:
    DC1394:                      YES (2.2.5)
    FFMPEG:                      YES
      avcodec:                   YES (57.107.100)
      avformat:                  YES (57.83.100)
      avutil:                    YES (55.78.100)
      swscale:                   YES (4.8.100)
      avresample:                YES (3.7.0)
    GStreamer:                   YES (1.14.5)
    v4l/v4l2:                    YES (linux/videodev2.h)

  Parallel framework:            pthreads

  Trace:                         YES (with Intel ITT)

  Other third-party libraries:
    Lapack:                      YES (/usr/lib/aarch64-linux-gnu/liblapack.so /usr/lib/aarch64-linux-gnu/libcblas.so /usr/lib/aarch64-linux-gnu/libatlas.so)
    Eigen:                       YES (ver 3.3.4)
    Custom HAL:                  YES (carotene (ver 0.0.1))
    Protobuf:                    build (3.5.1)

  NVIDIA CUDA:                   YES (ver 10.2, CUFFT CUBLAS FAST_MATH)
    NVIDIA GPU arch:             53 62 72
    NVIDIA PTX archs:

  cuDNN:                         YES (ver 8.0)

  OpenCL:                        YES (no extra features)
    Include path:                /tmp/build_opencv/opencv/3rdparty/include/opencl/1.2
    Link libraries:              Dynamic load

  Python 2:
    Interpreter:                 /usr/bin/python2.7 (ver 2.7.17)
    Libraries:                   /usr/lib/aarch64-linux-gnu/libpython2.7.so (ver 2.7.17)
    numpy:                       /usr/lib/python2.7/dist-packages/numpy/core/include (ver 1.13.3)
    install path:                lib/python2.7/dist-packages/cv2/python-2.7

  Python 3:
    Interpreter:                 /usr/bin/python3 (ver 3.6.9)
    Libraries:                   /usr/lib/aarch64-linux-gnu/libpython3.6m.so (ver 3.6.9)
    numpy:                       /usr/lib/python3/dist-packages/numpy/core/include (ver 1.13.3)
    install path:                lib/python3.6/dist-packages/cv2/python-3.6

  Python (for build):            /usr/bin/python2.7

  Java:                          
    ant:                         NO
    JNI:                         NO
    Java wrappers:               NO
    Java tests:                  NO

  Install to:                    /usr/local
-----------------------------------------------------------------

To verify import (and CUDA) works, i did:

[user@hostname] -- [~/Projects/nano_build_opencv] 
 $ python3
Python 3.6.9 (default, Apr 18 2020, 01:56:04) 
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv2
>>> cv2.cuda.printCudaDeviceInfo(0)
*** CUDA Device Query (Runtime API) version (CUDART static linking) *** 

Device count: 1

Device 0: "Xavier"
  CUDA Driver Version / Runtime Version          10.20 / 10.20
  CUDA Capability Major/Minor version number:    7.2
  Total amount of global memory:                 7764 MBytes (8140648448 bytes)
  GPU Clock Speed:                               1.11 GHz
  Max Texture Dimension Size (x,y,z)             1D=(131072), 2D=(131072,65536), 3D=(16384,16384,16384)
  Max Layered Texture Size (dim) x layers        1D=(32768) x 2048, 2D=(32768,32768) x 2048
  Total amount of constant memory:               65536 bytes
  Total amount of shared memory per block:       49152 bytes
  Total number of registers available per block: 65536
  Warp size:                                     32
  Maximum number of threads per block:           1024
  Maximum sizes of each dimension of a block:    1024 x 1024 x 64
  Maximum sizes of each dimension of a grid:     2147483647 x 65535 x 65535
  Maximum memory pitch:                          2147483647 bytes
  Texture alignment:                             512 bytes
  Concurrent copy and execution:                 Yes with 1 copy engine(s)
  Run time limit on kernels:                     No
  Integrated GPU sharing Host Memory:            Yes
  Support host page-locked memory mapping:       Yes
  Concurrent kernel execution:                   Yes
  Alignment requirement for Surfaces:            Yes
  Device has ECC support enabled:                No
  Device is using TCC driver mode:               No
  Device supports Unified Addressing (UVA):      Yes
  Device PCI Bus ID / PCI location ID:           0 / 0
  Compute Mode:
      Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) 

deviceQuery, CUDA Driver = CUDART, CUDA Driver Version  = 10.20, CUDA Runtime Version = 10.20, NumDevs = 1

It looks like the version you have build is not 4.4.0, rather 4.4.0-dev. Did you build from master? Since I first suggested that, 4.4.0 has been tagged for release, so the above command should be used. I will change the default version so only build_opencv.sh is required.

For your issue, I would suggest that if OpenCV is important to you, it might be easier to backup, reflash, fully update, and then start the script again. Alternatively you can retry with an umodified script and build_opencv.sh 4.4.0 like above. Let me know if either of those solutions work for you @ben-mcdonnell-hacksmith or if you continue to have issues with the build.

@ben-mcdonnell-hacksmith
Copy link
Author

Okay, thanks! I figured the issue is with my system having an old version linked somewhere. I will try the 4.4 build tomorrow. maybe try to find the old version and unlink it. If all else fails I will rebuild the system.

Thanks, I will report back as soon I have figured it out.

@ben-mcdonnell-hacksmith
Copy link
Author

So I removed the old OpenCV entirely, I installed using your script and cv 4.4. Still have the same import error, but when I run help('modules') in python3 it seg faults, does that have anything to do with this?

@mdegans
Copy link
Owner

mdegans commented Jul 21, 2020

So I removed the old OpenCV entirely, I installed using your script and cv 4.4. Still have the same import error, but when I run help('modules') in python3 it seg faults, does that have anything to do with this?

@ben-mcdonnell-hacksmith

I suspect there may be another version of OpenCV installed in python path other than the one under the /usr/local prefix. Did you run pip3 install opencv-python at some point? That would put one under ~/.local/

Can you open a python3 interpreter and provide the results of this:

>>> import os
>>> import sys
>>> import itertools
>>> for p in sys.path:
...     for root, dirs, files in os.walk(p):
...             for name in itertools.chain(dirs, files):
...                     if name.startswith('cv2'):
...                             print(os.path.join(root, name))
... 
/usr/lib/python3/dist-packages/cv2.cpython-38-x86_64-linux-gnu.so

(you should see something slightly different)

If you see a file/dir in /home/your_user/.local/... try deleting it (or that python path in it's entirety). If one is in /usr you can remove it with sudo apt remove python3-opencv. The only one that should exist is under /usr/local as shown by opencv_version --verbose

If that still doesn't work, the only thing I can suggest is to backup, reflash, update, and retry build_opencv.sh. Note: the scrip's default version is now 4.4.0 so there's no need to add anything.

@ben-mcdonnell-hacksmith
Copy link
Author

This is what the output of that looks like
`

import os
import sys
import itertools
for p in sys.path:
... for root, dirs, files in os.walk(p):
... for name in itertools.chain(dirs, files):
... if name.startswith('cv2'):
... print(os.path.join(root, name))
...
/usr/local/lib/python3.6/dist-packages/cv2
/usr/local/lib/python3.6/dist-packages/cv2/python-3.6/cv2.cpython-36m-aarch64-linux-gnu.so
`

Does that look right?

@mdegans
Copy link
Owner

mdegans commented Jul 22, 2020

That looks right, @ben-mcdonnell-hacksmith

Let's try something new.

sudo mv /usr/local/lib/python3.6/dist-packages/cv2 /tmp/cv2.bak
python3 -c "help('modules')"
sudo mv /tmp/cv2.bak /usr/local/lib/python3.6/dist-packages/cv2

If help('modules') works after removing the cv2 built by the script, that's definately the issue, and not some other module that cv2 might import (I don't know if it does or not, TBH). If it still crashes, then the problem is somthing else.

Also, Now that you've rebuilt 4.4.0, can you provide the results of opencv_version --verbose again? I'd like to compare against my own build config.

@ben-mcdonnell-hacksmith
Copy link
Author

That sadly didn't fix it. So something else is the issue. Apparently it is somewhat common to happen with python, so I am not too worried.

But the output is:

General configuration for OpenCV 4.4.0 =====================================
Version control: 4.4.0

Extra modules:
Location (extra): /tmp/build_opencv/opencv_contrib/modules
Version control (extra): 4.4.0

Platform:
Timestamp: 2020-07-21T17:09:21Z
Host: Linux 4.9.140-tegra aarch64
CMake: 3.17.3
CMake generator: Unix Makefiles
CMake build tool: /usr/bin/make
Configuration: RELEASE

CPU/HW features:
Baseline: NEON FP16
required: NEON
disabled: VFPV3

C/C++:
Built as dynamic libs?: YES
C++ standard: 11
C++ Compiler: /usr/bin/c++ (ver 7.5.0)
C++ flags (Release): -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Winit-self -Wsuggest-override -Wno-delete-non-virtual-dtor -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections -fvisibility=hidden -fvisibility-inlines-hidden -O3 -DNDEBUG -DNDEBUG
C++ flags (Debug): -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Winit-self -Wsuggest-override -Wno-delete-non-virtual-dtor -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections -fvisibility=hidden -fvisibility-inlines-hidden -g -O0 -DDEBUG -D_DEBUG
C Compiler: /usr/bin/cc
C flags (Release): -fsigned-char -W -Wall -Werror=return-type -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wshadow -Wuninitialized -Winit-self -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections -fvisibility=hidden -O3 -DNDEBUG -DNDEBUG
C flags (Debug): -fsigned-char -W -Wall -Werror=return-type -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wshadow -Wuninitialized -Winit-self -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections -fvisibility=hidden -g -O0 -DDEBUG -D_DEBUG
Linker flags (Release): -Wl,--gc-sections -Wl,--as-needed
Linker flags (Debug): -Wl,--gc-sections -Wl,--as-needed
ccache: NO
Precompiled headers: NO
Extra dependencies: m pthread cudart_static dl rt nppc nppial nppicc nppicom nppidei nppif nppig nppim nppist nppisu nppitc npps cublas cudnn cufft -L/usr/local/cuda/lib64 -L/usr/lib/aarch64-linux-gnu
3rdparty dependencies:

OpenCV modules:
To be built: alphamat aruco bgsegm bioinspired calib3d ccalib core cudaarithm cudabgsegm cudacodec cudafeatures2d cudafilters cudaimgproc cudalegacy cudaobjdetect cudaoptflow cudastereo cudawarping cudev datasets dnn dnn_objdetect dnn_superres dpm face features2d flann freetype fuzzy gapi hfs highgui img_hash imgcodecs imgproc intensity_transform line_descriptor ml objdetect optflow phase_unwrapping photo plot python2 python3 quality rapid reg rgbd saliency shape stereo stitching structured_light superres surface_matching text tracking video videoio videostab xfeatures2d ximgproc xobjdetect xphoto
Disabled: world
Disabled by dependency: -
Unavailable: cnn_3dobj cvv hdf java js julia matlab ovis sfm ts viz
Applications: apps
Documentation: NO
Non-free algorithms: YES

GUI:
GTK+: YES (ver 3.22.30)
GThread : YES (ver 2.56.4)
GtkGlExt: NO
OpenGL support: NO
VTK support: NO

Media I/O:
ZLib: /usr/lib/aarch64-linux-gnu/libz.so (ver 1.2.11)
JPEG: /usr/lib/aarch64-linux-gnu/libjpeg.so (ver 80)
WEBP: build (ver encoder: 0x020f)
PNG: /usr/lib/aarch64-linux-gnu/libpng.so (ver 1.6.34)
TIFF: /usr/lib/aarch64-linux-gnu/libtiff.so (ver 42 / 4.0.9)
JPEG 2000: build Jasper (ver 1.900.1)
OpenEXR: build (ver 2.3.0)
HDR: YES
SUNRASTER: YES
PXM: YES
PFM: YES

Video I/O:
DC1394: YES (2.2.5)
FFMPEG: YES
avcodec: YES (57.107.100)
avformat: YES (57.83.100)
avutil: YES (55.78.100)
swscale: YES (4.8.100)
avresample: YES (3.7.0)
GStreamer: YES (1.14.5)
v4l/v4l2: YES (linux/videodev2.h)

Parallel framework: pthreads

Trace: YES (with Intel ITT)

Other third-party libraries:
Lapack: YES (/usr/lib/aarch64-linux-gnu/liblapack.so /usr/lib/aarch64-linux-gnu/libcblas.so /usr/lib/aarch64-linux-gnu/libatlas.so)
Eigen: YES (ver 3.3.4)
Custom HAL: YES (carotene (ver 0.0.1))
Protobuf: build (3.5.1)

NVIDIA CUDA: YES (ver 10.2, CUFFT CUBLAS FAST_MATH)
NVIDIA GPU arch: 53 62 72
NVIDIA PTX archs:

cuDNN: YES (ver 8.0)

OpenCL: YES (no extra features)
Include path: /tmp/build_opencv/opencv/3rdparty/include/opencl/1.2
Link libraries: Dynamic load

Python 2:
Interpreter: /usr/bin/python2.7 (ver 2.7.17)
Libraries: /usr/lib/aarch64-linux-gnu/libpython2.7.so (ver 2.7.17)
numpy: /usr/lib/python2.7/dist-packages/numpy/core/include (ver 1.13.3)
install path: lib/python2.7/dist-packages/cv2/python-2.7

Python 3:
Interpreter: /usr/bin/python3 (ver 3.6.9)
Libraries: /usr/lib/aarch64-linux-gnu/libpython3.6m.so (ver 3.6.9)
numpy: /usr/lib/python3/dist-packages/numpy/core/include (ver 1.13.3)
install path: lib/python3.6/dist-packages/cv2/python-3.6

Python (for build): /usr/bin/python2.7

Java:
ant: NO
JNI: NO
Java wrappers: NO
Java tests: NO

Install to: /usr/local

It still doesn't work...

@mdegans
Copy link
Owner

mdegans commented Jul 22, 2020

That sadly didn't fix it. So something else is the issue.

@ben-mcdonnell-hacksmith If you moved cv2 out of the python path and it still crashes when you do help('modules'), then I think the problem is unrelated to cv2, rather something else that's being imported (numpy maybe?). We're narrowing it down at least, so progress :)

Can you cause a crash with import numpy or import scipy?

If one of those fails, then that would be the root cause. If that's the case, can you the same code as above, just for numpy or scipy instead of cv2 like:

>>> for p in sys.path:
...     for root, dirs, files in os.walk(p):
...             for name in itertools.chain(dirs, files):
...                     if name.startswith('numpy'):
...                             print(os.path.join(root, name))
... 
/usr/lib/python3/dist-packages/numpy-1.17.4.egg-info
/usr/lib/python3/dist-packages/numpy
/usr/lib/python3/dist-packages/numpy/distutils/numpy_distribution.py
/usr/lib/python3/dist-packages/numpy/distutils/__pycache__/numpy_distribution.cpython-38.pyc
/usr/lib/python3/dist-packages/numpy/core/include/numpy
/usr/lib/python3/dist-packages/numpy/core/include/numpy/numpyconfig.h

(that's not from Tegra, but your results should be similar).

@ben-mcdonnell-hacksmith
Copy link
Author

None of those throw errors. The cv error is probably related to an old version somewhere, but I can't find anything anywhere. I may try making a new image, but I do have a variety of other stuff installed and it would take a while to rebuild. And if this happens again I would like to figure out how to fix it.

@mdegans
Copy link
Owner

mdegans commented Jul 22, 2020

None of those throw errors. The cv error is probably related to an old version somewhere, but I can't find anything anywhere. I may try making a new image, but I do have a variety of other stuff installed and it would take a while to rebuild. And if this happens again I would like to figure out how to fix it.

Oh, I'm pretty sure it can be fixed by reflashing, updating, and running the script again, but like you, I'd like to know what's causing this.

Can you try this, only substituting gdb gui-player-vala with gdb --args python3 -c "import cv2"

 $ sudo apt install gdb
 $ gdb gui-player-vala 
GNU gdb (Ubuntu 9.1-0ubuntu1) 9.1
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from gui-player-vala...
(gdb) run
Starting program: /home/mdegans/Projects/gst-webinar-beginner/builddir/vala/gui-player-vala 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

(gui-player-vala:65278): Gtk-CRITICAL **: 14:22:31.062: _gtk_style_provider_private_get_settings: assertion 'GTK_IS_STYLE_PROVIDER_PRIVATE (provider)' failed

(gui-player-vala:65278): Gtk-CRITICAL **: 14:22:31.062: _gtk_style_provider_private_get_settings: assertion 'GTK_IS_STYLE_PROVIDER_PRIVATE (provider)' failed

(gui-player-vala:65278): Gtk-CRITICAL **: 14:22:31.062: _gtk_style_provider_private_get_settings: assertion 'GTK_IS_STYLE_PROVIDER_PRIVATE (provider)' failed

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff764b364 in ?? () from /usr/lib/x86_64-linux-gnu/libgtk-3.so.0
(gdb) bt 20
#0  0x00007ffff764b364 in  () at /usr/lib/x86_64-linux-gnu/libgtk-3.so.0
#1  0x00007ffff74f1e0c in  () at /usr/lib/x86_64-linux-gnu/libgtk-3.so.0
#2  0x00007ffff7512307 in  () at /usr/lib/x86_64-linux-gnu/libgtk-3.so.0
#3  0x00007ffff74fd492 in  () at /usr/lib/x86_64-linux-gnu/libgtk-3.so.0
#4  0x00007ffff7512232 in  () at /usr/lib/x86_64-linux-gnu/libgtk-3.so.0
#5  0x00007ffff751228d in  () at /usr/lib/x86_64-linux-gnu/libgtk-3.so.0
#6  0x00007ffff74fdec6 in  () at /usr/lib/x86_64-linux-gnu/libgtk-3.so.0
#7  0x00007ffff7e35131 in g_type_create_instance () at /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0
#8  0x00007ffff7e1434d in  () at /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0
#9  0x00007ffff7e15b45 in g_object_new_with_properties () at /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0
#10 0x00007ffff7e166f1 in g_object_new () at /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0
#11 0x00007ffff751ac1f in  () at /usr/lib/x86_64-linux-gnu/libgtk-3.so.0
#12 0x00007ffff76f9d8d in  () at /usr/lib/x86_64-linux-gnu/libgtk-3.so.0
#13 0x00007ffff7e35131 in g_type_create_instance () at /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0
#14 0x00007ffff7e1434d in  () at /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0
#15 0x00007ffff7e16378 in g_object_new_valist () at /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0
#16 0x00007ffff7e166cd in g_object_new () at /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0
#17 0x0000555555556928 in gui_player_constructPython Exception <class 'gdb.error'> No type named TypeNode.: 
 (object_type=, uri=0x555555558149 "") at ../vala/05/gui-player.vala:73
#18 0x00005555555572f6 in gui_player_new (uri=0x555555558149 "") at ../vala/05/gui-player.vala:59
#19 0x0000555555557c55 in _vala_main (argv=0x7fffffffdc18, argv_length1=1) at ../vala/05/gui-player.vala:176
(More stack frames follow...)
(gdb) 

@ben-mcdonnell-hacksmith
Copy link
Author

I have no idea how to interpret the output of this, but I think it could have something to do with the shared list library issue.

$ gdb --args python3 -c "import cv2"
GNU gdb (Ubuntu 8.1-0ubuntu3.2) 8.1.0.20180409-git
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "aarch64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from python3...(no debugging symbols found)...done.
(gdb) 
(gdb) run
Starting program: /usr/bin/python3 -c import\ cv2
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
[New Thread 0x7fb6d541f0 (LWP 18223)]
[New Thread 0x7fb55531f0 (LWP 18224)]
[New Thread 0x7fb3d521f0 (LWP 18225)]
[New Thread 0x7fb25511f0 (LWP 18226)]
[New Thread 0x7fb0d501f0 (LWP 18227)]
warning: Corrupted shared library list: 0xc3b390 != 0xd17d40
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python3.6/dist-packages/cv2/__init__.py", line 96, in <module>
    bootstrap()
  File "/usr/local/lib/python3.6/dist-packages/cv2/__init__.py", line 86, in bootstrap
    import cv2
ImportError: /usr/local/lib/libopencv_dnn.so.4.4: undefined symbol: cudnnGetConvolutionBackwardDataAlgorithm
[Thread 0x7fb6d541f0 (LWP 18223) exited]
[Thread 0x7fb55531f0 (LWP 18224) exited]
[Thread 0x7fb25511f0 (LWP 18226) exited]
[Thread 0x7fb0d501f0 (LWP 18227) exited]
[Thread 0x7fb3d521f0 (LWP 18225) exited]
[Inferior 1 (process 18220) exited with code 01]
(gdb) bt 20
No stack.
(gdb) 

@mdegans
Copy link
Owner

mdegans commented Jul 22, 2020

Nevermind. I was thinking of another issue on a separate project with a core dump. Can you try a sudo ldconfig and see if you can import cv2 after that?. If that doesn't fix it, can you provide the output of:

ldd /usr/local/lib/python3.6/dist-packages/cv2/python-3.6/cv2.cpython-36m-aarch64-linux-gnu.so | grep cudnn

That should give you something like:

libcudnn.so.8 => /usr/lib/aarch64-linux-gnu/libcudnn.so.8 (0x0000007f5cc83000)

@ben-mcdonnell-hacksmith
Copy link
Author

I get an output from the ldconfig that appears to have some errors, but it doesn't fix the import issue:

/sbin/ldconfig.real: /usr/local/cuda-10.2/targets/aarch64-linux/lib/libcudnn_adv_infer.so.8 is not a symbolic link

/sbin/ldconfig.real: /usr/local/cuda-10.2/targets/aarch64-linux/lib/libcudnn.so.8 is not a symbolic link

/sbin/ldconfig.real: /usr/local/cuda-10.2/targets/aarch64-linux/lib/libcudnn_cnn_infer.so.8 is not a symbolic link

/sbin/ldconfig.real: /usr/local/cuda-10.2/targets/aarch64-linux/lib/libcudnn_ops_infer.so.8 is not a symbolic link

/sbin/ldconfig.real: /usr/local/cuda-10.2/targets/aarch64-linux/lib/libcudnn_adv_train.so.8 is not a symbolic link

/sbin/ldconfig.real: /usr/local/cuda-10.2/targets/aarch64-linux/lib/libcudnn.so.7 is not a symbolic link

/sbin/ldconfig.real: /usr/local/cuda-10.2/targets/aarch64-linux/lib/libcudnn_cnn_train.so.8 is not a symbolic link

/sbin/ldconfig.real: /usr/local/cuda-10.2/targets/aarch64-linux/lib/libcudnn_ops_train.so.8 is not a symbolic link

From the other command I get the output: libcudnn.so.8 => /usr/lib/aarch64-linux-gnu/libcudnn.so.8 (0x0000007f88b43000)

@mdegans
Copy link
Owner

mdegans commented Jul 23, 2020

I get an output from the ldconfig that appears to have some errors, but it doesn't fix the import issue:

/sbin/ldconfig.real: /usr/local/cuda-10.2/targets/aarch64-linux/lib/libcudnn_adv_infer.so.8 is not a symbolic link

/sbin/ldconfig.real: /usr/local/cuda-10.2/targets/aarch64-linux/lib/libcudnn.so.8 is not a symbolic link

/sbin/ldconfig.real: /usr/local/cuda-10.2/targets/aarch64-linux/lib/libcudnn_cnn_infer.so.8 is not a symbolic link

/sbin/ldconfig.real: /usr/local/cuda-10.2/targets/aarch64-linux/lib/libcudnn_ops_infer.so.8 is not a symbolic link

/sbin/ldconfig.real: /usr/local/cuda-10.2/targets/aarch64-linux/lib/libcudnn_adv_train.so.8 is not a symbolic link

/sbin/ldconfig.real: /usr/local/cuda-10.2/targets/aarch64-linux/lib/libcudnn.so.7 is not a symbolic link

/sbin/ldconfig.real: /usr/local/cuda-10.2/targets/aarch64-linux/lib/libcudnn_cnn_train.so.8 is not a symbolic link

/sbin/ldconfig.real: /usr/local/cuda-10.2/targets/aarch64-linux/lib/libcudnn_ops_train.so.8 is not a symbolic link
From the other command I get the output: libcudnn.so.8 => /usr/lib/aarch64-linux-gnu/libcudnn.so.8 (0x0000007f88b43000)

Looks like there is somethign wrong with your cudnn install maybe?

Try:

sudo apt install nvidia-cudnn8 libcudnn8-dev --reinstall

That shuould perform a full reinstall of cudnn8. Cv2 may just start working after that. I'd imagine if you are getting the above errors, other things that use cuDNN will also crash in the same way.

@ben-mcdonnell-hacksmith
Copy link
Author

So that didn't fix it, but the error still appears when running ldconfig, is there a way to manually uninstall cudnn and reinstall it?

@mdegans
Copy link
Owner

mdegans commented Jul 26, 2020

So that didn't fix it, but the error still appears when running ldconfig, is there a way to manually uninstall cudnn and reinstall it?

@ben-mcdonnell-hacksmith

That would be the manual way :(

At this point, I'm really not sure what's wrong. Although i'm pretty sure a reflash will fix it, as you say, I can't guarantee it won't happen again. One way to find out is maybe to build OpenCV first and keep applying your other changes, testing after each one to see what breaks cv2. If you can figure out what did it, that would be very helpful. I'll leave this issue open.

@ben-mcdonnell-hacksmith
Copy link
Author

@mdegans I made a new SD Card with a fresh install of JetPack, It did fix it as we figured. I still have the old card laying around in case I want to revisit it. But thanks for all your help!

@mdegans
Copy link
Owner

mdegans commented Jul 28, 2020

YW, @ben-mcdonnell-hacksmith

Sorry we couldn't figure it out. Hopefully (?) somebody else has this issue and reopens this. I don't much like these kinds of mysteries :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants