Test failed with Jetpack 4.4 [L4T 32.4.3] and OpenCV 4.4.0

[nalzok]

I'm building OpenCV 4.4.0 headlessly on a Jetson Nano with Jetpack 4.4 [L4T 32.4.3]. The build is successful but many tests failed. See attached logs.

build.log
test.log
install.log
test-reports.zip

Apparently the cause is multiple missing input files like ./gpu/opticalflow/frame0.png. Any tips?

[mdegans]

Thanks for reporting this @nalzok . The cause is that the data needed by the tests isn't downloaded when it should be if the test option is used. Many tests will still fail, but it's more like 80% passing, rather than 20% without the test data. I will implement a fix as soon as I get a chance.

More detail here:
https://docs.opencv.org/3.4/d6/d15/tutorial_building_tegra_cuda.html

see "OpenCV Extra"

If you clone that and set the approprate environment variable, it should find the test data.

[nalzok]

Thanks for the timely response!

I tried to fix the issue myself by modifying your script to this. I commented out the "prepare for the build" part and manually clone the opencv, opencv_contrib and opencv_extra repositories under the directory /home/qingyao/opencv instead, so that when something goes wrong I don't have to re-clone them again. The build script is also placed in /home/qingyao/opencv.

I have changed CMAKEFLAGS to enable more features, which may or may not have something to do with the error I got later. (Yeah I know that will cause more trouble for you to reproduce the issue. Sorry about that ):

The additional dependencies I installed (mostly for -D BUILD_DOCS=ON) are

sudo apt install doxygen graphviz openblas python-pip
pip install beautifulsoup4

I mounted a 4GB swap file and start the building process with ./build_opencv.sh foo test (the first argument is ignore now). It took ~12 hours to finish on my Jetson Nano. This time, tests took longer to run, but many eventually still failed.

• build.log
• test.log
• install.log

You may download the full test report here. It's a 134.1 MB zip file including some image files.

There are a lot of different errors, but the "cuDNN did not return a suitable algorithm for convolution" error gets the most exposure. Here is an example of what it looks like

test-reports/accuracy/opencv_test_dnn.xml
8174:      <failure message="unknown file&#x0A;C++ exception with description &quot;OpenCV(4.4.0) /home/qingyao/opencv/opencv/modules/dnn/src/layers/../cuda4dnn/primitives/../csl/cudnn/convolution.hpp:303: error: (-217:Gpu API call) cuDNN did not return a suitable algorithm for convolution. in function &apos;ConvolutionAlgorithm&apos;&#x0A;&quot; thrown in the test body." type=""><![CDATA[unknown file
8440:      <failure message="unknown file&#x0A;C++ exception with description &quot;OpenCV(4.4.0) /home/qingyao/opencv/opencv/modules/dnn/src/layers/../cuda4dnn/primitives/../csl/cudnn/convolution.hpp:303: error: (-217:Gpu API call) cuDNN did not return a suitable algorithm for convolution. in function &apos;ConvolutionAlgorithm&apos;&#x0A;&quot; thrown in the test body." type=""><![CDATA[unknown file
8677:      <failure message="/home/qingyao/opencv/opencv/modules/dnn/test/test_common.impl.hpp:70&#x0A;Expected: (normL1) &lt;= (l1), actual: nan vs 1e-05" type=""><![CDATA[/home/qingyao/opencv/opencv/modules/dnn/test/test_common.impl.hpp:70
8684:      <failure message="/home/qingyao/opencv/opencv/modules/dnn/test/test_common.impl.hpp:70&#x0A;Expected: (normL1) &lt;= (l1), actual: nan vs 0.004" type=""><![CDATA[/home/qingyao/opencv/opencv/modules/dnn/test/test_common.impl.hpp:70
9230:      <failure message="unknown file&#x0A;C++ exception with description &quot;OpenCV(4.4.0) /home/qingyao/opencv/opencv/modules/dnn/src/layers/../cuda4dnn/primitives/../csl/cudnn/convolution.hpp:303: error: (-217:Gpu API call) cuDNN did not return a suitable algorithm for convolution. in function &apos;ConvolutionAlgorithm&apos;&#x0A;&quot; thrown in the test body." type=""><![CDATA[unknown file
9335:      <failure message="unknown file&#x0A;C++ exception with description &quot;OpenCV(4.4.0) /home/qingyao/opencv/opencv/modules/dnn/src/layers/../cuda4dnn/primitives/../csl/cudnn/convolution.hpp:303: error: (-217:Gpu API call) cuDNN did not return a suitable algorithm for convolution. in function &apos;ConvolutionAlgorithm&apos;&#x0A;&quot; thrown in the test body." type=""><![CDATA[unknown file

---

However, the OpenCV installation appears good

qingyao@jetson:~/opencv/opencv/build$ opencv_version --verbose

General configuration for OpenCV 4.4.0 =====================================
  Version control:               4.4.0

  Extra modules:
    Location (extra):            /home/qingyao/opencv/opencv_contrib/modules
    Version control (extra):     4.4.0

  Platform:
    Timestamp:                   2020-09-13T04:59:57Z
    Host:                        Linux 4.9.140-tegra aarch64
    CMake:                       3.10.2
    CMake generator:             Unix Makefiles
    CMake build tool:            /usr/bin/make
    Configuration:               RELEASE

  CPU/HW features:
    Baseline:                    NEON FP16
      required:                  NEON
      disabled:                  VFPV3

  C/C++:
    Built as dynamic libs?:      YES
    C++ standard:                11
    C++ Compiler:                /usr/bin/c++  (ver 7.5.0)
    C++ flags (Release):         -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Winit-self -Wsuggest-override -Wno-delete-non-virtual-dtor -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections    -fvisibility=hidden -fvisibility-inlines-hidden -fopenmp -O3 -DNDEBUG  -DNDEBUG
    C++ flags (Debug):           -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Winit-self -Wsuggest-override -Wno-delete-non-virtual-dtor -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections    -fvisibility=hidden -fvisibility-inlines-hidden -fopenmp -g  -O0 -DDEBUG -D_DEBUG
    C Compiler:                  /usr/bin/cc
    C flags (Release):           -fsigned-char -W -Wall -Werror=return-type -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wshadow -Wuninitialized -Winit-self -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections    -fvisibility=hidden -fopenmp -O3 -DNDEBUG  -DNDEBUG
    C flags (Debug):             -fsigned-char -W -Wall -Werror=return-type -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wshadow -Wuninitialized -Winit-self -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections    -fvisibility=hidden -fopenmp -g  -O0 -DDEBUG -D_DEBUG
    Linker flags (Release):      -Wl,--gc-sections -Wl,--as-needed
    Linker flags (Debug):        -Wl,--gc-sections -Wl,--as-needed
    ccache:                      NO
    Precompiled headers:         NO
    Extra dependencies:          m pthread cudart_static -lpthread dl rt nppc nppial nppicc nppicom nppidei nppif nppig nppim nppist nppisu nppitc npps cublas cudnn cufft -L/usr/local/cuda/lib64 -L/usr/lib/aarch64-linux-gnu
    3rdparty dependencies:

  OpenCV modules:
    To be built:                 alphamat aruco bgsegm bioinspired calib3d ccalib core cudaarithm cudabgsegm cudacodec cudafeatures2d cudafilters cudaimgproc cudalegacy cudaobjdetect cudaoptflow cudastereo cudawarping cudev datasets dnn dnn_objdetect dnn_superres dpm face features2d flann freetype fuzzy gapi hfs highgui img_hash imgcodecs imgproc intensity_transform line_descriptor ml objdetect optflow phase_unwrapping photo plot python2 python3 quality rapid reg rgbd saliency shape stereo stitching structured_light superres surface_matching text tracking ts video videoio videostab xfeatures2d ximgproc xobjdetect xphoto
    Disabled:                    world
    Disabled by dependency:      -
    Unavailable:                 cnn_3dobj cvv hdf java js julia matlab ovis sfm viz
    Applications:                tests perf_tests examples apps
    Documentation:               doxygen python
    Non-free algorithms:         YES

  GUI:
    GTK+:                        YES (ver 3.22.30)
      GThread :                  YES (ver 2.56.4)
      GtkGlExt:                  NO
    OpenGL support:              NO
    VTK support:                 NO

  Media I/O:
    ZLib:                        /usr/lib/aarch64-linux-gnu/libz.so (ver 1.2.11)
    JPEG:                        /usr/lib/aarch64-linux-gnu/libjpeg.so (ver 80)
    WEBP:                        build (ver encoder: 0x020f)
    PNG:                         /usr/lib/aarch64-linux-gnu/libpng.so (ver 1.6.34)
    TIFF:                        /usr/lib/aarch64-linux-gnu/libtiff.so (ver 42 / 4.0.9)
    JPEG 2000:                   build Jasper (ver 1.900.1)
    OpenEXR:                     build (ver 2.3.0)
    HDR:                         YES
    SUNRASTER:                   YES
    PXM:                         YES
    PFM:                         YES

  Video I/O:
    DC1394:                      YES (2.2.5)
    FFMPEG:                      YES
      avcodec:                   YES (57.107.100)
      avformat:                  YES (57.83.100)
      avutil:                    YES (55.78.100)
      swscale:                   YES (4.8.100)
      avresample:                YES (3.7.0)
    GStreamer:                   YES (1.14.5)
    v4l/v4l2:                    YES (linux/videodev2.h)

  Parallel framework:            OpenMP

  Trace:                         YES (with Intel ITT)

  Other third-party libraries:
    Lapack:                      YES (/opt/OpenBLAS/lib/libopenblas.so)
    Eigen:                       YES (ver 3.3.4)
    Custom HAL:                  YES (carotene (ver 0.0.1))
    Protobuf:                    build (3.5.1)

  NVIDIA CUDA:                   YES (ver 10.2, CUFFT CUBLAS FAST_MATH)
    NVIDIA GPU arch:             53 62 72
    NVIDIA PTX archs:

  cuDNN:                         YES (ver 8.0)

  OpenCL:                        YES (no extra features)
    Include path:                /home/qingyao/opencv/opencv/3rdparty/include/opencl/1.2
    Link libraries:              Dynamic load

  Python 2:
    Interpreter:                 /usr/bin/python2.7 (ver 2.7.17)
    Libraries:                   /usr/lib/aarch64-linux-gnu/libpython2.7.so (ver 2.7.17)
    numpy:                       /usr/lib/python2.7/dist-packages/numpy/core/include (ver 1.13.3)
    install path:                lib/python2.7/dist-packages/cv2/python-2.7

  Python 3:
    Interpreter:                 /usr/bin/python3 (ver 3.6.9)
    Libraries:                   /usr/lib/aarch64-linux-gnu/libpython3.6m.so (ver 3.6.9)
    numpy:                       /home/qingyao/.local/lib/python3.6/site-packages/numpy/core/include (ver 1.19.1)
    install path:                lib/python3.6/dist-packages/cv2/python-3.6

  Python (for build):            /usr/bin/python2.7

  Java:
    ant:                         NO
    JNI:                         NO
    Java wrappers:               NO
    Java tests:                  NO

  Install to:                    /usr/local
-----------------------------------------------------------------

The Python binding looks good as well

qingyao@jetson:~/opencv/opencv/build$ python3
Python 3.6.9 (default, Jul 17 2020, 12:50:27)
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv2
>>> cv2.__version__
'4.4.0'
>>> cv2.cuda.printCudaDeviceInfo(0)
*** CUDA Device Query (Runtime API) version (CUDART static linking) ***

Device count: 1

Device 0: "NVIDIA Tegra X1"
  CUDA Driver Version / Runtime Version          10.20 / 10.20
  CUDA Capability Major/Minor version number:    5.3
  Total amount of global memory:                 3964 MBytes (4156780544 bytes)
  GPU Clock Speed:                               0.92 GHz
  Max Texture Dimension Size (x,y,z)             1D=(65536), 2D=(65536,65536), 3D=(4096,4096,4096)
  Max Layered Texture Size (dim) x layers        1D=(16384) x 2048, 2D=(16384,16384) x 2048
  Total amount of constant memory:               65536 bytes
  Total amount of shared memory per block:       49152 bytes
  Total number of registers available per block: 32768
  Warp size:                                     32
  Maximum number of threads per block:           1024
  Maximum sizes of each dimension of a block:    1024 x 1024 x 64
  Maximum sizes of each dimension of a grid:     2147483647 x 65535 x 65535
  Maximum memory pitch:                          2147483647 bytes
  Texture alignment:                             512 bytes
  Concurrent copy and execution:                 Yes with 1 copy engine(s)
  Run time limit on kernels:                     Yes
  Integrated GPU sharing Host Memory:            Yes
  Support host page-locked memory mapping:       Yes
  Concurrent kernel execution:                   Yes
  Alignment requirement for Surfaces:            Yes
  Device has ECC support enabled:                No
  Device is using TCC driver mode:               No
  Device supports Unified Addressing (UVA):      Yes
  Device PCI Bus ID / PCI location ID:           0 / 0
  Compute Mode:
      Default (multiple host threads can use ::cudaSetDevice() with device simultaneously)

deviceQuery, CUDA Driver = CUDART, CUDA Driver Version  = 10.20, CUDA Runtime Version = 10.20, NumDevs = 1

>>> 

[mdegans]

Sorry for the late response @nalzok. Your issue may be that OPENCV_TEST_DATA_PATH (cmake) is not set. If you check the above link, there should be a line like:

    -DINSTALL_TESTS=OFF \
    -DOPENCV_TEST_DATA_PATH=../opencv_extra/testdata \

I will be pushing some updates today to fix this issue, hopefully. Thanks again for reporting it.