OpenCV does not track correctly libOpenNI2.so

[Lecrapouille]

System Information

This issue is architecture agnostic. Anyway:

OpenCV version: 4.6.0 SHA1 3f371fe
Operating System / Platform: Ubuntu 20.04
Compiler & compiler version: x86_64 with g++
Compiler & compiler version: x86_64 with cross-compiler ARM i.MX 8

Detailed description

My issue is in two points. A minor: some issues inside the documentation (out of date). I should have created a dedicated ticket, but this linked with my major issue: OpenCV does not track libOpenNI2.so to OpenCV_LIBS and it should do it.

Concerning the documentation

• the link to Orbbec is broken:
  

  opencv/doc/tutorials/app/orbbec_astra.markdown

  Line 12 in f7bee78

  This tutorial is devoted to the Astra Series of Orbbec 3D cameras (https://orbbec3d.com/product-astra-pro/).

• the link to the SDK 2.3 is broken:
  

  opencv/doc/tutorials/app/orbbec_astra.markdown

  Line 21 in f7bee78

  -# Download the latest version of Orbbec OpenNI SDK (from here <https://orbbec3d.com/develop/>).

This last can be found here https://orbbec3d.com/index/download.html "Orbbec OpenNI SDK for Linux". With a wget command, I downloaded it and compiled OpenCV against it (x86_64 and arm imx8).

Issue 1

• Since version 2.3, they have changed their install.sh. The creation of the OpenNIDevEnvironment file is not made. So the following commands in the tutorial are useless:

opencv/doc/tutorials/app/orbbec_astra.markdown

Line 32 in f7bee78

$ source OpenNIDevEnvironment

opencv/doc/tutorials/app/orbbec_astra.markdown

Line 38 in f7bee78

$ echo $OPENNI2_INCLUDE

• Since version 2.3 folders inside the SDK have changed. The following line shall be updated:

opencv/modules/videoio/cmake/detect_openni2.cmake

Line 18 in f7bee78

PATH_SUFFIXES "Redist" "Lib"

It is "libs" now. If I update detect_openni2.cmake and the set OPENNI2_DIR="$OPENNI_PATH/sdk" to the cmake command for compiling OpenCV, it does not find my local libOpenNI2.so but only the one installed in my operating system (with apt-get install I guess).

• As consequence, to compile OpenCV against OpenNI, I had to do:

cmake ... -DWITH_OPENNI=ON -DWITH_OPENNI2=ON
-DOPENNI2_INCLUDE="$OPENNI_PATH/sdk/Include" 
-DOPENNI2_DIR="$OPENNI_PATH/sdk" 
-DOPENNI2_LIBRARY="$OPENNI_PATH/sdk/libs/libOpenNI2.so"

I see:

--     OpenNI2:                     YES (2.3.0)

The next issue is my main concern for this ticket.

Issue 2

while OPENNI2_LIBRARIES is set in

opencv/modules/videoio/cmake/detect_openni2.cmake

Line 36 in f7bee78

set(OPENNI2_LIBRARIES "${OPENNI2_SYS_LIBRARY}")

When linking my project against OpenCV in my CMakeLists.txt:

message(${OpenCV_LIBS})
target_link_libraries(myapplication ${OpenCV_LIBS})

I have a linkage issue:

warning: libOpenNI2.so, needed by /home/xxx/yyy/build/armimx8/opencv/lib/libopencv_videoio.so.4.6.0, not found (try using -rpath or -rpath-link)
/home/xxxxxx/depend/build/armimx8/opencv/lib/libopencv_videoio.so.4.6.0: undefined reference to `oniDeviceEnableDepthColorSync'
/home/xxxxxx/depend/build/armimx8/opencv/lib/libopencv_videoio.so.4.6.0: undefined reference to `oniStreamGetSensorInfo'
/home/xxxxxx/depend/build/armimx8/opencv/lib/libopencv_videoio.so.4.6.0: undefined reference to `oniFrameRelease'

Because OpenCV_LIBS does not contain libOpenNI2.so. This can be solved if I choose one of the following solution:

• Install libOpenNI2.so on my system (apt-get or in the cross-compiler lib/ folder).
• Do target_link_libraries(myapplication ${OpenCV_LIBS} /xxx/libOpenNI2.so)

The first solution is not possible for me (not admin, CI ...). The second solution works, but my project depends on third part libraries using the OpenCV compiled locally will fail because they also need libOpenNI2.so (and I cannot modify their CMake files). The LD_LIBRARY_PATH does not work (cross compilation, I don't know why).

As general issue: When linking to external third parts libs such OpenNI, OpenCV shall set them in the CMake variable OpenCV_LIBS (same thing for library paths) and shall do not depend on the presence of installed system libraries. Therefore:

message(${OpenCV_LIBS})

shall display ... libopencv_videoio.so ... libOpenNI2.so

Steps to reproduce

• Compile OpenCV against OpenNI2 locally downloaded (with -DWITH_OPENNI ... as explained previously).
• libOpenNI2.so shall not be installed on the operating system (no apt-get install, no cp commands to /usr/lib).
• Compile an application linking against OpenCV (target_link_libraries(myapplication ${OpenCV_LIBS}))
• A linkage issue shall appear (libOpenNI2.so needed by libopencv_videoio.so)
• You do not even need an Orbbec camera.

What desired ?

To include OPENNI2_LIBRARY inside the OpenCV_LIBS variable as well for lib folder paths.

Issue submission checklist

• I report the issue, it's not a question
• I checked the problem with documentation, FAQ, open issues, forum.opencv.org, Stack Overflow, etc and have not found any solution
• I updated to the latest OpenCV version and the issue is still there
• There is reproducer code and related data files (videos, images, onnx, etc)

Update

• I'm on SHA1 3f371fe. On the HEAD of 4.X branch, I reached in fact an assert.
• The tutorial says colorStream.set(CAP_PROP_FRAME_WIDTH, 640) but this function is failing (return false). We cannot change the resolution. The get() function returns correctly the resolution and FPS (et least for with Mini Astra).

[fengyuentau]

Hello @Lecrapouille , thank you for the report!

Documentation issues are confirmed. We will update the links and also the outdated step-by-step tutorial.

As for the linkage problem, I personally think that ${OpenCV_LIBS} should have only libs from OpenCV so that libs do not mess around. Dependencies should be found either from system lib paths or user specified. It is not a bug but more like a usage problem. For example, if your CI is meant to test and build OpenCV with OpenNI2, you should install it system-wise or define a path in your CMake script to link against it if you do not have admin rights.

[Lecrapouille]

@fengyuentau thanks you, good news :)

Concerning dependencies, you can ignore my issue 2: ${OpenCV_LIBS} should have only libs from OpenCV yes, I totally agree with you. My case is not standard :) Generally third part libs are installed on the system and linker search them. LD_LIBRARY_PATH would have helped me in a normal situation, but because I'm cross compiling to ARM from X86_64 the linker does not detect libOpenNI2.so but copying this lib to a location where the cross compiling is looking it fix my issue. I did not try to play with -Wl,-rpath,$(xxx) for my own CMake. The issue concerning the second lib is because their CMake is touchy : compiles a lib but also a demo application, and the linker fails for this last (compiling only the lib would have been fine).

I guess it does not exist ${OpenCV_THIRDPARTS_LIBS} referring them to mimic a pkg-config --libs ? So I guess I will have the same issue for lib ffmpeg :) By the way I have not tested if pkg-config --libs opencv4 returns -lOpenNI2.

[Lecrapouille]

@fengyuentau PS: oh by the way, have you fixed depthStream.set(CAP_PROP_FRAME_WIDTH, 640) (same for height) ? It seems to me we cannot set the depth sensor this way, and for Orbbec Astra Mini, the BGR image generator does not seem to work on my side (this is may normal but I have also issue with my USB bandwidth too limited): the capture.grab() (or capture.retrieved() ?) return false (too many empty frames). Therefore the only possible resolution is 640x480 and cannot be changed.

[fengyuentau]

oh by the way, have you fixed colorStream.set(CAP_PROP_FRAME_WIDTH, 640) (same for height)

Does this issue only happen on Orbbec cameras or other cameras as well with your USB bandwidth limited device?

[kaingwade]

@fengyuentau PS: oh by the way, have you fixed colorStream.set(CAP_PROP_FRAME_WIDTH, 640) (same for height) ? It seems to me we cannot set the depth sensor this way, and for Orbbec Astra Mini, the BGR image generator does not seem to work on my side (this is may normal but I have also issue with my USB bandwidth too limited): the capture.grab() (or capture.retrieved() ?) return false (too many empty frames). Therefore the only possible resolution is 640x480 and cannot be changed.

Do you mean the size of the color image cannot be set or of the depth image cannot?

[Lecrapouille]

@kaingwade @fengyuentau the size of the depth image cannot be set. Sorry, I fixed my typo in my previous message.

In the tutorial:

opencv/samples/cpp/tutorial_code/videoio/orbbec_astra/orbbec_astra.cpp

Lines 59 to 65 in b522148

	//! [Setup streams]
	// Set color and depth stream parameters
	colorStream.set(CAP_PROP_FRAME_WIDTH, 640);
	colorStream.set(CAP_PROP_FRAME_HEIGHT, 480);
	depthStream.set(CAP_PROP_FRAME_WIDTH, 640);
	depthStream.set(CAP_PROP_FRAME_HEIGHT, 480);
	depthStream.set(CAP_PROP_OPENNI2_MIRROR, 0);

(my current code is different from this tutorial)

the colorStream.set() will return true while the depthStream.set() will return false because I guess the orbbec needs to change the 3 settings together (width, height and FPS) like it was probably set with the previous (?!) Openni CAP_OPENNI_VGA_30HZ, CAP_OPENNI_SXGA_15HZ ...

which needs to set first the generator, but in my case, it seems not working with my mini astra:
https://github.com/opencv/opencv/blob/f7bee7883b0320a9b52f505b366b059778b84da4/doc/tutorials/app/orbbec_astra.markdown?plain=1#L100C14-L106

Reading back the spec https://www.reichelt.com/ch/fr/orbbec3d-astra-mini-s-astra-mini-s-p263253.html The Orbbec3D Mini S is a very small 3D camera with a high RGB/image resolution of 640 x 480 pixels and up to 30 fps. The depth images are available in a resolution of 640 x 480 p or 320 x 240 p at 30 fps.

The tutorial could help clarify and by the way it's sad that enum are not enum class this would prevent error but the API reference is not explaining for which functions these params are applied.

[kaingwade]

@Lecrapouille Thank you for the feedback. The tutorial indeed needs to be updated and improved. Btw newer UVC Orbbec cameras are now supported as well, and are much simpler to call: videocapture_obsensor.cpp, videocapture_obsensor.py

[Lecrapouille]

@kaingwade Yes I was aware of that and I suggested them to my team.

[fengyuentau]

the size of the depth image cannot be set.

Do you have another device with bigger USB bandwidth to test whether it is the source of your problem? Or you can provide me your USB version and I will doublecheck with Orbbec devs.

[Lecrapouille]

@fengyuentau no sorry I do not have a bigger device. My USB is 2.0 but also related to #23787 while in this ticket, initially I could not run more than 2 RGB webcam simultaneously, I can only run the Orbbec in RGB at 320x240 and the depth at 640x480. If the RGB is let to the default size the ioctl() stalls and if do open/grab/retrieve/release with the depth sensor I will have 2 or 3 black images upon 4 captures.

[opencv-alalek]

OpenCV does not track libOpenNI2.so to OpenCV_LIBS and it should do it.

No.
OpenCV_LIBS is a top-level list of OpenCV modules only.
Dependencies should be handled by CMake-generated files (e.g. in case of static builds). This is why we don't care about bzip2 dependency of ffmpeg (transient dependency).

According to the message from linker you need to properly install (standard location) or properly specify user-defined location of libOpenNI2.so on the target device.

LD_LIBRARY_PATH

Try LIBRARY_PATH: https://stackoverflow.com/a/4250666

---

OpenCV build system not a "package manager" and it doesn't responsible for installing and configuration of 3rdparty dependencies.

[mshabunin]

I have a linkage issue:
warning: libOpenNI2.so, needed by /home/xxx/yyy/build/armimx8/opencv/lib/libopencv_videoio.so.4.6.0, not found (try using -rpath or -rpath-link)

Use -DCMAKE_EXE_LINKER_FLAGS="-Wl,--allow-shlib-undefined" in your application build process to fix this problem.

[asmorkalov]

Status:

• issue 1: @fengyuentau Could you update documentation for the release?
• issue 2: not an opencv issue. Proposed workaround.