Realsense2: ClassNotFoundException

[Faaux]

Hi :),
I am trying to use procamcalib (latest) with a Realsense D435 device. Sadly I cannot pick Realsense2 for the frameGrabber dropdown (it wont select). When changing a settings file (see attached file settings.log) manually to use the Realsense2 Framegrabber and then loading it, I get an ClassNotFoundException.
settings.log

The module folder does contain the librealsense2 module as well as the needed DLL so I dont think i need to install additional dependencies.

Any help would be much appreciated.

Best,
Faaux

[saudet]

That's a bug in JavaCV, thanks for reporting! I've fixed it in the latest commit, but to make it work,
you can simply replace javacv.jar and javacv-platform.jar with the latest versions from here:
https://oss.sonatype.org/content/repositories/snapshots/org/bytedeco/javacv/1.5.5-SNAPSHOT/
https://oss.sonatype.org/content/repositories/snapshots/org/bytedeco/javacv-platform/1.5.5-SNAPSHOT/

/cc @cansik

[Faaux]

Hi! Thanks for the quick reply, I hadn't had a chance to try it before.

While the original error is now fixed it seems like that procamcalib does not enable any streams of the Realsense2 device and I haven't seen an option where I could do that.

Specifically the following line throws: https://github.com/bytedeco/javacv/blob/00793cc82ad94112971623e72093a854c87f2c92/src/main/java/org/bytedeco/javacv/RealSense2FrameGrabber.java#L187

Since procamcalib only works on CameraDevices and FrameGrabbers no stream is enabled. Is there currently a way to do this via the UI? Below is my current settings file.

Furthermore: How would I calibrate multiple D435 cameras at once? E.g left and right IR as well as RGB.

<?xml version="1.0" encoding="UTF-8"?>
<java version="15.0.1" class="java.beans.XMLDecoder">
 <object class="org.bytedeco.javacv.CameraSettings" id="CameraSettings0">
  <void method="add">
   <object class="org.bytedeco.javacv.CameraDevice$CalibrationSettings" id="CameraDevice$CalibrationSettings0">
    <void property="beanContext">
     <object idref="CameraSettings0"/>
    </void>
    <void id="Integer0" property="bitsPerPixel"/>
    <void property="deviceNumber">
     <object idref="Integer0"/>
    </void>
    <void property="format">
     <string>RS2_STREAM_INFRARED</string>
    </void>
    <void property="frameGrabber">
     <class>org.bytedeco.javacv.RealSense2FrameGrabber</class>
    </void>
    <void property="frameRate">
     <double>30.0</double>
    </void>
    <void property="imageHeight">
     <int>1280</int>
    </void>
    <void property="imageWidth">
     <int>720</int>
    </void>
   </object>
  </void>
  <void property="frameGrabber">
   <class>org.bytedeco.javacv.RealSense2FrameGrabber</class>
  </void>
 </object>
 <object class="org.bytedeco.javacv.ProjectorSettings" id="ProjectorSettings0">
  <void method="add">
   <object class="org.bytedeco.javacv.ProjectorDevice$CalibrationSettings">
    <void property="beanContext">
     <object idref="ProjectorSettings0"/>
    </void>
   </object>
  </void>
 </object>
 <object class="org.bytedeco.javacv.Marker$ArraySettings"/>
 <object class="org.bytedeco.javacv.MarkerDetector$Settings"/>
 <object class="org.bytedeco.procamcalib.CalibrationWorker$GeometricSettings"/>
 <object class="org.bytedeco.procamcalib.CalibrationWorker$ColorSettings"/>
</java>

Cheers,
Faaux

[saudet]

We'd have to implement using the FrameGrabber.videoStream property as is done in others like FFmpegFrameGrabber.
@cansik Would you be able to work on that?

As for supporting multiple cameras, simply increase the "quantity" property to 2 or more.

[Faaux]

That would be awesome! Thanks again for the quick reply.

As for multiple cameras let me rephrase:
I can have multiple cameras pointing at the Realsense device, but I would still need to configure which image stream I am referring to on each camera. Which is probably what the FrameGrabber.videoStream is for :)

[saudet]

The bug fix has been released with version 1.5.5, but work to support multiple video streams hasn't started yet, so I'll leave this issue opened for that. Contributions are welcome!

[cansik]

Just for clarification, the videostream tells the camera which stream id to use?

I am thinking about how this would be implemented for the RealSense2FrameGrabber. The problem there is that the user has to enable the different streams, so videostream would refer to the order, the streams have been enabled? Or is it a fictional order 1=video, 2=ir1, 3=ir2, 4=depth, 5=pose?

And what about the stream format, because the color stream is currently RGB8, IR is Y8 and depth Z16. So it would be necessary to convert the IR and depth channel into the RGB8 format? And is the depth colorized or not? How about the depth range, because just mapping the 0-65536 to 8bits does not look good.

@saudet Is there a similar FrameGrabber with different input channels that already implements videostream so I can have a look how they solved these problems?

[saudet]

videoStream gets used by FFmpegFrameGrabber, but there it makes sense because there is actually an order in the streams. It looks like the streams already have numbers in the case of RealSense too so let's just use those: https://github.com/bytedeco/javacpp-presets/blob/master/librealsense2/src/gen/java/org/bytedeco/librealsense2/global/realsense2.java#L2683

As for the video format, well, don't worry about it. ProCamCalib might not be able to use them all, but calibration only makes sense for unprocessed images (color and IR), and it can use both RGB24 and GRAY8, so that's not an issue.

[cansik]

We could use the stream order defined by realsense, but there are cameras with two infrared cameras or two colour cameras. There would then be no way to switch the stream from the left to the right channel. But we could implement it like that for now.

Thank you for the clarification on the image formats.

[saudet]

How can we differentiate between 2 cameras if they have the same stream number?

[cansik]

Ok, sorry, camera was not clear.

In the RS terminology there is a device, which contains sensors, which can be streamed. A stream can be identified by the sensor and a stream_index. A D435 for example has the following structure:

• Device D435
  • RGB Sensor
    • Color Stream (type COLOR, index 0)
  • Depth Sensor
    • Depth Stream (type DEPTH, index 0)
    • IR Stream 1 (type INFRARED, index 1)
    • IR Stream 2 (type INFRARED, index 2)

By default no stream is active and has to be enabled by the user of the library. I have now added the default color stream with the PR (will add the width / height setting). So the question is now how to index these different streams, because depending on the enabled streams, the camera may send us different streams which are not identifiable by a single number.

Of course we could just define the order on our own.

[saudet]

Ok, I see. Well, I think videoStream for FFmpeg sort works like the default value for your streams in grab(), so instead of picking streams.get(0), you'd do streams.get(videoStream) and that'd be good enough for our purposes. That said, it wouldn't allow a user to enable the streams, and that's not something that is part of the interface for FrameGrabber, so if none are selected by default, let's just enable the ones that are most used like RS2_STREAM_DEPTH, RS2_STREAM_COLOR, and RS2_STREAM_INFRARED. Does that make sense?

[cansik]

Sounds good to me, I just changed the order of the streams. Color should be index 0 to have this one as default stream. I have implemented it in the PR.

[saudet]

Thanks @cansik for implementing support for video stream selection! @Faaux please give it a try by replacing modules/javacv.jar with this file: https://oss.sonatype.org/content/repositories/snapshots/org/bytedeco/javacv/1.5.8-SNAPSHOT/javacv-1.5.8-20220215.010447-6.jar
Contrary to the above, we probably need to set "videoStream" to 1 or possibly larger values to get the color or IR ones.

[cansik]

@saudet Do you think it would make sense to release a new version with the fix applied in 1.5.8? At the moment, calibration with realsense cameras is not possible (except if you manually replace the jar).

[saudet]

It's going to be released as 1.5.8, yes.

[cansik]

@saudet I tried to calibrate a RealSense D455 with a projector and it seems that there are still problems. As mentioned in this comment, the order of the streams if depth, color, ir. So grab() is not returning the color image but the depth image. But I also could not find a property in the UI to change the videoStream to another stream index. How is it possible to change the stream id?

And second, it seems that ProCamCalib expects a gray-8bit image, but what the RealSense2 delivers (if grab() is overwritten with grabColor() is RGB-8bit. This leads to an incorrect channel number error in the CalibrationWorker.java#L357. But maybe this has todo with me changing and analysing the code. Does ProCamCalib convert input frames to the correct format? And which format information is used for that decision (because RealSense2 has multiple active streams)?

[saudet]

@saudet I tried to calibrate a RealSense D455 with a projector and it seems that there are still problems. As mentioned in this comment, the order of the streams if depth, color, ir. So grab() is not returning the color image but the depth image. But I also could not find a property in the UI to change the videoStream to another stream index. How is it possible to change the stream id?

That's probably something that we need to add, yes. It shouldn't be too hard. Would you mind giving it a try?

And second, it seems that ProCamCalib expects a gray-8bit image, but what the RealSense2 delivers (if grab() is overwritten with grabColor() is RGB-8bit. This leads to an incorrect channel number error in the CalibrationWorker.java#L357. But maybe this has todo with me changing and analysing the code. Does ProCamCalib convert input frames to the correct format? And which format information is used for that decision (because RealSense2 has multiple active streams)?

Everything gets converted to grayscale since the ImageMode is set to GRAY here:
https://github.com/bytedeco/procamcalib/blob/master/src/main/java/org/bytedeco/procamcalib/CalibrationWorker.java#L195
If RealSense2FrameGrabber doesn't respect that property, that's a problem with RealSense2FrameGrabber, not ProCamCalib.