D455 vs. D435i vs L515 for 3D Room scanning

[peta-peta]

Dear all,
I want to scan rooms and create water tight meshes. I have started exploring intel realsense L515 with RTABMAP-Ros but I am not happy with the results. Mapping is very unstable and sensitive to daylight. Since I can see many successful applications using realsense D435i I am thinking to buy one, too, and use it directly with RTABMap. Today i learned that Intel has already released a new version D455 and I am wondering if any of you has some experience with it using RTABmap for mapping and odometry. Which one shall I buy, or do you suggest even a different camera or a combination of different devices for room scanning with RTABMap - Room has to be a watertight model with floor, walls, windows and ceiling.

Cheers Peter

[rlabs-oss]

Hi,
I own the D455, L515 and D415 (but not the D435) - the L515 I got working with rtabmap following the earlier posts on the camera. For the D455 it works out of the box. - Once the IMU gravity is adjusted the D455 works very well. Ive used it to scan my house, outside structures, roads etc. The only issue I have with the cameras is the lack of portability.*1 I tried building rtabmap for the pi4 except it segfaults on start. Realsense viewer is working.

*1 Ive also built my own 3D lens for a canon EOS M as a portable rig, knock off of the Lumix 3D which I also own, neither will calibrate in rtabmap even with ffmpeg tricks to crop/resize/lenscorrect the stream before passing it to the v4l2 loopback (and opening that in rtabmap) -Looking into opencv2 and how that mangles reality next.

D455 top, L515 bottom - both scanned with rtabmap (and changes for the L515 code)
https://urlme.net/dl/Sofa.jpg
Not complete due to the laptop dropping frames using the L515.
Mapping with the D455
https://urlme.net/dl/mapping.gif

[peta-peta]

Hi @rlabs-oss
thank you very much for your detailed explanation. I will give the D455 a try.
Regarding the L515, can you share your launch script with us? I suppose you run roslaunch rtabmap_ros with a launch script, right?
Best wishes,
Peter

[rlabs-oss]

Hi,
I added the changes to the standalone app as suggested in this issue: #574 then rebuilt it. Main changes being the L500 Sensor:
} else if ("L500 Depth Sensor" == module_name) { sensors[1] = elem; } else if ("Wide FOV Camera" == module_name)

and the depth sensor selection

if ((!ir_ && !irDepth_ && video_profile.stream_type() == RS2_STREAM_DEPTH) // L515 fix, use first available depth stream || (video_profile.width() == cameraWidth_ && video_profile.height() == cameraHeight_ && video_profile.fps() == cameraFps_)) //D400 series:

Currently I'm trying to figure out how to select the L515's infrared stream rather than the RGB stream which will be more useful in the dark.
-M

[matlabbe]

For rtabmap standalone, a new commit has been added for L515: 3047b7d

[peta-peta]

I tried the new commit for realsense L515.
I executed ./rtabmap-rgbd_camera 11 and got this error
[ERROR] (2020-10-20 20:04:53.761) CameraRealSense2.cpp:1374::captureImage() Missing frames (received 1)

[matlabbe]

Can you comment this line and see if it changes something?

[peta-peta]

I comment
//sensors[i].set_option(rs2_option::RS2_OPTION_GLOBAL_TIME_ENABLED, 1);
and recompiled cmake .. and make -j4 (or do I need to clean bin and build folders?)

Same Error.

[rlabs-oss]

I can detail the workarounds I made - 1st fix was to the gyro which if using Ubuntu20.04 the kernel stream sync isn't active.

This line
to:
`if(((iterA == iterB) && (stamp == iterA->first)) || isL500_)'

Since steams maybe de synced, I ignored them / pass them for the L500

And this line
to
` if((!isL500_) && sensors[i].supports(rs2_option::RS2_OPTION_GLOBAL_TIME_ENABLED))'

Desired frames is also problematic if the IMU info doesnt arrive on time, comment it out
` UTimer timer;
int desiredFramesetSize = 2;
//if(isL500_)
// desiredFramesetSize = 2;

More hacks really since if the kernel driver /stream sync is present the changes should not be needed.

[peta-peta]

@matlabbe and @rlabs-oss thank you very much! It works like charm.
I ve tried already some tools under ros, but the results with rtabmap standalone are already very impressive and much more stable! I will keep you posted about my progress.
Cheers Peter

[rlabs-oss]

The D455 + T265 quick room scan. The scanning was faster than shown in the video since I was post processing it and recording for upload.
https://youtu.be/IDWoLNk0l5o

[samhoff20]

Hi guys, even after the new commit I don't see the l515 in my source list in the standalone app? How do I initialize the camera? I do have librealsense2 and the standalone app.

[rlabs-oss]

Manually setup the camera via the Select sensor driver/More Options button, or Detection - Select source - more options.

• Source Type RGB-D
  Grabber for RGB-D devices...
• Driver Realsense2
• IR Emitter enabled
• IR mode unchecked
• Use Depth... unchecked.
  640 Stream width
  480 Stream height
  30 Hz rate
  In the standalone app should work.

[samhoff20]

Thank you! I seem to be having some realsense issues also but I will try this once I get them worked out!

[peta-peta]

I confirm that solution provided by rslabs-oss is working without problems. I was also able to use the standard settings of the D435 provided by RTABmap standalone tool for the L515.
Settings by rlabs-oss work better though.
Cheers Peter

[samhoff20]

So trying the custom source and the 415 both get me this output. Do you have any insight?

[samhoff20]

[rlabs-oss]

Hi,
I have not had a chance to try 0.20.6 with the workarounds added but will check it now.
-M

[rlabs-oss]

Leaving the gyro code alone (which will warn: "CameraRealSense2.cpp:409::getPoseAndIMU() Could not find gyro data to interpolate at image time 76198.835000 (earliest is 1604436689619.677734). Are sensors synchronized?" when using the L515 and Commenting out those lines to read
//if(isL500_)
// desiredFramesetSize = 3;

And adding the previous hack for global sync in this line
So it reads:
if((!isL500_) && sensors[i].supports(rs2_option::RS2_OPTION_GLOBAL_TIME_ENABLED))

Will enable the L515 at 640x480x30 in release 0.20.6 without causing the D415 to stop working.
I tested this in dual mode T265 + D415.
Single: D415, D455 (640x480x30, 60hz, and 1280x720x30)
L515: 640x480x30

(librealsense 2.39.0 October 1st release, and all camera firmwares updated)
Note: The above changes are "hacks" at best to work around the frame sync/global time issue on a standard kernel.

[samhoff20]

Thanks for the help. I tried the above edits but still can't initialize the camera. This is the output of the terminal running rtab (plus a few above that were cut off but can be seen in previous picture):

[rlabs-oss]

@samhoff20 Is the camera working with the realsense-viewer without error? Turn on all streams/modules in the viewer and watch the console log.

The difference for the changes is shown below.

diff ~/external-source/GitHub_RTAB/rtabmap/corelib/src/camera/CameraRealSense2.cpp corelib/src/camera/CameraRealSense2.cpp
1009c1009
< if(sensors[i].supports(rs2_option::RS2_OPTION_GLOBAL_TIME_ENABLED))

                   if((!isL500_) && sensors[i].supports(rs2_option::RS2_OPTION_GLOBAL_TIME_ENABLED))

1145,1146c1145,1146
< if(isL500_)
< desiredFramesetSize = 3;

          //if(isL500_)
          //      desiredFramesetSize = 3;

check the output from ldd against rtabmap in the bin folder to show which version of librealsense2 its using.

ldd rtabmap | grep real
librealsense2.so.2.39 => /usr/local/lib/librealsense2.so.2.39

[samhoff20]

This is the error I'm getting in the realsense terminal when trying to press play on rtabmap. realsense-viewer and having both depth sensor and rgb camera on (not motion module though).
$ realsense-viewer 04/11 07:44:02,123 ERROR [139668014343936] (types.h:313) set_xu(...). xioctl(UVCIOC_CTRL_QUERY) failed Last Error: No such device 04/11 07:44:02,124 ERROR [139668014343936] (global_timestamp_reader.cpp:239) Error during time_diff_keeper polling: set_xu(...). xioctl(UVCIOC_CTRL_QUERY) failed Last Error: No such device 04/11 07:44:02,247 ERROR [139667856922368] (types.h:313) xioctl(VIDIOC_DQBUF) failed for fd: 23 Last Error: No such device 04/11 07:44:02,247 ERROR [139667856922368] (backend-v4l2.cpp:1354) xioctl(VIDIOC_DQBUF) failed for fd: 23 Last Error: No such device 04/11 07:44:02,247 ERROR [139667659785984] (types.h:313) xioctl(VIDIOC_DQBUF) failed for fd: 27 Last Error: No such device 04/11 07:44:02,247 ERROR [139667659785984] (backend-v4l2.cpp:1354) xioctl(VIDIOC_DQBUF) failed for fd: 27 Last Error: No such device 04/11 07:44:02,247 ERROR [139667571734272] (types.h:313) xioctl(VIDIOC_DQBUF) failed for fd: 41 Last Error: No such device 04/11 07:44:02,247 ERROR [139667571734272] (backend-v4l2.cpp:1354) xioctl(VIDIOC_DQBUF) failed for fd: 41 Last Error: No such device 04/11 07:44:03,124 ERROR [139668014343936] (types.h:313) acquire: Cannot open '/dev/video0 Last Error: No such file or directory 04/11 07:44:03,125 ERROR [139668014343936] (global_timestamp_reader.cpp:239) Error during time_diff_keeper polling: acquire: Cannot open '/dev/video0 Last Error: No such file or directory

Then this last ERROR[139668014343936] repeats endlessly.

[samhoff20]

If I am just in the realsense viewer and I turn on both depth sensor and rgb turn on fine, but I get the following error when turning on the motion module if that is an issue: $ realsense-viewer 04/11 07:46:18,485 WARNING [140248687920960] (ds5-motion.cpp:480) IMU Calibration is not available, default intrinsic and extrinsic will be used. 04/11 07:46:18,492 WARNING [140247727863552] (backend-hid.cpp:681) HID set_power 1 failed for /sys/devices/pci0000:00/0000:00:14.0/usb2/2-7/2-7:1.7/0003:8086:0B64.0026/HID-SENSOR-200073.2.auto/iio:device0/buffer/enable 04/11 07:46:18,540 WARNING [140247711078144] (sensor.cpp:970) HID timestamp not found, switching to Host timestamps.

[rlabs-oss]

To clarify: realsense-viewer should not be open when using rtabmap they are both independent applications.
Also are you using the D415 or the L515 ? The changes above are only related to the L515. The D415 should be unaffected or as functional as in the previous releases.

[rlabs-oss]

Then this last ERROR[139668014343936] repeats endlessly.

I do not experience that. With just realsense-viewer open (and not rtabmap) do you see the same error? If so it could be related to the USB bandwidth, USB port type, how the usb cable is plugged in -sounds strange but its better to plug into the usb port first then the camera.
In realsense-viewer, with the camera connected click the Info button. is the "Usb Type Descriptor" 3.2? it should look something like this for the L515:

If any one of the L500 depth sensor, RGB camera and Motion module in realsense-viewer cannot be activated or generate lots of errors those should be resolved before using rtabmap.

[samhoff20]

I am using the L515.

Here is a snapshot of the info depth, rgb and motion turned on.

[Anderson191919]

Hello guys,
I have an issue with the L515 on windows with rtabmap (v 0.20.6 from https://ci.appveyor.com/project/matlabbe/rtabmap/branch/master/artifacts). unfortunately I get following error(s) when starting on rtabmap (like samhoff20)

[Anderson191919]

[Anderson191919]

Oh ok, I will install 2.39 librealsense. I will immediately give a response :)

[Anderson191919]

This version, correct?

[matlabbe]

The current Windows build uses 2.39:

rtabmap/.appveyor.yml

Line 119 in 80199f2

- ps: wget 'https://github.com/IntelRealSense/librealsense/releases/download/v2.39.0/Intel.RealSense.SDK-WIN10-2.39.0.2337.exe' -outfile realsense2.exe

Maybe if you have upgraded to 2.40 and a new firmware update happened, this could have make the sensor incompatible with 2.39 used by rtabmap. I will upgrade to 2.40 on my linux and windows computer and see if my D435i and T265 still work ok.

[Anderson191919]

ok thank you, I have tried 2.39.0.2337 but anyway the same error :/

[Anderson191919]

Just noticed librealsense is now V2.40.0 which may cause changes.

Using librealsense 2.39 on Windows 10 Pro, the L515 connected via USB3.2: RTAB-Map started and mapped as in Linux. -The default resolution is 848x480x60 - I changed that to 640x480x30 before starting.

log output is here

In device manager I see the L515 under USB devices and under Cameras Depth + RGB.

Ill check if this is related to the new V2.40 librealsense and find out if that changes the behavior.

do you see the l515 as one device? like herre?

[matlabbe]

Is RealSenseViewer 2.39 working correctly with your device? (can you change resolution to 640x480 of the RGB camera?)

[rlabs-oss]

do you see the l515 as one device? like herre?

yes. In realsense-viewer @Anderson191919 you should see for the (/opps motion sensor) click the RGB slider to enable the RGB camera:-

[Anderson191919]

same here

[matlabbe]

This is the depth camera, and for the RGB camera?

[Anderson191919]

Is RealSenseViewer 2.39 working correctly with your device? (can you change resolution to 640x480 of the RGB camera?)

yes I can change to 640x480x30 and it works until I start the motion module, which kills the rgb stream. then I get some errors:

[Anderson191919]

This is the depth camera, and for the RGB camera?

it is somehow not possible to change to 640x480 for the rgb stream

[rlabs-oss]

Should be, e.g

@matlabbe Im in the process of build librealsense v2.40 and will also test that, not for this issue but generally over the cams I own. - The release notes recommend cmake .. -DFORCE_RSUSB_BACKEND=TRUE For V5 kernels.

[matlabbe]

I tried 2.40 on Ubuntu with D435i and T265 and it works as usual. Currently rebuilding Windows artifacts with 2.40, but I think it won't change anything, it is like the two cameras have not the same version, or that the firmwares are different. @Anderson191919 make sure you plug in the camera in a USB3 port, as long as you cannot have RGB 640x480 working in RealSenseViewer, it won't work in rtabmap either. You could try RealSenseViewer 2.40 to see if you can get the RGB 640x480 working.

[rlabs-oss]

I tried 2.40 on Ubuntu with D435i and T265 and it works as usual.

@matlabbe Rebuilt rtabmap + V2.40 librealsense: After camera firmware upgrades. The L515 works with global sync unchecked. The D455 reported a motion module failure on starting it and recovered (in realsense-viewer). - didn't affect rtabmap, The D455 is exceptionally fast with the madgwick filter enabled. D415 works however the IR mode inc depth IR looks quite underexposed.

[matlabbe]

Thank you @rlabs-oss! For the Global Sync issue, I think the original patch I received was from someone working on Windows, so the driver may work differently and Global Sync may work only on Windows (as you stated that kernel versions can affect this issue).

For D415 IR exposure, maybe there are some options available to change that. Check in RealSenseViewer for such options. You can then save the profile and load the presets in rtabmap with "D400 Series Visual Presets" option in Preferences->Source dialog. Some presets (look for best accuracy presets for rtabmap).

[Anderson191919]

I tried 2.40 on Ubuntu with D435i and T265 and it works as usual. Currently rebuilding Windows artifacts with 2.40, but I think it won't change anything, it is like the two cameras have not the same version, or that the firmwares are different. @Anderson191919 make sure you plug in the camera in a USB3 port, as long as you cannot have RGB 640x480 working in RealSenseViewer, it won't work in rtabmap either. You could try RealSenseViewer 2.40 to see if you can get the RGB 640x480 working.

Have installed 2.40 but still no option of 640x480 for rgb stream :(

[Anderson191919]

and plugged in usb 3

[Anderson191919]

okkkkkkkkkeeeeyyyy i have solved it. thanks again for your help. maybe someone else have or will have the problem. here is the solution:

1. download following firmware: 1.5.0.0 for the l515 (https://downloadcenter.intel.com/de/product/201775)
   
2. extract it
3. run Intel.RealSense.Viewer.exe
4. update the firmware by selecting the .bin-file
   
5. now you have the option to select 640x480x30
   
6. start rtabmap (v0.20.6)
7. under preferences... configure these settings:
   
8. enjoy your scan :)

again thanks for your help @matlabbe and @rlabs-oss 👍

[rlabs-oss]

@Anderson191919 Happy to read you have solved this. The realsense-viewer normally prompts to update firmware but perhaps there was something preventing this and nice to see the images how to do that which will help others with the same issue.

[Anderson191919]

@Anderson191919 Happy to read you have solved this. The realsense-viewer normally prompts to update firmware but perhaps there was something preventing this and nice to see the images how to do that which will help others with the same issue.

I hope so :) the viewer prompts me with an update but the updated firmware does not work either. I have also tried version

• 1.4.1.0
• 1.4.1.2
• 1.5.1.3
• 1.5.2.0.

All of them do not work for me but 1.5.0.0 works

[matlabbe]

Small note: for convenience, you don't have to explicitly set 640x480x30Hz, the resolution 848x480x60Hz parameters are ignored when L500 is detected, 640x480x30 is hard-coded:

rtabmap/corelib/src/camera/CameraRealSense2.cpp

Lines 704 to 707 in 7859313

	if(isL500_ &&
	(video_profile.width() == 640 &&
	video_profile.height() == 480 &&
	video_profile.fps() == 30))

When I will get my hands on a L515, I will check to add more resolution options and maybe also support IR-only mode (to test this demo on L515).

[rlabs-oss]

@Anderson191919 Happy to read you have solved this. The realsense-viewer normally prompts to update firmware but perhaps there was something preventing this and nice to see the images how to do that which will help others with the same issue.

I hope so :) the viewer prompts me with an update but the updated firmware does not work either. I have also tried version

* 1.4.1.0

* 1.4.1.2

* 1.5.1.3

* 1.5.2.0.

All of them do not work for me but 1.5.0.0 works

Does the update process show an error? Updating the firmware on the L515 does take time and appears to stop but eventually worked for me. Maybe check the USB port or try to confirm the port really is USB3.2 and also not via a hub. Original cable and try the rear motherboard ports if using a desktop. The speed at which the plug is inserted also matters its best to plug to the computer first then the camera.

[Anderson191919]

During the firmware update the console showed some errors. Nevertheless the flash worked and got the corresponding firmware version. The firmware flashes with the other versions also work fine but are not showing 640x480x30 options in realsense.viewer. Therefore i used firmware 1.5.0.0

Maybe this also helps @samhoff20

Best regards ^^

[shuixing85]

@matlabbe and @rlabs-oss thank you very much! It works like charm.
I ve tried already some tools under ros, but the results with rtabmap standalone are already very impressive and much more stable! I will keep you posted about my progress.
Cheers Peter

Hi Peta,

I have same error like
[ERROR] (2020-10-20 20:04:53.761) CameraRealSense2.cpp:1374::captureImage() Missing frames (received 1)

The only difference is I'm running on win10,

can you please share your source code/commit to me?

thanks a lot,
William

[peta-peta]

Hi William, this means that your camera is not set up properly within RTABmap. As explained above you need to untick the option Global time sync. See in the following picture:

I am really interested on the accuracy improvement with synchronization. Are there any attempts to solve this issue?

[sumitsarkar1]

this works fine after using this commit...I did the 3 hacks on Ubuntu 16 with L515...However I found only the last hack // if(isL500_) // desiredFramesetSize = 3; good enough to get L515 functional with RTabmap...However the 3D map created is not atall good due to some reason..I keep getting this warning CameraRealSense2.cpp:312::getPoseAndIMU() Could not find acc data to interpolate at image time 1606805391716.049561 after waiting 35 ms ...with all the 3 hacks too L515 is functional but the 3D map created is not good..it has multiple impressions of the same object as the lidar is moved slowly...looks like the 3D maps are not getting "stitched" togethar due to inaccurate motion estimation...any suggestions...

EDIT : this solved my problem
So...
1... do the firmwire update
2....do the git checkout 3047b7d
3....do this hack only desiredFramesetSize = 2; //3
4....set 640 x 480 x 30Hz
5....uncheck everything except IR emitter enabled
RtabMap should work with L515