Significant depth data mistakes, "Ghost Object" on D435 camera

[li-peize]

Required Info
Camera Model	D435
Firmware Version	05.12.05.00
Operating System & Version	Ubuntu 16.04
Kernel Version (Linux Only)	4.4.38-Tegra aarch64
Platform	NVIDIA Jetson TX2
SDK Version	2.13.0
Language	C++
Segment	Robot

Issue Description

I am using D435 and D435i to do robot obstacle avoidance. There are some mistakes in depth map that the robot misreport as obstacle.

Sample depth and RGB images:

1. Facing a wall more than 10 meters away.
   Cyan “Ghost Object” on depth map, 1.47 meters away.
   

2. Facing real object 3.4 meters away.
   Yellow “Ghost Objects” on depth map, 1.45 meters away.
   Green “Ghost Objects” on depth map, 1.07 meters away.
   

3. “Ghost Cliff” on the ground, one side 1.04 meters, other side 1.77 meters
   Blue “Ghost Object” top right corner, 0.718 meters away.
   

4. Facing a glass wall 2.5 meters away.
   Blue “Ghost Objects” on depth map, 0.47 meters away.
   

5. Facing a glass wall 2.5 meters away.
   Blue “Ghost Objects” on depth map, 0.47 meters away.
   

6. Facing a building more than 10 meters away.
   Blue “Ghost Objects” on depth map, 0.84 meters away.
   

[MartyG-RealSense]

Hi @lipeize-pku I was going to suggest using the On-Chip Calibration function of the RealSense Viewer to try to improve your depth image results. I see though from the description at the top of this issue that you are using an extremely old RealSense SDK version from June 2018 (2.13.0). If you are unable to use a modern SDK version such as 2.35.2 for some reason then I would recommend using the Dynamic Calibrator calibration tool.

Please note that the tool requires a USB 3 connection.

Page 14 onward of the user guide for the tool explains how to install it on Ubuntu 16.04.

https://www.intelrealsense.com/wp-content/uploads/2019/04/RealSense_D400_Dyn_Calib_User_Guide.pdf

If you are using SDK 2.13.0 though then this could be why you are not getting best results, as the RealSense SDK has improved massively since that very early version.

[li-peize]

Thanks for your timely reply! I will update the SDK and try On-Chip Calibration. I will report back after that.

One little thing:

If you are using SDK 2.13.0 though then this could be why you are not getting best results, as the RealSense SDK has improved massively since that very early version.

Isn't the depth calculation supposed to be done on D435 itself? Then how come the SDK version can affect the result?

[MartyG-RealSense]

The Vision Processor D4 hardware inside the camera performs a set of adjustments described in the PDF document linked to below. The data is then passed to the computer / computing device to be organised and used by applications.

https://www.stemmer-imaging.com/media/uploads/cameras/13/136603-Intel-RealSense-D4-Vision-Processor-Brochure.pdf

Operations performed on the data received by the computer, including post-processing, are changes that are performed in the computer's CPU / GPU chips by the RealSense SDK software and not the camera hardware. So the quality of the results can depend partly on the SDK software version and also the firmware version, as features are added and improved and bugs fixed.

[dorodnic]

Hi @lipeize-pku

Calibration can only help, but I believe what you seeing is caused by horizontal/vertical repetitive patterns in the scene. This is somewhat unfortunate, but well known issue with stereo matching technology. We published two white-papers you may find useful:

1. Depth Map Improvements for Stereo-based Depth Cameras on Drones - this paper goes over very similar issues in context of collision avoidance for drones and suggests aggressive software filtering to invalidate problematic depth.

2. Optical Filters for Intel® RealSense™ Depth Cameras D400 - see section 4.2.2 on long-pass filter. If this can be applicable in your use-case, we have good feedback from collision avoidance customers using this method.

[MartyG-RealSense]

@dorodnic Yes I suspected repetitive pattern as well when I saw the outdoor far wall shaped like a series of fence posts, but was not so sure if the tree-tops were shaped similar enough to be perceived as a repetitive pattern or the vertical indoor window frames were close enough together. Thanks for the guidance!

[dorodnic]

In the paper, one of the conclusions was that repetitive patterns can be introduced into the scene artificially by the process of rectification in hardware. When image is being resampled during rectification, this can introduce Moir patterns. These subtle patterns are sometimes enough to confuse the matching algorithm. In most cases using High Accuracy preset is enough to filter out these artefacts, the paper also presents "Super High Accuracy" JSON, and if all else fails, there is the opencv filter. It will invalidate almost all pixels, but will keep object edges, making it acceptable for collision avoidance.

[dorodnic]

Hi @lipeize-pku
I hope you found some of provided information useful. Please open new issue if you have follow-up issues.