How to measure the distance from camera to a pixel on the camera feed?

[Dilip-Gurumurthy]

---

Required Info
Camera Model	D415
Firmware Version	05.11.01.100
Operating System & Version	Linux (Ubuntu 18.04)
Kernel Version (Linux Only)	5.3.0
Platform	PC
SDK Version	2.17
Language	Python 3
Segment	Others

Issue Description

I am trying to get the distance from the camera to a pixel on the camera feed. The following piece of code gets the depth of pixels given (x,y),

import pyrealsense2 as rs
import numpy as np
import time

pipeline = rs.pipeline()
config = rs.config()
config.enable_stream(rs.stream.depth, 1280, 720, rs.format.z16, 30)

pipeline.start(config)
count = 0
try:
    while True:
        frames = pipeline.wait_for_frames()
        depth = frames.get_depth_frame()

        if not depth: continue

        count +=1
        x1, y1 = 80, 30
        midx, midy = 640, 360

        #Pixel 1
        dist = depth.get_distance(x1, y1)
        dist = dist*1000
        print("Pixel 1 Distance in mm:", dist)
        
        #Mid Pixel
        dist = depth.get_distance(midx, midy)
        dist = dist*1000
        # print("mid Pixel Distance in mm:", dist)

    pipeline.stop()
    exit(0)

except Exception as e:
    print(e)
    pass

Here, what I exactly want to do is measure the distance from camera to the specified pixel.

If I use a tape to cross verify the distance to the middle point of video feed(1280X720 frame), then the value is quite accurate(I have calibrated the camera pretty well using Realsense Tool) while comparing with the code's output. But, the issue starts when I measure for pixels that are not in the middle. The issue can be broken down like the following,

1. The above code calculates the depth. What should be done to get the distance to a pixel from camera?
2. The model D415(image below) has multiple sensors(RGB and IRs). From which point on the surface of the camera should I consider for measuring(with something like a tape) the distance to a particular point(or object)? so that, I can cross-verify the output with actual values.

[kafan1986]

@Dilip-Gurumurthy If you using only the depth camera, then your code should work fine. But if you are looking at the RGB image and trying to pass the pixel value of RGB image and measure the distance in depth image, then alignment is required. Also, by measuring the distance using tape, you mean you are measuring the "Z" distance only and not the euclidean distance from camera to the point in real world? Also, the do remember the depth measurement of D415 has some error and it increases with distance from camera. What kind of discrepancies are your seeing?

[MartyG-RealSense]

Thanks so much @kafan1986 for your help with this case!

@Dilip-Gurumurthy I was researching the case too, so here's my input:

1. There are more references for getting the distance from a pre-recorded bag file with pyrealsense2 than there are for analysing a live camera stream (which your program seems to do). If you are using a live stream, the link below may be useful reading.

#1904

2. I hope that the link below will be helpful in answering this question about the point from which depth is measured on the D415 camera.

#6239 (comment)

[Dilip-Gurumurthy]

Thanks @kafan1986 and @MartyG-RealSense for providing valuable suggestions.

@kafan1986 Yes! I was using RGB image to get the pixel values, thought the cameras would be aligned. But, as you said they weren't, so I used align() method to align RGB image to Depth image and modified my code as shown below.

align_to = rs.stream.depth #added
align = rs.align(align_to )#added
try:
    while True:
        frames = pipeline.wait_for_frames()
        aligned_frames =  align.process(frames)

        depth_frame = aligned_frames.get_depth_frame()
        aligned_color_frame = aligned_frames.get_color_frame()

        if not depth_frame or not aligned_color_frame: continue

        depth_image = np.asanyarray(depth_frame.get_data())
        color_image = np.asanyarray(aligned_color_frame.get_data())

I am trying to measure the distance from camera to a point in the real world as shown below.

I am looking to get the values of d1 and d2. To do this, I use opencv's imshow() method to display the aligned image (color_image from above code). In the window displayed by imshow(), I point the mouse at a pixel and get its (x,y) values. Later, I pass these values to get_distance(x,y) method and collect the depth value. This seems to work but I don't know whether this is the right way to do it. Is there a z-axis parameter that I should consider? or will the get_distance() method handle it?

I didn't know that depth measurement of D415 has some error and it increases with distance from camera. I calculated z-accuracy, plane fit RMS error, subpixel RMS error using depth analysis tool provided by realsense. How should I handle these values?

@MartyG-RealSense Your links pointed me in the right direction. I was able to find information on alignment and also from which point on camera surface to measure the distance to real-world object. It is the from the centre of the left-imager for D415. But I need a little more information to resolve my issue. Could you please help me out with the above mentioned queries(highlighted in Bold).

[MartyG-RealSense]

@Dilip-Gurumurthy Section 3e on page 3 of Intel's camera tuning guide for the 400 Series cameras provides a formula for calculating RMS error.

https://www.intel.com/content/dam/support/us/en/documents/emerging-technologies/intel-realsense-technology/BKMs_Tuning_RealSense_D4xx_Cam.pdf

You may get better accuracy performance from the new RealSense D455 camera model, as it has increased accuracy over distance by a factor of 2 compared to the D435 models. It also has improved support for alignment between RGB and depth, thanks to the RGB and depth both having the same field of view (FOV) size and being mounted on the same stiffener. Its RGB imager also has a fast global shutter for the first time like the shutter on the IR imagers.

If you prefer to continue using the D415 then the optimal resolution to use for depth accuracy is 1280x720.

My understanding is that get_distance() is suited to getting (x,y), and z is obtained by aligning depth to color (as @kafan1986 mentioned above) and using rs2_deproject_pixel_to_point to obtain the z coordinate.

[kafan1986]

@Dilip-Gurumurthy Alignment is a CPU intensive process and rather than aligning the entire image, that does it for all the pixels, you can rather do it just for the pixel(s) under consideration. In your case, rs2_project_color_pixel_to_depth_pixel will be followed by rs2_deproject_pixel_to_point, which will give you x, y, z coordinate of the pixel. Then the distance you are looking for is euclidean distance i.e. sqrt(x^2 + y^2 + z^2). You can search the issue thread, I have previously posted code for this functionality.

[Dilip-Gurumurthy]

@kafan1986 and @MartyG-RealSense your suggestion to use rs2_deproject_pixel_to_point was spot-on! After aligning color frame to depth frame using align method as I mentioned in my previous comment, I was able to get x,y,z values, using which I calculated the euclidean distance. I have included the script which finds the distance from camera to a given pixel below.

import pyrealsense2 as rs
import numpy as np
import math

pipeline = rs.pipeline()
config = rs.config()
config.enable_stream(rs.stream.depth, 1280, 720, rs.format.z16, 30)
config.enable_stream(rs.stream.color, 1280, 720, rs.format.bgr8, 30)
pipeline.start(config)

align_to = rs.stream.depth
align = rs.align(align_to)

try:
    while True:
        # This call waits until a new coherent set of frames is available on a device
        frames = pipeline.wait_for_frames()
        
        #Aligning color frame to depth frame
        aligned_frames =  align.process(frames)
        depth_frame = aligned_frames.get_depth_frame()
        aligned_color_frame = aligned_frames.get_color_frame()

        if not depth_frame or not aligned_color_frame: continue

        color_intrin = aligned_color_frame.profile.as_video_stream_profile().intrinsics
        depth_image = np.asanyarray(depth_frame.get_data())
        color_image = np.asanyarray(aligned_color_frame.get_data())
        #Use pixel value of  depth-aligned color image to get 3D axes
        x, y = 640, 360
        depth = depth_frame.get_distance(x, y)
        dx ,dy, dz = rs.rs2_deproject_pixel_to_point(color_intrin, [x,y], depth)
        distance = math.sqrt(((dx)**2) + ((dy)**2) + ((dz)**2))
        print("Distance from camera to pixel:", distance)
        print("Z-depth from camera surface to pixel surface:", depth)

except Exception as e:
    print(e)
    pass

finally:
    pipeline.stop()

So far, the results are quite satisfactory. However, I have one small doubt regarding depth measurement. As illustrated in the image below, irrespective of which pixel I consider on a flat surface perpendicular to the camera, the z-depth(z-axis value) returned from rs2_deproject_pixel_to_point should be the same for all the pixels, isn't it?
I see a gradual increase up to 0.4m in z-depth values as I consider pixels further away from the middle-point of frame. What could be the possible cause for it?

P.S : Inorder to use rs2_project_color_pixel_to_depth_pixel as suggested by @kafan1986 liberealsense version has to be above 2.25

[MartyG-RealSense]

@Dilip-Gurumurthy I considered your problem with the greater inaccuracy at the side of the image compared to the center. You are certainly not the first person to experience this.

Does the accuracy of measurement at the sides of the image improve if the camera moves further away from the observed object, and decrease as the camera moves closer to the object, please?

[kafan1986]

@Dilip-Gurumurthy I am not sure about the reason for this error. You mentioned the depth changes upto 0.4m. What is the actual depth at the center perpendicular plane? It will give the idea about error in terms of percentage and maybe it might be within the device's depth error or the camera might actually be not held at true perpendicular but rather at a slight angle, this can be confirmed that depth will increase on one side the center and will decrease on other side of the center by around Cosine of the angle that the device makes with the plane. Also, another reason might be actually due to the wrong depth, maybe re-calibrating the camera might increase the accuracy.

[Dilip-Gurumurthy]

Thanks a lot @kafan1986 and @MartyG-RealSense You guys are the best! Your suggestions are always spot-on!
Regarding my last issue about variation in depth from camera surface to a flat real-world object, I have not set up the camera to be perfectly perpendicular to the region of interest surface. Maybe this could be the possible reason for it. However, when I cross verified the results with that of ground truth, I found them to be quite accurate. So, no more issues.

[MartyG-RealSense]

Thanks so much for the update - I'm glad that @kafan1986 and I could be of help :)