[Realsense Customer Engineering Team Comment]

[scizors]

[Realsense Customer Engineering Team Comment]
Hi @alt01,

Your result should be mapped to color pixel. I modified the python code based on #1890 to convert from depth to 2D color.

you can also refer to below article.
https://github.com/IntelRealSense/librealsense/wiki/Projection-in-RealSense-SDK-2.0

import pyrealsense2 as rs
import numpy as np
config = rs.config()
config.enable_stream(rs.stream.depth, 1280, 720, rs.format.z16, 30)
config.enable_stream(rs.stream.color, 640, 480, rs.format.bgr8, 30)
pipeline = rs.pipeline()
pipe_profile = pipeline.start(config)
frames = pipeline.wait_for_frames()
depth_frame = frames.get_depth_frame()
color_frame = frames.get_color_frame()
# Intrinsics & Extrinsics
depth_intrin = depth_frame.profile.as_video_stream_profile().intrinsics
color_intrin = color_frame.profile.as_video_stream_profile().intrinsics
depth_to_color_extrin = depth_frame.profile.get_extrinsics_to(color_frame.profile)
color_to_depth_extrin = color_frame.profile.get_extrinsics_to(depth_frame.profile)
print("\n Depth intrinsics: " + str(depth_intrin))
print("\n Color intrinsics: " + str(color_intrin))
print("\n Depth to color extrinsics: " + str(depth_to_color_extrin))
# Depth scale - units of the values inside a depth frame, i.e how to convert the value to units of 1 meter
depth_sensor = pipe_profile.get_device().first_depth_sensor()
depth_scale = depth_sensor.get_depth_scale()
print("\n\t depth_scale: " + str(depth_scale))
depth_image = np.asanyarray(depth_frame.get_data())
depth_pixel = [200, 200]   # Random pixel
depth_value = depth_image[200][200]*depth_scale
print("\n\t depth_pixel@" + str(depth_pixel) + " value: " + str(depth_value) + " meter")
# from pixel to 3D point
depth_point = rs.rs2_deproject_pixel_to_point(depth_intrin, depth_pixel, depth_value)
print("\n\t 3D depth_point: " + str(depth_point))
# from 3D depth point to 3D color point
color_point = rs.rs2_transform_point_to_point(depth_to_color_extrin, depth_point)
print("\n\t 3D color_point: " + str(color_point))
# from color point to 2D color pixel
color_pixel = rs.rs2_project_point_to_pixel(color_intrin, color_point)
print("\n\t color_pixel: " + str(color_pixel))

Originally posted by @RealSense-Customer-Engineering in #2204 (comment)

[scizors]

why the resolution is different and how the match will happen

config.enable_stream(rs.stream.depth, 1280, 720, rs.format.z16, 30)
config.enable_stream(rs.stream.color, 640, 480, rs.format.bgr8, 30)

[lramati]

The matching occurs through the rs2_deproject_pixel_to_point, rs2_transform_point_to_point, and rs2_project_point_to_pixel functions. Their source code (In C++) can be found at https://github.com/IntelRealSense/librealsense/blob/master/include/librealsense2/rsutil.h
Note that even if the resolutions were the same, you would still need to call this sequence of functions because the depth and color imagers are not located in the exact same physical location