Get the top view of the object from the 3d point cloud xy coordinates

[EthanUNC]

---

Required Info
Camera Model	D435i
Firmware Version
Operating System & Version	win11
Platform	PC
SDK Version	sdk2.0
Language	python

Issue Description

My problem was that I had a scene where I was going to shoot a gear with my camera tilted, and after I had obtained his 3D point cloud model, I wanted to capture a top view of the gear, which should have been drawn from the xy coordinates of the 3D point cloud. There's some way I can get this picture.

[MartyG-RealSense]

Hi @EthanUNC Are you trying to remove height from the pointcloud so that only the top of the gear is shown as though it is a flat 2D drawing, please?

If you are, and if the camera is in a fixed position, then you could use a post-processing Threshold filter to set a maximum distance from the camera that corresponds to the top of the gear. Depth values greater than the defined maximum distance should then be excluded from the image. #8170 (comment) has an example of Python code for defining and applying the threshold filter.

An alternative approach, if your pointcloud uses the rs2_deproject_pixel_to_point instruction to obtain the 3D coordinates, would be to then use rs2_project_point_to_pixel to convert the 3D XYZ values back to 2D XY values.

[EthanUNC]

@MartyG-RealSense Thanks for your quick reply, but I noticed that rs2_project_point_to_pixel is undefined. How do I use this function? Any examples? In response to my question, is it unnecessary to generate a 3D point cloud model? Do I just need to get the 3D coordinates of the pixels and align the rgb with the depth image?

[MartyG-RealSense]

Once rs2_deproject_pixel_to_point has been used to convert 2D XY pixel coordinates to 3D XYZ points then you can use rs2_project_point_to_pixel to convert 3D points back to 2D pixels. The link below has a Python script that demonstrates this.

https://stackoverflow.com/questions/50716249/mapping-depth-pixels-to-colour-pixels-using-pyrealsense2

However, instead of using deproject / project, the other main alternative way to generate a pointcloud - pc.calculate - might work better for you. A Python script for this pointcloud method is at #4612 (comment)

[EthanUNC]

Hi,@MartyG-RealSense, I just run the code you mentioned, but the coordinates I got are all 0, so here's the code and the output.Is it a wrong configuration of the camera? Sorry, I'm just starting to learn how to use this camera.

import pyrealsense2 as rs
import numpy as np

pipeline = rs.pipeline()
config = rs.config()
config.enable_stream(rs.stream.depth, 640, 480, rs.format.z16, 30)
config.enable_stream(rs.stream.color, 640, 480, rs.format.bgr8, 30)
pc = rs.pointcloud()
pipeline.start(config)

i = 0

while i<1:
frames = pipeline.wait_for_frames()
depth_frame = frames.get_depth_frame()
print(depth_frame)
color_frame = frames.get_color_frame()
print(color_frame)
i = i+1
pc.map_to(color_frame)
points = pc.calculate(depth_frame)
vtx = np.asanyarray(points.get_vertices())
tex = np.asanyarray(points.get_texture_coordinates())
print(type(points), points)
print(type(vtx), vtx.shape, vtx)
print(type(tex), tex.shape, tex)
pipeline.stop()

<pyrealsense2.frame Z16 #0 1709546607096.326416>
<pyrealsense2.frame BGR8 #1 1709546607053.878174>
<class 'pyrealsense2.pyrealsense2.points'> <pyrealsense2.frame XYZ32F #0 1709546607096.326416>
<class 'numpy.ndarray'> (307200,) [(-0., -0., 0.) (-0., -0., 0.) (-0., -0., 0.) ... ( 0., 0., 0.)
( 0., 0., 0.) ( 0., 0., 0.)]
<class 'numpy.ndarray'> (307200,) [(0., 0.) (0., 0.) (0., 0.) ... (0., 0.) (0., 0.) (0., 0.)]

[MartyG-RealSense]

#3631 (comment) advices that when retrieving vertices, the values may be zero due to how numpy summarizes when printing large arrays. Further advice in this discussion suggests converting the large array into a list and then printing it to get more information from the vertices.

Guides can be found online for numpy to list conversion by googling for the term convert numpy array into list, which returns search results such as the link below.

https://www.digitalocean.com/community/tutorials/python-convert-numpy-array-to-list

[EthanUNC]

@MartyG-RealSense hi，I've got the coordinates I want for x and z, but if I wanted to display these coordinates directly, I wouldn't be able to identify the image I want. So I figured I should get the rgb information for each pixel, and I've seen many examples of texture coordinates being used. I want to know if I want to get the texture coordinates if I just want to get the rgb information of the pixel.

[MartyG-RealSense]

My understanding is that each texture coordinate has a (u, v) value which ranges from [0-1] that can be mapped to the color frame.

[EthanUNC]

@MartyG-RealSense I think I need to take the rgb information of each 3D coordinate point and turn this rgb information into an array so that I can distinguish my target when displaying the image. So, are there examples of getting rgb information for coordinate points? thanks！.

[MartyG-RealSense]

As you mention distinguishing your target, are you thinking of an array in terms of a grid overlay on the RGB image like the project in the link below?

https://support.intelrealsense.com/hc/en-us/community/posts/360053086413-D435-Area-without-any-depth-information

Or do you have something else in mind? For example, performing depth-color alignment and then creating 3D pointcloud coordinates with the instruction rs2_deproject_pixel_to_point and converting 3D points back into 2D pixel coordinates with rs2_project_point_to_pixel. Thanks!

[EthanUNC]

@MartyG-RealSense I'm sorry, my expression may be wrong. My goal is to get a top view of the object and shoot the object from the side. So I want to get the three-dimensional coordinates of this scene. And then remove the z-axis coordinates from the three-dimensional coordinates. And then use the dots to show these three-dimensional coordinates. But there is a problem, if there is no rgb information for every coordinate point and only a few points are displayed on the image, I can't tell which points are my objects. So I want to get the rgb information for each pixel, which should be a numpy array.
I know that 3d point clouds can get rgb information, but I don't know how to get the top view I want. In my understanding, rs2_project_point_to_pixel can not achieve the purpose I want. Or there's something wrong with my understanding.

[MartyG-RealSense]

It sounds as though your camera will be pointing at the side of the object but you want to be able to view the top of the object as well. Is that correct, please?

If the camera is in a fixed position and pointing towards the side of the object then the camera will not be able to see the top. There will therefore be no data representing a top view as it is not in the camera's field of view. This means that a 3D image viewing an object in side-on perspective cannot simply be 'collapsed' in height into a paper-thin 2D blueprint view to show what is on top of the object.

In this situation, you can either add a second camera placed above the object pointing down and then combine the side-on and top-down views of the cameras together into a single combined image. Or you can move a single camera, take individual pointcloud captures of the side and top views and save them to a file, and then 'stitch' the pointcloud files together to create the combined view.

[EthanUNC]

@MartyG-RealSense
Not exactly on the side, but a position where the camera can see the side and the top at the same time. So, I need a way to reconstruct the top view of the object.
I have seen information about 3d reconstruction, but it seems that specific software is required, such as "RecFuiso", or the reconstruction of more crude models in ANN.

[MartyG-RealSense]

3D reconstruction does not necessarily need commercial software such as RecFusion, but the complexities of stitching separate images together does make use of commercial tools preferable if there is the option to do so.

If it is possible to use C++ instead of Python then the RealSense SDK's rs-kinfu tool allows a pointcloud to be constructed by moving a single camera around an object.

https://github.com/IntelRealSense/librealsense/tree/master/wrappers/opencv/kinfu

If you can capture the top on a pointcloud then it is possible to rotate the pointcloud to a different viewing perspective, like in the SDK's opencv_pointcloud_viewer.py Python example program at the link below.

https://github.com/IntelRealSense/librealsense/blob/master/wrappers/python/examples/opencv_pointcloud_viewer.py

[EthanUNC]

@MartyG-RealSense
My method is to get the three-dimensional coordinates directly. use

points = pc.calculate(depth_frame)
vtx = np.asanyarray(points.get_vertices())
vtx_list = vtx.tolist()
new_list = [(a, c) for (a, b, c) in vtx_list]

so that i get the x z coordinates.i wonder if this is possible?

[MartyG-RealSense]

Retrieving vertices individually as x, y and z instead of printing all of the coordinates is rare. If retrieving x and y individually from the vertices is your goal then the Python script at the link below looks as though it may be helpful.

https://support.intelrealsense.com/hc/en-us/community/posts/21674251140883/comments/21885537480851

[MartyG-RealSense]

Hi @EthanUNC Do you require further assistance with this case, please? Thanks!

[MartyG-RealSense]

Case closed due to no further comments received.