from numpy to open3d.image

[julienguegan]

___________________________ Description ___________________________
In python, I want to create a point cloud from numpy rgb image and depth image. But it is quite unclear how to do it as function o3d.geometry.Image() return confusing results for me. I am using images from IntelRealSense D415 :

# get data from D415
frame_present, frames = pipeline.try_wait_for_frames()
image_frame = frames.get_color_frame()
depth_frame = frames.get_depth_frame()
image = np.array(image_frame.get_data()) 
depth = np.array(depth_frame.get_data())

To this point, if I display my data with matplotlib everything looks normal (blurred on purpose) :

Then I convert it to open3d.Image object and call create_from_color_and_depth and display it content. What I don't understand is that the depth image is then fill of zeros ... :

image_o3d = o3d.geometry.Image(image)
depth_o3d = o3d.geometry.Image(depth)
rgbd_image = o3d.geometry.RGBDImage.create_from_color_and_depth(image_o3d, depth_o3d)

Then an other problem I have is that I also would like to rotate my images before working with open3d, like following:

image = np.rot90(image, axes=(1,0))
depth = np.rot90(depth, axes=(1,0))

And now, if I convert it to open3d.Image, I have the rgb image that seems to have rows et columns reverse or something like that (see below) and the depth image conversion is not working at all as I have the error RuntimeError: Image can only be initialized from c-style buffer.

From what I see and what @qianyizh said in #417, I feel that it is a problem of column-major array but I did not find real solution to my problem ...

Does someone ever encounter this problem and could help me ?

Thank you !

___________________________ Environment ___________________________

• Operating system: Windows 10 64-bit
• Python version: 3.8
• Open3D version: 0.13.0
• Is this remote workstation?: no
• How did you install Open3D?: pip

[sanskar107]

np.rot90 doesn't actually rotate the array in memory. It changes the stride making it incompatible with C-style arrays. You can use this function to make it contiguous before feeding it to open3d.
https://numpy.org/doc/stable/reference/generated/numpy.ascontiguousarray.html

[julienguegan]

@sanskar107 Indeed, it works with np.ascontiguousarray() ! Also do you have an answer for my first problem : why the depth is set to zero after applying o3d.geometry.RGBDImage.create_from_color_and_depth() ?

I have juste seen in the tutorial rgbd_image that it seems to have multiple dataset format,I will look into that ...

[julienguegan]

It is weird that in the tutorial, it says that depth is in 16-bit but when I look into the data it is only uint8 ... Does not really matter, so I am trying to cast my data to 8-bit (with astype('uint8')) but the image dynamic is not preserved :

[julienguegan]

After that I have write a custom function to cast my data to 8-bit and finally be able to send it to o3d.geometry.RGBDImage.create_from_color_and_depth() without having a depth set artificially to zero and then to point cloud (see below). But I feel that the data casting has deteriorated too much the accuracy of the point cloud. If someone has an idea, it would be a great help !

[ssheorey]

It is weird that in the tutorial, it says that depth is in 16-bit but when I look into the data it is only uint8 ... Does not really matter, so I am trying to cast my data to 8-bit (with astype('uint8')) but the image dynamic is not preserved :

This looks like int overflow. You have to compress the depth range (e.g.: divide by 2^8 if converting from uint16 to uint8) before switching to a smaller datatype.

[ssheorey]

After that I have write a custom function to cast my data to 8-bit and finally be able to send it to o3d.geometry.RGBDImage.create_from_color_and_depth() without having a depth set artificially to zero and then to point cloud (see below). But I feel that the data casting has deteriorated too much the accuracy of the point cloud. If someone has an idea, it would be a great help !

This result may be OK. You can estimate the quantization error in mm (16bit -> 8bit) and decide if that is OK for your application and depth sensor noise level. Usually it's not useful to have a very high depth resolution compared to the depth error from the RGBD sensor.

[julienguegan]

This looks like int overflow. You have to compress the depth range (e.g.: divide by 2^8 if converting from uint16 to uint8) before switching to a smaller datatype.

@ssheorey Could setting artificially the value of the background (that is currently equal to zero) to the maximum value of the interesting depth will help to compress the depth range and then may allow reduce the depth accuracy in the point cloud ?

(I have to try it ^^)

Just about the accuracy, I clearly need to have a better definition/resolution, just look at this side face 😄 :

It may be a matter of displaying but I don't have this if I save a .ply and display it in an other viewer tool like CloudCompare :

[julienguegan]

Also it is unclear that your depth data should be an uint8, I feels that it is quite restrictive. It says here that 'depth (open3d.geometry.Image) – The input depth image can be either a float image, or a uint16_t image.' ==> If I use float the depth is filled with zero ... Does someone know how to cast it to a uint16_t ? (is it the same as uint16 or a uint8 ?)

[julienguegan]

If someone has a full toy example on a point cloud generation from numpy to open3D, it will also be quite useful to understand all the insights and how to do it correctly ! 👍

[ssheorey]

Also it is unclear that your depth data should be an uint8, I feels that it is quite restrictive. It says here that 'depth (open3d.geometry.Image) – The input depth image can be either a float image, or a uint16_t image.' ==> If I use float the depth is filled with zero ... Does someone know how to cast it to a uint16_t ? (is it the same as uint16 or a uint8 ?)

Open3D typically uses uint16 to represent and process depth data. However any displayed image (RGB or depth) (through either matplotlib or Open3D) can only be 8 bits, so the conversion is needed only for display. You should use uint16 or float for your entire processing.

[julienguegan]

@ssheorey But even when I print the data with the console, it returns zero (if I use uint16) :

So this is why my understanding was that I need to do some conversion and do not work with this data that seems corrupted ...

[ssheorey]

This looks like int overflow. You have to compress the depth range (e.g.: divide by 2^8 if converting from uint16 to uint8) before switching to a smaller datatype.

@ssheorey Could setting artificially the value of the background (that is currently equal to zero) to the maximum value of the interesting depth will help to compress the depth range and then may allow reduce the depth accuracy in the point cloud ?

(I have to try it ^^)

Just about the accuracy, I clearly need to have a better definition/resolution, just look at this side face smile :

It may be a matter of displaying but I don't have this if I save a .ply and display it in an other viewer tool like CloudCompare :

The depth scale (units / mm) is determined by the camera -- check camera docs for best settings for your environment. The background value of zero means invalid data and is used by some algorithms to skip processing it.

The first image actually looks distorted - the nose is too long (?). You are also seeing depth quantization artifacts. Converting to float and smoothing the depth data should help. Some display software may be doing this automatically.

[julienguegan]

@ssheorey For information, I have another technique to generate the point cloud that does not use the o3d.geometry.RGBDImage.create_from_color_and_depth from open3D but use the pyrealsense2 library. With this method, I don't do any data type conversion and I use o3d.utility.Vector3dVector() to be able to visualize result in open3D.

import pyrealsense2 as rs
pc = rs.pointcloud()
pc.map_to(image_frame)
points        = pc.calculate(depth_frame)
verts          = np.asanyarray(points.get_vertices()).view(np.float32).reshape(-1, 3)
pcd.points = o3d.utility.Vector3dVector(verts)
o3d.visualization.draw_geometries([pcd])

Currently, i don't know how to put color to the point cloud with this way of doing so the visualization is not so good ... I put below an example of both methods (top : pyrealsense2 , bottom : open3D). From what I see, it confirm that the quantization is too heavy but I currently don't know how to generate correctly the point cloud with open3D without quantization (does not work with uint16 neither with float32 the depth is set to zero ?) ...