Depth map and intrinsic

[jzhzhang]

Thanks for your amazing dataset!

I encountered some weird results (as shown below) when planning to back project the depth map to generate the point cloud. The intrinsic matrix is obtained by @liuyuan in issue#4, and the depth map is directly from the car/106_12650_23736/depths/frame000001.jpg.geometric.png. It seems the intrinsic matrix is not related to the depth map.

Can you give me some quick advice or references?

[jzhzhang]

The vaule of the depth map car/106_12650_23736/depths/frame000001.jpg.geometric.png range from [16490-22649], which indicates the camera center is far from the object.

The color map of the depth (scaled by 1e-3):

[davnov134]

Hi, we provide tooling for operating with the depth maps loaded using the Co3DDataset object.
Specifically, this function

co3d/dataset/visualize.py

Line 92 in 8ad0f03

point_cloud = get_rgbd_point_cloud(

contains an example of loading depth maps and converting to point clouds. Can you try to use it to unproject your depth maps? There are few more examples in /tests/test_dataset_visualize.py

[jzhzhang]

Thanks. I will follow the scripts and let you know how it turns out.

[jzhzhang]

@colesbury Sorry to bother you agiain. I want to know some details about the depth maps.

Do the depth maps directly come from the COLMAP? Have you made any post-processing modifications to the depth maps?

[jzhzhang]

I guess i found the resaon why the depth map is not working.

The depth value in the depth maps should be loaded as 16 bit float number. The same as the _load_16bit_depth

co3d/dataset/co3d_dataset.py

Line 728 in b22f145

def _load_16big_png_depth(depth_png):

This is how it looks:

[AlexisStdp]

I guess i found the resaon why the depth map is not working.

The depth value in the depth maps should be loaded as 16 bit float number. The same as the _load_16bit_depth

co3d/dataset/co3d_dataset.py

Line 728 in b22f145

def _load_16big_png_depth(depth_png):

This is how it looks:

Hello, this projection is exactly what I need in my current project. However I didn't fully understand how you obtained it, I tried quite a few provided functions in this repository but it's a bit unclear how to use them. Please, could you tell us what steps you followed and which functions you used to get this nice point cloud?

For example, loading depth as 16 bit had a similar result as your first projection.

[shapovalov]

@AlexisStdp Did you use the provided function to load depth? Please note the files are not a standard 16-bit PNG; the function reinterprets binary 16-bit values as floats.

[AlexisStdp]

@shapovalov Thank you very much for your answer! I indeed used some of the provided functions to load depth, and to load RGB image. And some functions to get the frame_annotations.
Ideally if I could have the following setting it would be perfect: given an RGB image and a depth image (and the .jgz annotations), we load RGB and depth, we get the intrinsic (and maybe scale adjustment), and we backproject to get the partial point cloud.

In the provided code, it seems that the function "get_rgbd_point_cloud" is exactly that, but I'm unfortunately unable to make it work because I'm not sure how to get the argument "camera: CamerasBase" (is there a simple way I could get it?).
There is a function "get_co3d_sequence_pointcloud" which I tried to use in order to get this "camera" (because it itself uses "get_rgdp_point_cloud"), but inside of it the Dataloader didn't work for various reasons.

Is there maybe an alternative approach? - I tried using open3d for example but this gives me a "flat" point cloud (where all points seem to lie on a 2D plane).

Thanks in advance, and happy new year!

[shapovalov]

@AlexisStdp Co3dDataset should load the data in the required format. In particular, frame_data.camera is in PyTorch3D format, which should be compatible with get_rgbd_point_cloud. If that does not work for you, could you share more details?

If you want to dig deeper, this is the code that reads the Cameras object: https://github.com/facebookresearch/co3d/blob/main/dataset/co3d_dataset.py#L490

Note that the camera is in NDC coordinate system. Please refer to PyTorch3D documentation for details: https://pytorch3d.org/docs/cameras