How to get points.npy

[longtimenotalk2]

I noticed that points.npy is included in the dataset for background regularization. But it can't be created in the demo of "Process a video into a Nerfie dataset".
So how can I get it?

[keunhong]

Hi, the Colab demo doesn't use the background regularization. It's not a super crucial component of the system, so I would skip it unless you want to compute some quantitative metrics.

If you want to generate it, you'll have to run COLMAP with a background mask or otherwise filter out the pointcloud to only contain background points.

points.npy is super simple and should just contain a (N,3) array of 3D points.

[navuboy]

Is there any option of avoiding the points.npy file?

[keunhong]

Just set

TrainConfig.use_background_loss = False

in the config

[rockywind]

Hi, @keunhong
What is the definition of the coordinate system of points in the points.npy,
and what are the orientations of x, y, and z?

[razaghmanesh]

Hi, @keunhong What is the definition of the coordinate system of points in the points.npy, and what are the orientations of x, y, and z?

I am not sure about NeRfies but in instant ngp, the point cloud was rotated 90 degrees around the x and z axis. It was registered on blender mesh. I have done this on CloudCompare.

[rockywind]

Hi, thanks, @razaghmanesh
Can you share me some link about the point cloud coordination description in instant ngp?

[keunhong]

points.npy are just in world coordinate i.e., the coordinates from COLMAP.

[razaghmanesh]

Hi, thanks, @razaghmanesh Can you share me some link about the point cloud coordination description in instant ngp?

The problem of non-registration is due to the difference between Blender's coordinate system and other coordinate systems such as Opencv, Colmap, etc.
You can search on Google
'how to convert Blender coordinate to open-cv'
Such as : https://stackoverflow.com/questions/64977993/applying-opencv-pose-estimation-to-blender-camera
There are many links to explain the problem.
You should convert Blender coordinates to opencv.

[rockywind]

Hi, @keunhong
I think the definition of the world coordination coordinate system in the points.npy is below.

points.npy are just in world coordinate i.e., the coordinates from COLMAP.

[rockywind]

Hi, @razaghmanesh
Thanks.

The problem of non-registration is due to the difference between Blender's coordinate system and other coordinate systems such as Opencv, Colmap, etc.
You can search on Google
'how to convert Blender coordinate to open-cv'
Such as : https://stackoverflow.com/questions/64977993/applying-opencv-pose-estimation-to-blender-camera
There are many links to explain the problem.
You should convert Blender coordinates to opencv.

[julzzheng]

points.npy are just in world coordinate i.e., the coordinates from COLMAP.

Hi @keunhong, I managed to create the masks using deeplab. Can you explain briefly how I can use these masks to generate the points.npy file using colmap? What steps should I take to create the background points? I hope that you can help me here.

[thegenerativegeneration]

points.npy are just in world coordinate i.e., the coordinates from COLMAP.

Hi @keunhong, I managed to create the masks using deeplab. Can you explain briefly how I can use these masks to generate the points.npy file using colmap? What steps should I take to create the background points? I hope that you can help me here.

I guess you pass the masks to colmap like described here https://colmap.github.io/faq.html#mask-image-regions.
But I'm also unsure how the points.npy comes into existence.

My guess is that you have to read out the points from colmap's points3D.bin.

This could be a good pointer to start from: https://github.com/colmap/colmap/blob/ff8842e7d9e985bd0dd87169f61d5aaeb309ab32/scripts/python/read_write_model.py

[songwei100110]

Through points3D.bin obtained by data processing script of colmap reconstruction. The .bin file is corresponding to the data "./data/hypernerf/virg/vrig-chicken/points3D. npy"?

[julzzheng]

Sorry it's a little late but that's exactly it. I managed it in two steps. First to convert the binary to text using the colmap model converter and then load the text file using numpy and export it as .npy file.