Mesh/object export

[Echolink50]

I did some searching and there was very little information about the quality of 3d mesh you can get from instant ngp. Is it possible to get a decent quality? The few I did see don't really look anything like they do in ngp. I have yet to install this so I haven't tried for myself. I wanted to ask about the capabilities first. Thanks everyone.

[Tom94]

The 3D meshing support is rudimentary and only provided as a proof-of-concept feature. The quality is worse than methods that explicitly focus on providing a good-quality mesh, such as https://nvlabs.github.io/nvdiffrec/ . Cheers!

[Tom94]

Heads-up that nvdiffrec code is now out at https://github.com/NVlabs/nvdiffrec

Sweet. Can we use the point cloud, or images generated by Instant NGP and create a mesh in nvdiffrec?

You don't need instant NGP -- nvdiffrec can directly take images with known camera poses as input. :)

Of course, you can still first run instant-ngp, for example to generate background segmentation masks in larger real-world scenes. These will help isolate objects for better nvdiffrec reconstruction. But I'd recommend using those in conjunction with the original video data / poses, rather than re-rendering color from instant-ngp.

[Echolink50]

Ok, maybe I'm confused about what these codes actually do. I thought instant NGP takes a few images and generates some in between images with very quick training then makes a scene. INGP makes a great scene quickly but can't create a mesh that looks remotely close to the scene. Does nvdiffrec do this also and as quickly? If so then great.

[Tom94]

nvdiffrec's representation of the scene is the mesh, so it has much less opportunity to cheat geometry than a volumetric representation like instant-ngp's NeRF has.

But it trains more slowly than instant-ngp. There's a lot more nuance to the comparison between the two methods (e.g. nvdiffrec learns relightable models), but for this sort of detail I'd recommend reading the paper.

[Camilochiang]

Hei @Tom94 please correct me if i'm wrong (I try to read both papers, i-ngp and nvdiffrec, but this is not my area of expertise, so see my as a simple user):

• i-ngp is generating/rending images for non recorded positions, what can be trained really fast (minutes). We can generate a mesh but i-ngp is not optimized for mesh production, therefore you recommend using nvdiffrec for this. So far you have recommended for point cloud exportation, to extract it from the produced mesh via the vertex positions of the marching cubes mesh. Question: Does that mean that the mesh is been produced after the neural network? if yes, do you go directly from images to mesh? Sorry if my question is weird, but i though that you normally go from pointcloud to mesh
• The generated mesh is colorless. Is there any way to get it with colors?

[Tom94]

Marching cubes is an algorithm that goes from scalars on a grid (in the case of NeRF: density) directly to a mesh. I don't think the points on the grid are necessarily what you're looking for in a point cloud. There are other algorithms that utilize more generic point clouds in intermediate steps, but those aren't implemented in this codebase.

Yes, in instant-ngp the mesh is produced the neural network and is not supervised from images. This is why it is not as good as, for example, nvdiffrec, where the rendered mesh is directly supervised from images.

The generated mesh does have per-vertex colors (but no textures). Unfortunately, many viewers are unable to display the per-vertex colors in the format we output them. But again, these are rather low fidelity as they're produced from the neural network in a rudimentary way.