Extracted fern mesh with appearance

[RuiYangQuan]

hi,thank you very much for your work, but I have encountered a problem here, when I generate the mesh of fern, I get this kind of graph, what is this situation, I would appreciate your advic!!!

e

[DomainFlag]

Hi @RuiYangQuan, I've checked the model on my regular laptop and it seems there is a problem with the parameters (most likely). Will check throughly next week when getting back to my workstation.

[RuiYangQuan]

Okay, thank you for your reply. If you find anything new, please let me know immediately. @DomainFlag

[DomainFlag]

Hi @RuiYangQuan, I did some checking, and it seems that the uploaded model has some issues (either was not properly uploaded or alike). I don't have past models locally, thus, I will try re-do the training and slightly make the project up-to-date. Will try complete it by the end of this week, as I am running other experiments for other projects as well. Thank you a lot for submitting the issue.

[RuiYangQuan]

@DomainFlag

Hi,i wonder like to know if this problem has been solved.Could you please provide the latest pre-training model.

[DomainFlag]

Hi @RuiYangQuan, I've actually did some testing, and there wasn't an issue with the model.

The actual problem is the data which is used for reconstructing the scene, or I would better say the limitations of the model itself when trained on that:

If you take LLFF as an example, the poses from which the photos were taken are not spherically uniformly distributed as in the Synthetic dataset. That results in unresolved scene mass which is not any useful for scene reconstruction if applying Marching Cubes algorithm. For that case we have mesh_surface_ray.py which does not solely rely on the scene density inferred from the trained parametric function, but uses ray-casting and triangulation on the fly. But, we didn't have enough time to fully port it for production use-cases.

Also note: the BuFF model is suitable for sparse scenes and closed ones, because the volumetric encoded scene will be sparse as well, thus that's where the gain comes from.

I did some changes, and trained a few more models on (Fern, Room) and did some changes to mesh_surface_ray.py but will need more time. I will push this weekend whatever is workable but don't expect a full solution as I am jointly working on another paper that has a deadline soon.

[wjlszhang]

Hi, @DomainFlag
Have you solved the issue? I'm not sure it is cause by the same reason.
When I use mesh_nerf.py to generate mesh.obj, it usually takes long time and the size of mesh.obj is purly large, i.e. around 20G

Notes:
Firstly I use colmap to generate my own dataset as suggested, and it is dental related data, one of the pictures is as bellow

And then use rain_nerf.py to train the model, the result looks good

Last, use mesh_nerf.py to generate mesh obj, as blow
python src/mesh_nerf.py --log-checkpoint logs/nerf-colmap-fern/default/version_18 --checkpoint model_last.ckpt --save-dir ./meshdata --limit 1.2 --res 480 --iso-level 32 --view-disparity-max-bound 1e0

Thanks much

[DomainFlag]

@RuiYangQuan, @wjlszhang

I will try to clear things up with the following comment, as it seems there is some misunderstanding although the comment will require some additional re-editing. Feel free to jump to the solution part if needed, however, it might not be the one based on your requirements.

Problem

NeRF and all its equivalents have no prior knowledge to work with modeling objects in the scene (not to say dealing with scene understanding or other variants alike). It's like asking someone who hasn't seen a chair in his whole life to model it by only having the front view of it with the other 2 legs occluded. Of course, a model like NeRF won't be able to model it properly and you'll end up with a chair with missing legs at best. That's the reason you can't extract a mesh from the examples above, it's just there is no prior knowledge to work with, although it can be done something (some great progress is done recently for generative methods in 2D). However, that won't be called NeRF anymore (most likely).

It's important to set the expectations right about the mesh to be extracted - "what do we expect based on what is given". As an example, I've added one such example (Room from LLFF dataset), just to clear up the confusion. If having insufficient imagery, you won't even get the sort-of billboards (e.g., back of the seats) that you can find in 3D game development as compensation but some sort of interpolated mess instead. Additionally, the model is overly reliant on view dependence (no 360deg imagery available, all shots are done from the front of the room) from the train set as it fails to recover the back of the scene. No such information is available in the training data thus you end up with poor local generalization. Even so, a slight perturbation applied to the GT ray origins can drastically affect the results. Note, that may not happen if considering all the mentioned points at the bottom's solution. The model isn't able to handle such constraints, unfortunately, thus you end up with ill-defined parametrization of the space (density) which is not any good for something like Marching Cubes.

A single reference image - Room from LLFF Dataset (Link)

Pre-trained model PSNR (34.65) on test set:

A few renders:

Example 1	Example 2	Example 3

Ray-casting based on poses uniformly sampled across a sphere (360deg):

Example 1	Example 2

Ray-casting based on the poses from the LLFF dataset (Colmap):

Example 1	Example 2	Example 3

What can I do then? (Solution)

It's possible to model only under certain scenarios that may be tedious to accomplish or sometimes may even be impossible. But it can be worth the try.

Examples of objects that might be feasible to model:

1. Small: Figurine on a table, pendant light, apple hanging on a tree.
2. Large: Building, house, monument, stop sign.

A few things have to be considered:

• Minimal occlusion (e.g., teacup handle) to none is preferable.

• If something is occluding the object (e.g., a few parts) and you can't do anything. It's worth trying a few things. (e.g., inpainting)

• The generated imagery needs to be rich. (i.e., resolve the whole target object) Use image segmentation (U-Net, Photoshop) and mask everything except the target object - green fill in the referenced image (e.g., air, base). The objective is to have such data such that the object hangs in a void no matter how you look at it such that the model can parametrize all the space around the object as negative density.
  

• For very large objects, you should have a drone or something similar, which may work only if you're able to achieve accurate pose estimation.

• Objects are not deformable (e.g., humans moving). Although there are a few existing solutions that handle it.

• Have at least 30 shots (from different angles), more is better.