-
Notifications
You must be signed in to change notification settings - Fork 35
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Extracted fern mesh with appearance #18
Comments
Hi @RuiYangQuan, I've checked the model on my regular laptop and it seems there is a problem with the parameters (most likely). Will check throughly next week when getting back to my workstation. |
Okay, thank you for your reply. If you find anything new, please let me know immediately. @DomainFlag |
Hi @RuiYangQuan, I did some checking, and it seems that the uploaded model has some issues (either was not properly uploaded or alike). I don't have past models locally, thus, I will try re-do the training and slightly make the project up-to-date. Will try complete it by the end of this week, as I am running other experiments for other projects as well. Thank you a lot for submitting the issue. |
Hi,i wonder like to know if this problem has been solved.Could you please provide the latest pre-training model. |
Hi @RuiYangQuan, I've actually did some testing, and there wasn't an issue with the model. The actual problem is the data which is used for reconstructing the scene, or I would better say the limitations of the model itself when trained on that: If you take LLFF as an example, the poses from which the photos were taken are not spherically uniformly distributed as in the Synthetic dataset. That results in unresolved scene mass which is not any useful for scene reconstruction if applying Marching Cubes algorithm. For that case we have Also note: the BuFF model is suitable for sparse scenes and closed ones, because the volumetric encoded scene will be sparse as well, thus that's where the gain comes from. I did some changes, and trained a few more models on (Fern, Room) and did some changes to |
Hi, @DomainFlag Notes: Thanks much |
I will try to clear things up with the following comment, as it seems there is some misunderstanding although the comment will require some additional re-editing. Feel free to jump to the solution part if needed, however, it might not be the one based on your requirements. ProblemNeRF and all its equivalents have no prior knowledge to work with modeling objects in the scene (not to say dealing with scene understanding or other variants alike). It's like asking someone who hasn't seen a chair in his whole life to model it by only having the front view of it with the other 2 legs occluded. Of course, a model like NeRF won't be able to model it properly and you'll end up with a chair with missing legs at best. That's the reason you can't extract a mesh from the examples above, it's just there is no prior knowledge to work with, although it can be done something (some great progress is done recently for generative methods in 2D). However, that won't be called NeRF anymore (most likely). It's important to set the expectations right about the mesh to be extracted - "what do we expect based on what is given". As an example, I've added one such example (Room from LLFF dataset), just to clear up the confusion. If having insufficient imagery, you won't even get the sort-of billboards (e.g., back of the seats) that you can find in 3D game development as compensation but some sort of interpolated mess instead. Additionally, the model is overly reliant on view dependence (no 360deg imagery available, all shots are done from the front of the room) from the train set as it fails to recover the back of the scene. No such information is available in the training data thus you end up with poor local generalization. Even so, a slight perturbation applied to the GT ray origins can drastically affect the results. Note, that may not happen if considering all the mentioned points at the bottom's solution. The model isn't able to handle such constraints, unfortunately, thus you end up with ill-defined parametrization of the space (density) which is not any good for something like Marching Cubes. A single reference image - Room from LLFF Dataset (Link) Pre-trained model PSNR (34.65) on test set: A few renders:
Ray-casting based on poses uniformly sampled across a sphere (360deg):
Ray-casting based on the poses from the LLFF dataset (Colmap):
What can I do then? (Solution)It's possible to model only under certain scenarios that may be tedious to accomplish or sometimes may even be impossible. But it can be worth the try. Examples of objects that might be feasible to model:
A few things have to be considered:
|
hi,thank you very much for your work, but I have encountered a problem here, when I generate the mesh of fern, I get this kind of graph, what is this situation, I would appreciate your advic!!!
![3](https://user-images.githubusercontent.com/90050796/182093836-2ce55a39-fc99-425d-9181-1f995ad2490f.png)
e
The text was updated successfully, but these errors were encountered: