tips for training real 360 inward-facing scene

[kwea123]

I tried to train on my own 360 inward-facing scenes, however, there is a huge portion of noise in the output: It doesn't disappear no matter how many steps I train.

I follow the suggestion in the readme and add --no_ndc --spherify --lindisp, the other settings are same as fern.
I suspect that it is due to the fact that input images has many arbitrary backgrounds, which deteriorates the training? For example some of the training images:


These arbitrary backgrounds are inevitable unless I have a infinite ground or an infinitely large table...
What's your opinion on this problem? Is it really due to the background or any other reason?

[bmild]

These arbitrary backgrounds are inevitable unless I have a infinite ground or an infinitely large table...

That actually would be my first suggestion -- we got the best results for 360 scenes when capturing them on the ground, not on a table. Having them on a table where the background changes more between views hurt performance in the examples we tried.

Another problem we had was keeping exposure constant across all captures. This is important for good performance and is trickier with 360 because it's hard to get a 360 capture where lighting doesn't vary dramatically between directions (since usually one direction faces a window and one does not). One of the example views you posted has a huge blown out specular reflection on the table which is probably causing big problems.

360 captures also require a lot more inputs than forward facing since they effectively cover a huge amount of disparity. You probably want at least 80-100 inputs.

[kwea123]

I see. Thanks for your suggestions. I currently only use 32 images...
The best I can get now:

There's a huge noise cloud above her head, but I tuned the viewing angle so that it's not visible.
I will try the methods you mentioned and report back!

[kwea123]

I retook the photos this time on my bed with almost the same white backgrounds and same exposure for all images, I have 65 images, but the noise problem is still there...

The training process doesn't look plausible, in particular, I suspect that the model is overfitting: I validate on only 1 image, which is the closest-to-center one, but the validation loss only goes down a little during the first 5-6 epochs, then it stagnates... or even goes up a little (the training loss goes down constantly, and even reached 32~33PSNR).

I then inspect the inference on a training image, and find the following:

As you can see, the images look almost the same with very high PSNR, however as seen in the depth prediction, it is very noisy even in the regions where the images look the same, so basically it means the model predicts well the final color, but with the wrong composition spatially, is that correct?

What I haven't tried in your suggestion above is to take 80 ~ 100 photos, is that the true solution to get rid of this overfitting/noise problem? I didn't try that because I think it will limit the actual usage... the 65 images I took is already 2 rounds of the object (720 degrees), 80 ~ 100 means one need to rotate at least 3 times around.

[gipeto]

Hi. I am just starting to experiment with 360 scans using this network as well (great work btw). Looking at your results it seems that the depth estimate on the statuette is quite consistent while it falls apart for background pixels.
My guess is that the issue you are having may be due to the very uniform background. I assume that when sampling random rays non intersecting the statuette the rendered color will be some sort of white with high probability, and that almost independently of the actual depth.
What you could try to do is to use some textured cloth on the bed instead of a uniform one in order to give to the system some more photo consistency constraints to respect.
In my little experiments that did help in getting more consistent depths (at least in partial results as I am still training).

[kwea123]

Uniform background(floor) might also be a problem, as you said. But actually what I found is that even a little arbitrary background (like I mentioned in the previous posts) makes training difficult. After deleting those images, the result looks better, even still with some incorrect depth prediction due to the uniform floor.

So what I found good practices:

1. Do not include any image that has arbitrary background (only the object and the floor)
2. The floor needs to have some texture

I will close this issue by posting the final result... still experimenting.

[kwea123]

So this is my result, still some black noise on the background but acceptable:

Do not take any photo that has arbitrary background (only the object and the floor) is my final lesson. Floor texture seems to have only small influence. Number of images seems also irrelevant, as I only have 48 images (1 round) this time.

[SteveSZF]

I tried to train on my own 360 inward-facing scenes, however, there is a huge portion of noise in the output: It doesn't disappear no matter how many steps I train.

I follow the suggestion in the readme and add --no_ndc --spherify --lindisp, the other settings are same as fern.
I suspect that it is due to the fact that input images has many arbitrary backgrounds, which deteriorates the training? For example some of the training images:


These arbitrary backgrounds are inevitable unless I have a infinite ground or an infinitely large table...
What's your opinion on this problem? Is it really due to the background or any other reason?

Hi, thank you for your lessons. I'd like to know how you take photos . Do you use some tools to revolve around the object for capturing photos with uniform distribution of angels? Or you just randomly capture some photos around the object?

[andrewsonga]

@kwea123 Hello, I'm also trying to train NeRF with 360 captures scene (the AIST++ dataset https://aistdancedb.ongaaccel.jp) and I had some questions related to formatting the dataset and setting configurations:

1. Did you format your custom dataset in the "LLFF" or "blender" dataset format?

2. If you used the "LLFF" dataset format and the --spherify flag, then how did you generate the render_poses for 360 degrees free-viewpoint video?

• the render_poses outputed by the spherify method doesn't lie on the same plane as does my 8 cameras do, as shown below

3. When using the --no_ndc flag, how did you set near and far bounds?

• I tried using COLMAP for these, but the dataset I'm trying to use (the AIST++ dataset https://aistdancedb.ongaaccel.jp) only has 8 views and therefore doesn't pickup enough keypoints to do reconstruction
• would it be ok to set near and far as small enough and large enough values respectively?
• can near and far be understood as the closest and furthest distance along the camera-axis (z-axis) where there exists some scene content?

[ynma-hanvo]

i request a feature which takes in the ordinary positions format like T_cam_w, which is common