Does your code work on NERF synthetic dataset ?

[phongnhhn92]

Hi, I have noticed that the original NERF paper have trained and tesed their methods on synthetic dataset (cars,drum,...) so I wonder if your code works on those datasets ?

[yenchenlin]

Hello, for now, the answer is YES and NO. I am able to get it to work on ship, hotdog, and material by copying the config for lego. Results attached:

However, it currently doesn't work for drums. I think we can debug further if the authors released the training configs for all the scenes. Now we only can access configs for lego and fern.

[phongnhhn92]

I didnt run your code on drums but I doubt that images in drums set has transparent background. If you check images from other sets then they are all having white background. Why dont you use this command and convert those drums image with white background :convert image.png -background white -alpha remove -alpha off white.png

I hope it works ^^

[bmild]

I've just added the original configs for each dataset type here:
https://github.com/bmild/nerf/tree/master/paper_configs
We used the same settings across all scenes within one dataset type.

[yenchenlin]

thank you @bmild , I will find times to re-run these experiments.

[yenchenlin]

BTW, @bmild can you comment on what @phongnhhn92 just said? My intuition is that as long as white_bkgd = True is set, no further data pre-processing (e.g., changing images' background) is needed.

Additionally, what's the intuition of no_batching = True for synthetic dataset? My understanding is that batching it should give us better gradients with less variance as each data point in a batch is more independent.

[bmild]

All the blender datasets are stored four channel RGBA images, so the white_bkgd = True flag should be sufficient, yes (it takes care of changing the background).

no_batching here is just a matter of memory consumption, since the batching implementation is very simple, instantiating the camera ray for every pixel in the dataset in one very very large array and then shuffling. In practice we found that our learning rate was small enough that it didn't make a noticeable difference.

[yenchenlin]

@bmild thanks for the answer!

[bmild]

Quick note: if you are seeing problems with the training diverging to only render a white background in the early stages (within 1000 iters), see my comments in this issue.

[yenchenlin]

Hello @bmild , thanks for the update. I tried it and it did solve the "only rendering white background" issue at 1000 iters. Will test it out further.

[yenchenlin]

@phongnhhn92 @bmild following are results trained with 10k iterations, will close the issue.