A neural radiance field (NeRF) is a fully-connected neural network that can generate novel views of complex 3D scenes, based on a partial set of 2D images. It is trained to use a rendering loss to reproduce input views of a scene. It works by taking input images representing a scene and interpolating between them to render one complete scene. NeRF is a highly effective way to generate images for synthetic data.
A NeRF network is trained to map directly from viewing direction and spatial location (5D input) to opacity and color (4D output), using volume rendering to render new views. NeRF is a computationally-intensive algorithm, and processing of complex scenes can take hours or days. However, new algorithms are available that dramatically improve performance.
pip install tqdm==4.64.0 configargparse==1.5.3 imageio==2.19.3 opencv-python==4.6.0.66
Lego:
Fern:
Lego | Speed | PSNR ↑ | SSIM ↑ | LPIPS ↓ |
---|---|---|---|---|
Coarse: 192 pts/ray (torch converted ckpt) | 13.37s/img | 30.52 | 0.9565 | 0.032 |
Coarse: 192 pts/ray (Train from scratch) | 13.37s/img | 28.16 | 0.9233 | 0.055 |
Coarse: 64 pts/ray + Fine: 128 pts/ray (torch converted ckpt) | 1305.28s/img | N/A | N/A | N/A |
Coarse: 64 pts/ray + Fine: 128 pts/ray (Train from scratch) | 1305.28s/img | N/A | N/A | N/A |
Paper: 64 pts/ray + Fine: 128 pts/ray | 4.69s/img | 32.54 | 0.961 | 0.050 |
Fern | Speed | PSNR ↑ | SSIM ↑ | LPIPS ↓ |
---|---|---|---|---|
Coarse: 192 pts/ray (torch converted ckpt) | 13.37s/img | 26.06 | 0.8305 | 0.095 |
Coarse: 192 pts/ray (Train from scratch) | 13.37s/img | 25.43 | 0.7963 | 0.121 |
Coarse: 64 pts/ray + Fine: 128 pts/ray (torch converted ckpt) | 1305.28s/img | N/A | N/A | N/A |
Coarse: 64 pts/ray + Fine: 128 pts/ray (Train from scratch) | 1305.28s/img | N/A | N/A | N/A |
Paper: 64 pts/ray + Fine: 128 pts/ray | 4.69s/img | 25.17 | 0.792 | 0.280 |
Note: the significant speed drop comes from the unsupported N-dim ms.numpy.searchsorted
that instead implemented with a for loop. Related issue.
Please check out the config script.
Here, how to use NeRF model will be introduced as following.
At first, you should download dataset by yourself. NeRF Synthetics.
After you get the dataset, make sure your path is sturctured as following:
.datasets/
└── nerf_synthetics
└── lego
├── test [600 entries exceeds filelimit, not opening dir]
├── train [100 entries exceeds filelimit, not opening dir]
├── transforms_test.json
├── transforms_train.json
├── transforms_val.json
└── val [100 entries exceeds filelimit, not opening dir]
and modify the datadir
in the config file accordingly.
python {train,eval}.py [data_related_options] \
[--ckpt /path/to/checkpoint] [--no_reload] [--gpu 0] \
[--render_test]
# `--no_reload`: not resume from last run.
# `--gpu`: the used gpu.
# `--ckpt`: specify the path of a checkpoint (from trained or downloaded).
# `--render_test`: render test images to perform quantitative comparison.
python {train,eval}.py --name fern_coarse --data_dir ../datasets/nerf_llff/fern/ --dataset_type llff --factor 8 --llff_hold 8 --cap_n_rand 4096 --cap_n_samples 192 --cap_n_importance 0 --use_view_dirs --raw_noise_std 1e0 [--ckpt /path/to/checkpoint]
Note: the significant speed drop comes from the unsupported N-dim ms.numpy.searchsorted
that instead implemented with a for loop. Related issue.
python {train,eval}.py --name fern --data_dir ../datasets/nerf_llff/fern/ --dataset_type llff --factor 8 --llff_hold 8 --cap_n_rand 4096 --cap_n_samples 64 --cap_n_importance 128 --use_view_dirs --raw_noise_std 1e0 [--ckpt /path/to/checkpoint]
python {train,eval}.py --name lego_coarse --data_dir ../datasets/nerf_synthetics/lego --dataset_type blender --half_res --cap_n_rand 4096 --cap_n_samples 192 --cap_n_importance 0 --use_view_dirs --raw_noise_std 1e0 --white_bkgd [--ckpt /path/to/checkpoint]
Note: the significant speed drop comes from the unsupported N-dim ms.numpy.searchsorted
that instead implemented with a for loop. Related issue.
python {train,eval}.py --name lego --data_dir ../datasets/nerf_synthetics/lego --dataset_type blender --half_res --cap_n_rand 4096 --cap_n_samples 64 --cap_n_importance 128 --use_view_dirs --raw_noise_std 1e0 --white_bkgd [--ckpt /path/to/checkpoint]