Research Paper Analysis

Purpose

Here will be the analysis's of potential papers to integrate into the backend of our project. Since the focus of this project is to get NeRF working in a web app, the most relevant factors are open license, computational resources, robustness to real world data. Breaking this down further these are the most important qualities ranked:

1. Source code is available and fully open source

• Open source licenses are licenses that comply with the Open Source Definition

2. Speed and memory use in training should be low enough to allow for ease of development

• Sub hour NeRF training time
• Less then 8GB memory consumption
• Not reliant on custom kernels or specific GPU models

3. Inference speed should be fast enough to allow for video rendering

• Inference of at least 1 fps

4. Robustness to real world data

• Quality should be on par or better then the original NeRF paper
• Should work with SFM techniques
• Insensitive to depth inaccuracies or image artifacts

List of potential project papers

• MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo
• FastNeRF: High-Fidelity Neural Rendering at 200FPS
• SqueezeNeRF: Further factorized FastNeRF for memory-efficient inference
• EfficientNeRF: Efficient Neural Radiance Field
• Fast Dynamic Radiance Fields with Time-Aware Neural Voxels
• TensoRF: Tensorial Radiance Fields

Paper Analysis

Note: some of the excerpts and descriptions are copied directly from the paper linked in each section.

• PlenOctrees
  Desc: For real-time rendering of Neural Radiance Fields they train NeRFs to predict a spherical harmonic representation of radiance, removing the viewing direction as an input to the neural network. This representation can be directly optimized on and allows for slightly faster convergence.

  • Pros: This technique makes in browser rendering possible
  • Cons: Long training time
  • Memory: 2GB quantized to 34-125MB
  • Training: 1.2 hours finetuning + 6 hours traditional method
  • Inference: Real-time 200+ FPS

• BARF : Bundle-Adjusting Neural Radiance Fields
  Desc: BARF can effectively optimize the neural scene representations and resolve large camera pose misalignment at the same time. A modified positional encoding method along with a structured learning process of course to fine alignment allows the camera poses to be estimated at training time.

  • Pros: No accurate camera position needed. Better view synthesis on real world data. Compatible with other papers.
  • Cons: No improvements to speed
  • Memory: same as original NeRF
  • Training: same as original NeRF
  • Inference: same as original NeRF

• Instant Neural Graphics Primitives with a Multiresolution Hash Encoding
  Desc: A versatile new input encoding that permits the use of a smaller network without sacrificing quality, thus significantly reducing the number of floating point and memory access operations: a small neural network is augmented by a multi resolution hash table of trainable feature vectors whose values are optimized through stochastic gradient descent. The positional encoding stores trainable vectors stored by a hash table.

  • Pros: Extremely fast, moderate memory consumption, compatible with other papers, No loss in quality
  • Cons: Closed source, Results from custom kernel
  • Memory: 8GB (variable)
  • Training: Seconds!
  • Inference: miliseconds!

• Plenoxels - Radiance Fields without Neural Networks

• TensoRF Tensorial Radiance Fields
  Desc: Instead of using a MLP this paper repesents the radiance field of a scene with a 4D tensor. This tensor is then factorized into multiple compact low-rank tensors which optimization can be directly ran on with the image inputs.

  • Pros: Open source, Fast, better quality, low memory consumption, compatible with ngp-nerf, no custom kernal
  • Cons: No machine learning, Potentially difficult extension, method currently only supports bounded scenes with a single bounding box and cannot handle unbounded scenes with both foreground and background content
  • Memory: < 75 MB
  • Training: < 10-30 min
  • Inference: unknown (probably fast?)

• MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo

  • Pros: Trainable for multiple scenes, better then Pixel NeRF in quality
  • Cons: Worse then Nerf in quality
  • Memory: (all tests below done on 2080Ti)
  • Training: 15 min decent results (sometimes as slow as NeRF like with 360)
  • Inference: unknown (likely same as NeRF)

• EfficientNeRF: Efficient Neural Radiance Fields

  • Training: few hours (sub hour if combined with MSVNerf)
  • Inference: Real-time 200+ FPS

• Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction Desc: Replace NeRF with voxel grid

  • Pros: At or above NeRF view synthesis
  • Cons:
  • Memory: (all tests below done on 2080Ti)
  • Training: < 15 min
  • Inference: near 1 fps