NePO: Neural Point Octrees for Large-scale Novel View Synthesis

Noah Lewis, Darius Rückert, Marc Stamminger, Linus Franke

Abstract: Point-based radiance field rendering produces impressive results for novel-view synthesis tasks. Established methods work with object-centric datasets or room-sized scenes, as computational resources and model capabilities are limited. To overcome this limitation, we introduce neural point octrees (NePOs) to radiance field rendering, which enables optimisation and rendering of large-scale datasets at varying detail levels, including different acquisition modalities, such as camera drones and LiDAR vehicles.

Our method organises input point clouds into an octree from the bottom up, enabling level of detail (LOD) selection during rendering. Appearance descriptors for each point are optimised using the RGB captures, enabling our system to self-refine and address real-world challenges such as capture coverage discrepancies and SLAM pose drift. The refinement is achieved by adaptively densifying octree nodes during training and optimising camera poses via gradient descent. Overall, our approach efficiently optimises scenes with thousands of images and renders scenes containing hundreds of millions of points in real time.

[Paper] [Project Page]

This project is based on ADOP, originally licensed under MIT.

Modifications made by Noah Lewis.

Compile Instructions

• NePO is implemented in C++/CUDA using libTorch.
• A python wrapper for pyTorch is currently not available. Feel free to submit a pull-request on that issue.
• The detailed compile instructions can be found here: src/README.md

NePO Viewer

The nepo_viewer can now be run by passing the path to a scene. It will automatically search for fitting pretrained models in the experiments/ directory. For example:

cd NePO
./build/bin/nepo_viewer --scene_dir scenes/boat

• The working dir of nepo_viewer must be the NePO root directory. This is required because the shaders and experiments are search on relative paths.
• The most important keyboard shortcuts are:
  • F1: Switch to 3DView
  • F2: Switch to neural view
  • F3: Switch to split view (default)
  • WASD: Move camera
  • Center Mouse + Drag: Rotate around camera center
  • Left Mouse + Drag: Rotate around world center
  • Right click in 3DView: Select camera
  • Q: Move camera to selected camera

Scene Description

• NePO uses ADOP's scene format.
• ADOP uses a simple, text-based scene description format.
• To run on your scenes you have to convert them into this format.
• If you have created your scene with COLMAP you can use the colmap2adop converter.
• More infos on this topic can be found here: scenes/README.md

Training NePO

The NePO pipeline is fitted to your scenes by the nepo_train executable. All training parameters are stored in a separate config file. The basic syntax is:

cd NePO
./build/bin/nepo_train --config configs/train_boat.ini

Make again sure that the working directory is the NePO root. Otherwise, the loss models will not be found.

Parameters

In NePO/configs/ you will find the train configurations files that we used to create the pretrained models. We recommend to start with configs/default.ini for your scenes.

Memory Consumption

Our reference training config file was created for a 24GB 4090 GPU. If you run these on a lower-end GPU you will most likely run out of memory. The important config params that control memory consumption are:

# Settings for 40GB A100
# Size in pixels of the random crop during training
train_crop_size = 512
# How many crops are taken per image
inner_batch_size = 4
# How many images are batched together. One batch will have inner_batch_size x batch_size = 16 crops!
batch_size = 4

Citation

@inproceedings{lewis2025nepo,
  title={NePO: Neural Point Octrees for Large-Scale Novel View Synthesis},
  author={Lewis, Noah and R{\"u}ckert, Darius and Stamminger, Marc and Franke, Linus},
  booktitle={Computer Graphics Forum},
  pages={e70287},
  year={2025},
  organization={Wiley Online Library}
}