Happy Pi Day! This project is now in open alpha testing. For the full story, as well as instructions and support resources, please see this video and its description: https://youtu.be/TeWYAbhgaiU
Enjoy!
- Python API
- Blender Bridge
- Save NeRF
- Load NeRF
- Multi-GPU
- Multi-NeRF
- Dataset Preparation Tools
- Downloadable Binaries
Wishlist:
- Drastically improve rendering quality / reduce training loss
Hello NeRF enthusiasts! Here you will find my NeRF rendering and training library. The core principles of this NeRF method are based on the incredible work of Thomas Müller, Alex Evans, Christoph Schied, and Alex Keller, in their paper Instant neural graphics primitives with a multiresolution hash encoding.
Yes, I realize there is already a CUDA implementation, but I wanted to take a crack at reimplementing this myself for the challenge, and for artistic uses such as:
- Spatial distortions
- Multiple NeRFs in one scene
- Multi-GPU capabilities
- Shadertoy-style effects
- Fractals
Since everything here has been written from scratch*, this codebase is permissively licensed and commercial-use-friendly.
(*with generous help from NeRF, nerfstudio, NerfAcc, and completely built on the tiny-cuda-nn backend.)
Enjoy!
-James
https://twitter.com/jperldev
Required toolkits:
Tested Configuration:
Build steps:
git clone git@github.com:JamesPerlman/TurboNeRF --recursive
cd TurboNeRF
cmake . -B build
cmake --build build -j
Until we have an extensible data loader, the test data I'm working with is here:
https://www.dropbox.com/sh/qkt4t1tk1o7pdc6/AAD218LLtAavRZykYl33mO8ia?dl=1
This project has only been tested on that one Lego scene. Real scenes appear to be broken for now.
There is an open issue when using CUDA 12 and PyBind11 (the latter is used by TurboNeRF for the Python module). Currently, a patch manually needs to be applied after checking out the TurboNeRF repository as shown above:
git clone git@github.com:JamesPerlman/TurboNeRF --recursive
cd TurboNeRF
# Apply patch
patch -p1 < ../patches/pybind11-cuda12.patch
# Build as usual
cmake . -B build
cmake --build build -j
Extreme gratitude to open source projects that will allow this project to reach its full potential (in order of integration date):
- CUDA CMake Starter
- tiny-cuda-nn
- JSON for modern C++
- stb
- NeRF
- nerfstudio
- NerfAcc
- torch-ngp
- Nerfies
- glfw
- pybind11
- RenderMan
- glad
- cuda-cmake-github-actions
- pure-torch-ngp
- torch_efficient_distloss
Next-level respect to the researchers much of this codebase is based off. Thank you for making your research public. This would not have been possible without you.
@inproceedings{mildenhall2020nerf,
title={NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis},
author={Ben Mildenhall and Pratul P. Srinivasan and Matthew Tancik and Jonathan T. Barron and Ravi Ramamoorthi and Ren Ng},
year={2020},
booktitle={ECCV},
}
Instant Neural Graphics Primitives with a Multiresolution Hash Encoding
@article{mueller2022instant,
author = {Thomas M\"uller and Alex Evans and Christoph Schied and Alexander Keller},
title = {Instant Neural Graphics Primitives with a Multiresolution Hash Encoding},
journal = {ACM Trans. Graph.},
issue_date = {July 2022},
volume = {41},
number = {4},
month = jul,
year = {2022},
pages = {102:1--102:15},
articleno = {102},
numpages = {15},
url = {https://doi.org/10.1145/3528223.3530127},
doi = {10.1145/3528223.3530127},
publisher = {ACM},
address = {New York, NY, USA}
}
@inproceedings{martinbrualla2020nerfw,
author = {Martin-Brualla, Ricardo
and Radwan, Noha
and Sajjadi, Mehdi S. M.
and Barron, Jonathan T.
and Dosovitskiy, Alexey
and Duckworth, Daniel},
title = {{NeRF in the Wild: Neural Radiance Fields for
Unconstrained Photo Collections}},
booktitle = {CVPR},
year={2021}
}
@article{barron2022mipnerf360,
title={Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields},
author={Jonathan T. Barron and Ben Mildenhall and
Dor Verbin and Pratul P. Srinivasan and Peter Hedman},
journal={CVPR},
year={2022}
}
@software{Muller_tiny-cuda-nn_2021,
author = {Müller, Thomas},
license = {BSD-3-Clause},
month = apr,
title = {{tiny-cuda-nn}},
url = {https://github.com/NVlabs/tiny-cuda-nn},
version = {1.7},
year = {2021}
}
@inproceedings {10.2312:sr.20231122,
booktitle = {Eurographics Symposium on Rendering},
editor = {Ritschel, Tobias and Weidlich, Andrea},
title = {{Floaters No More: Radiance Field Gradient Scaling for Improved Near-Camera Training}},
author = {Philip, Julien and Deschaintre, Valentin},
year = {2023},
publisher = {The Eurographics Association},
ISSN = {1727-3463},
ISBN = {978-3-03868-229-5},
DOI = {10.2312/sr.20231122}
}
Max, Nelson. Optical Models for Direct Volume Rendering. IEEE Transactions on Visualization and Computer Graphics (1995) - https://courses.cs.duke.edu/spring03/cps296.8/papers/max95opticalModelsForDirectVolumeRendering.pdf
Fawzi, A., Balog, M., Huang, A. et al. Discovering faster matrix multiplication algorithms with reinforcement learning. Nature 610, 47–53 (2022) - https://doi.org/10.1038/s41586-022-05172-4
Alman, Josh, and Virginia Vassilevska Williams. A Refined Laser Method and Faster Matrix Multiplication. arXiv, 2020, doi:10.48550/arxiv.2010.05846 - https://arxiv.org/abs/2010.05846
Sabour, Sara, et al. RobustNeRF: Ignoring Distractors with Robust Losses. arXiv, 2023, arXiv:2302.00833 - https://arxiv.org/abs/2302.00833
Extreme thank yous to these subscribers on Twitch (https://twitch.tv/jperldev) who support this project's development!
madclawgonzo - Requested a haiku written by ChatGPT: "Madclawgonzo / Subscribing to your stream / Software project."
anonymous - Requested to remain anonymous
gusround - https://github.com/candidogustavo
slowcon - "uncle slowcon is here with the 4090"
likid_3 - <3
cognitrol - Supporting cool work that helps the community explore technology
dankmatrix - (pending message)
seferidis - (pending message)
memepp - (pending message)
Dakren12 - (pending message)
Relakin - (Confused)
flouwr - (pending message)