instruct-nerf2nerf-gaussian-splatting

An attempt to see if the instruct-nerf2nerf idea works with 3D Gaussian Splatting.

Set up

Use conda environment for local development, and docker to build the Python wheels / run colmap. Benefits:

Can still use neovim for development and do not require remote-container support
Only need to install nvidia-driver, do not need to install CUDA in the system path

Credits go to authors of instruct-nerf2nerf and gaussian-splatting. I mostly ported and edited from their codebase.

Download bear.zip from this google drive. And put it into data/ folder. Use colmap to get camera pose.

You can refer to run_colmap.sh for colmap commands.

Go to gaussian_splatting folder
Run python train.py -s ../data/bear (where you put the data). It makes sure you can run gaussian splatting correctly
Run python train_in2n.py -s ../data/bear. It will run gaussian splatting, and edit the image using instruct-pix2pix every 20 iterations.

Ground Truth

Rendered

Some findings:

A good instruct-pix2pix model is very important. During the optimization stage, I can see some edited images are very off..
The colors of tree leaves get brighter. Need more targeted edits in the diffusion model.
The idea of instruct-nerf2nerf is very cool!! I am surprised by the final results. Despite rendering at some angles can still generate weird images, it overall regularizes the editing pretty well.

Name		Name	Last commit message	Last commit date
Latest commit History 16 Commits
.devcontainer		.devcontainer
data		data
gaussian_splatting		gaussian_splatting
gifs		gifs
third_party		third_party
weights		weights
.clangd		.clangd
.gitignore		.gitignore
.gitmodules		.gitmodules
LICENSE		LICENSE
Makefile		Makefile
README.md		README.md
downsize.py		downsize.py
environment.yaml		environment.yaml
make_gif.py		make_gif.py
requirements.txt		requirements.txt
run_colmap.sh		run_colmap.sh
run_hf_pix2pix.py		run_hf_pix2pix.py