RI3D Implementation

Implementation of RI3D - state of the art few-shot gaussian splatting.

Makes a gaussian splat from very few images (down to 3). Existing solutions (COLMAP + postshot) look terrible with few cameras - holes, floaters, and they're picky about camera placement. COLMAP needs ~12 cameras minimum for a full body scan and still produces results that need manual cleanup.

RI3D cleans up the floaters and inpaints holes with fine tuned stable diffusion models automatically.

---

Since we are just fine tuning models, you can train it yourself on a consumer level gpu. Works on my RTX 2060 Super with 8GB VRAM. Only tested on Cuda GPUs, may work on others too tho.

Dataset

I used the mip-nerf 360 dataset for training.

wget http://storage.googleapis.com/gresearch/refraw360/360_v2.zip -O 360_v2.zip
./utils/extract_dataset.sh

Downloads & extracts only the images from each scene in 360_v2 into dataset/. Requires p7zip (7z command).

Install

Install Python dependencies:

pip install -r requirements.txt

Then install DUSt3R by following its README. Once it's set up, move its source folders into this repo so the following paths exist:

• src/dust3r_visloc
• ./dust3r
• ./croco

Usage

Make a new folder with your scene name in dataset. Then place your three images there. Then run the following:

python src/run_pipeline.py --prep --scene dataset/your-scene 
python src/run_pipeline.py --train --scene dataset/your-scene
python src/run_pipeline.py --optimize --scene dataset/your-scene

Note: you can omit --scene and it will run steps across all scenes it finds (in a memory efficient way)

--train runs all three training stages (LOO data, repair model, inpainting model). If you want to run only some of them, use the granular flags — they compose:

python src/run_pipeline.py --prep --scene dataset/your-scene      # LOO pairs only
python src/run_pipeline.py --train_loo --scene dataset/your-scene      # LOO pairs only
python src/run_pipeline.py --train_repair --scene dataset/your-scene   # repair model only (needs LOO pairs)
python src/run_pipeline.py --train_inpaint --scene dataset/your-scene  # inpainting model only
python src/run_pipeline.py --train_loo --train_repair --scene dataset/your-scene  # any combination

This takes quite a while. with 3 input views and RTX 2060S, 30-40min per scene. This can surely be improved a lot and im working on that while also getting the quality up at the same time. So its eventually usable for gsplat video :D