Skip to content

Official implementation of "Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction"

License

Notifications You must be signed in to change notification settings

lizhaohu/Deformable-3D-Gaussians

 
 

Repository files navigation

Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction

Teaser image

This repository contains the official implementation associated with the paper "Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction".

News

  • 23/11/16, full code and real-time viewer released.
  • 23/11/4, update the computation of LPIPS in metrics.py. Previously, the lpipsPyTorch was unable to execute on CUDA, prompting us to switch to the lpips library (~20x faster).
  • 23/10/25, update real-time viewer on project page. Many, many thanks to @yihua7 for implementing the real-time viewer adapted for Deformable-GS. Also, thanks to @ashawkey for releasing the original GUI.

Dataset

In our paper, we use:

We organize the datasets as follows:

├── data
│   | D-NeRF 
│     ├── hook
│     ├── standup 
│     ├── ...
│   | NeRF-DS
│     ├── as
│     ├── basin
│     ├── ...
│   | HyperNeRF
│     ├── interp
│     ├── misc
│     ├── virg

I have identified an inconsistency in the D-NeRF's Lego dataset. Specifically, the scenes corresponding to the training set differ from those in the test set. This discrepancy can be verified by observing the angle of the flipped Lego shovel. To meaningfully evaluate the performance of our method on this dataset, I recommend using the validation set of the Lego dataset as the test set.

Pipeline

Teaser image

Run

Environment

git clone https://github.com/ingra14m/Deformable-3D-Gaussians --recursive
cd Deformable-3D-Gaussians

conda create -n deformable_gaussian_env python=3.7
conda activate deformable_gaussian_env

# install pytorch
pip install torch==1.13.1+cu116 torchvision==0.14.1+cu116 torchaudio==0.13.1 --extra-index-url https://download.pytorch.org/whl/cu116

# install dependencies
pip install -r requirements.txt

Train

D-NeRF:

python train.py -s path/to/your/d-nerf/dataset -m output/exp-name --eval --is_blender

NeRF-DS/HyperNeRF:

python train.py -s path/to/your/real-world/dataset -m output/exp-name --eval

6DoF Transformation:

We have also implemented the 6DoF transformation of 3D-GS, which may lead to an improvement in metrics but will reduce the speed of training and inference.

# D-NeRF
python train.py -s path/to/your/d-nerf/dataset -m output/exp-name --eval --is_blender --is_6dof

# NeRF-DS & HyperNeRF
python train.py -s path/to/your/real-world/dataset -m output/exp-name --eval --is_6dof

You can also train with the GUI:

python train_gui.py -s path/to/your/dataset -m output/exp-name --eval --is_blender
  • click start to start training, and click stop to stop training.
  • The GUI viewer is still under development, many buttons do not have corresponding functions currently. We plan to :
    • reload checkpoints from the pre-trained model.
    • Complete the functions of the other vacant buttons in the GUI.

Render & Evaluation

python render.py -m output/exp-name
python metrics.py -m output/exp-name

We provide several modes for rendering:

  • render: render all the test images
  • time: time interpolation tasks for D-NeRF dataset
  • all: time and view synthesis tasks for D-NeRF dataset
  • view: view synthesis tasks for real-world dataset
  • original: time and view synthesis tasks for real-world dataset

Results

D-NeRF Dataset

Quantitative Results

Image1

Qualitative Results

Image1 Image1 Image3 Image4

Image5 Image6 Image7 Image8

NeRF-DS Dataset

Image1

See more visualization on our project page.

HyperNeRF Dataset

Since the camera pose in HyperNeRF is less precise compared to NeRF-DS, we use HyperNeRF as a reference for partial visualization and the display of Failure Cases, but do not include it in the calculation of quantitative metrics. The results of the HyperNeRF dataset can be viewed on the project page.

Real-Time Viewer

viewer.mp4

BibTex

@article{yang2023deformable3dgs,
    title={Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction},
    author={Yang, Ziyi and Gao, Xinyu and Zhou, Wen and Jiao, Shaohui and Zhang, Yuqing and Jin, Xiaogang},
    journal={arXiv preprint arXiv:2309.13101},
    year={2023}
}

And thanks to the authors of 3D Gaussians for their excellent code, please consider also cite this repository:

@Article{kerbl3Dgaussians,
      author       = {Kerbl, Bernhard and Kopanas, Georgios and Leimk{\"u}hler, Thomas and Drettakis, George},
      title        = {3D Gaussian Splatting for Real-Time Radiance Field Rendering},
      journal      = {ACM Transactions on Graphics},
      number       = {4},
      volume       = {42},
      month        = {July},
      year         = {2023},
      url          = {https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/}
}

About

Official implementation of "Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction"

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Python 100.0%