DreamGaussian4D:
Generative 4D Gaussian Splatting

Jiawei Ren^* Liang Pan^* Jiaxiang Tang Chi Zhang Ang Cao Gang Zeng Ziwei Liu^†

S-Lab, Nanyang Technological University  Shanghai AI Laboratory 
Peking University   University of Michigan  
^*equal contribution
^†corresponding author

Arxiv 2023

examples.mp4

---

[Project Page] • [Paper]

Install

# install customized diffusers
pip install ./diffusers

pip install -r requirements.txt

# a modified gaussian splatting (+ depth, alpha rendering)
git clone --recursive https://github.com/ashawkey/diff-gaussian-rasterization
pip install ./diff-gaussian-rasterization

# simple-knn
pip install ./simple-knn

# nvdiffrast
pip install git+https://github.com/NVlabs/nvdiffrast/

# kiuikit
pip install git+https://github.com/ashawkey/kiuikit

Tested on:

• python 3.8 & torch 2.1 & CUDA 11.8 on an 80GB A100.
  • Stage I: 2min. Stage II: 4.5min. Stage III: 3.5min.
• python 3.8 & torch 2.0 & CUDA 11.6 on an 24GB A5000.
  • Stage I: 4.5min. Stage II: 9min. Stage III: 5.5min with oom_hack.

Usage

# generate driving video
python gen_vid.py --name anya_rgba --seed 42 --bg white

# Stage I: train 500 iters (~2min) and export ckpt & coarse_mesh to logs
python main.py --config configs/image.yaml input=data/anya_rgba.png save_path=anya

# Stage II: temporal optimization stage (export meshes by default)
python main_4d.py --config configs/4d.yaml input=data/anya_rgba.png save_path=anya

# Stage III: texture optimization (optional, it requires large GPU memory and we are optimzing it)
python main2_4d.py --config configs/4d_svd.yaml input=data/anya_rgba.png save_path=anya

# to turn on viser GUI, add `gui=True`, e.g.:
python main.py --config configs/image.yaml input=data/anya_rgba.png save_path=anya gui=True

Meshes will be automatically exported to logs in Stage II. Visulizations will be saved to vis_data.

Load exported meshes in Blender

• Install the Stop-motion-OBJ add-on
• File -> Import -> Mesh Sequence
• Go to logs directory, type in the file name (e.g., 'anya'), and tick Material per Frame.

blender_lowfps.mov

Tips

• CUDA OOM.

  • Stage I: Reduce batch_size to 4.
  • Stage II: Reduce n_views to 1.
  • Stage III: Add argument oom_hack=True or disable SVD by setting lambda_svd to 0.

• Black video after running gen_vid.py.

  • Make sure pytorch version is >=2.0

Acknowledgement

This work is built on many amazing research works and open-source projects, thanks a lot to all the authors for sharing!

• 4DGaussians
• DreamGaussian
• gaussian-splatting and diff-gaussian-rasterization
• threestudio
• nvdiffrast

Citation

@article{ren2023dreamgaussian4d,
  title={DreamGaussian4D: Generative 4D Gaussian Splatting},
  author={Ren, Jiawei and Pan, Liang and Tang, Jiaxiang and Zhang, Chi and Cao, Ang and Zeng, Gang and Liu, Ziwei},
  journal={arXiv preprint arXiv:2312.17142},
  year={2023}
}