SweetDreamer: Aligning Geometric Priors in 2D Diffusion for Consistent Text-to-3D (ICLR 2024)

Weiyu Li, Rui Chen, Xuelin Chen, Ping Tan

Project Page | ArXiv | Paper | Video

All Code and Ckpt will be released in the next few days, sorry for the delay due to some to some permission issues :( 🏗️ 🚧 🔨

Inportant: This repo. is under contrustruction. Pre-trained models are not provided yet, we will provide them in the next few days.

• Release the reorganized code
• Release the pretrained model (tiny-version)
• Release the full model

Prerequisite

Setup environment (Install threestudio)

This part is the same as original threestudio. Skip it if you already have installed the environment.

See installation.md for additional information, including installation via Docker.

• You must have an NVIDIA graphics card with at least 20GB VRAM and have CUDA installed.
• Install Python >= 3.8.
• (Optional, Recommended) Create a virtual environment:

python3 -m virtualenv venv
. venv/bin/activate

# Newer pip versions, e.g. pip-23.x, can be much faster than old versions, e.g. pip-20.x.
# For instance, it caches the wheels of git packages to avoid unnecessarily rebuilding them later.
python3 -m pip install --upgrade pip

• Install PyTorch >= 1.12. We have tested on torch1.12.1+cu113 and torch2.0.0+cu118, but other versions should also work fine.

# torch1.12.1+cu113
pip install torch==1.12.1+cu113 torchvision==0.13.1+cu113 --extra-index-url https://download.pytorch.org/whl/cu113
# or torch2.0.0+cu118
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu118

• (Optional, Recommended) Install ninja to speed up the compilation of CUDA extensions:

pip install ninja

• Install dependencies:

pip install -r requirements.txt

Download the pretrained CCM model(TBD)

sh download.sh

Quick demo

python launch.py --config configs/sweetdreamer-stage1.yaml --train --gpu 0 \
                 system.prompt_processor.prompt="Albert Einstein with grey suit is riding a bicycle" \
                 system.cmm_prompt_processor.prompt="Albert Einstein with grey suit is riding a bicycle" \
                 tag=einstein

python launch.py --config configs/sweetdreamer-stage2.yaml --train --gpu 0 \
                 system.prompt_processor.prompt="Albert Einstein with grey suit is riding a bicycle" \
                 system.cmm_prompt_processor.prompt="Albert Einstein with grey suit is riding a bicycle" \
                 tag=einstein

Acknowledgement

This code is built on the amazing open-source projects:

• threestudio-project
• diffusers
• stable-diffusion
• deep-floyed

We also thank Jianxiong Pan and Feipeng Tian for the help of the data and GPU server.

Citation

If you find our work useful for your research, please consider citing using the following BibTeX entry.

@article{sweetdreamer,
  author    = {Weiyu Li and Rui Chen and Xuelin Chen and Ping Tan},
  title     = {SweetDreamer: Aligning Geometric Priors in 2D Diffusion for Consistent Text-to-3D},
  journal   = {arxiv:2310.02596},
  year      = {2023},
}