DreamControl

Implementation of CVPR 2024 paper "DreamControl: Control-Based Text-to-3D Generation with 3D Self-Prior"

DreamControl generates a coarse 3D self-prior and refines texture with control-based distillation, aiming to create 3D content with both high-consistent geometries and high-fidelity textures.

Installation

We build our source code based on original threestudio. See installation.md for additional information.

Qucik Start

We provide two ways to load shape conditions for our control-based score distillation.

# load an obj file as a condition:
python launch.py --config ./configs/dreamcontrol.yaml --train --gpu 0 system.prompt_processor.prompt="{your text prompt}" system.geometry_c.shape_init="mesh:{path of the obj file}"

# load a threestudio checkpoint as a condition:
python launch.py --config ./configs/dreamcontrol.yaml --train --gpu 0 system.prompt_processor.prompt="{your text prompt}" system.geometry_c.shape_init="{path of the last checkpoint file}"

For example, you can use the template geometry we provide in ./load/rot_shapes or pre-process a new shape with correct orientation (Note that, the orientation of original geoemtries in threestudio's ./load/shapes is different from ours):

# rotate the geometry
cd ./load/rot_shapes
python rot.py ../shapes/nascar.obj

# optimize a NeRF with DreamControl
cd ../..
python launch.py --config ./configs/dreamcontrol.yaml --train --gpu 0 system.prompt_processor.prompt="A Tesla Model3 sedan" system.geometry_c.shape_init="mesh:./load/rot_shapes/nascar.obj"

Or, you can use a threestudio checkpoint. We provide an example here. You can download and unzip it, and then:

python launch.py --config ./configs/dreamcontrol.yaml --train --gpu 0 system.prompt_processor.prompt="a corgi" system.geometry_c.shape_init="./a_corgi/ckpts/last.ckpt"

TODO

• Code of Control-Based Score Distribution.
• Code of 3D Self-Prior Generation.

Gallery

"Lionel Messi in a suit, holding the Ballon d'Or"	"Elon musk, using a laptop"	"Batman is riding a moto"	"An astronaut is riding a horse"
"A chimpanzee dressed like Henry VIII king of England"	"Michelangelo style statue of dog reading news on a cellphone"	"A blue jay standing on a large basket of rainbow macarons"	"A baby bunny sitting on top of a stack of pancakes"
"A model of a house in Tudor style"	"Tower Bridge made out of gingerbread and candy"	"A highly detailed sand castle"	"A pavilion in a Chinese garden"
"A gundam"	"Woody from Toy Story"	"Captain America"	"Spider Man"
"A wizard"	"A corgi"	"A Tesla Model3 sedan"	"A classic Packard car"
"A teddy bear"	"A plate piled high with chocolate chip cookies"	"A plate of fried chicken and waffles with maple syrup on them"	"A vase with pink flowers"

Acknowledgement

Our codes are built on threestudio, and ControlLoRA.

Citation

@article{huang2023dreamcontrol,
  title={Dreamcontrol: Control-based text-to-3d generation with 3d self-prior},
  author={Huang, Tianyu and Zeng, Yihan and Zhang, Zhilu and Xu, Wan and Xu, Hang and Xu, Songcen and Lau, Rynson WH and Zuo, Wangmeng},
  journal={arXiv preprint arXiv:2312.06439},
  year={2023}
}