DepthCrafter: Generating Consistent Long Depth Sequences for Open-world Videos

   

<<<<<<< HEAD ...

   

Wenbo Hu^{1* †}, Xiangjun Gao^2*, Xiaoyu Li^{1* †}, Sijie Zhao¹, Xiaodong Cun¹,
Yong Zhang¹, Long Quan², Ying Shan<sup>3, 1

1</sup>Tencent AI Lab ²The Hong Kong University of Science and Technology ³ARC Lab, Tencent PCG

arXiv preprint, 2024

🔆 Introduction

🤗 DepthCrafter can generate temporally consistent long depth sequences with fine-grained details for open-world videos, without requiring additional information such as camera poses or optical flow.

🎥 Visualization

We provide some demos of unprojected point cloud sequences, with reference RGB and estimated depth videos. Please refer to our project page for more details.

365030500-ff625ffe-93ab-4b58-a62a-50bf75c89a92.mov

parent of 6e06eb5 (update)

🚀 Quick Start

🛠️ Installation

1. Clone this repo:

   git clone https://github.com/Tencent/DepthCrafter.git

2. Install dependencies (please refer to requirements.txt):

   pip install -r requirements.txt

🤗 Model Zoo

DepthCrafter is available in the Hugging Face Model Hub.

🏃‍♂️ Inference

1. Using Image Sequence, requires appropriate GPU memory based on resolution:

• Full inference (~0.6 fps on A100, recommended for high-quality results):

  python run.py --image-folder examples/images_example_01

• Fast inference through 4-step denoising and without classifier-free guidance (~2.3 fps on A100):

  python run.py --image-folder examples/images_example_01 --num-inference-steps 4 --guidance-scale 1.0

🤖 Gradio Demo

We provide a local Gradio demo for DepthCrafter, which can be launched by running:

gradio app.py

🤝 Contributing

• Welcome to open issues and pull requests.
• Welcome to optimize the inference speed and memory usage, e.g., through model quantization, distillation, or other acceleration techniques.