Official implementation of the paper "Can Language Understand Depth?"
Accepted by ACM Multimedia 2022 (Brave New Idea)
Paper Link: https://arxiv.org/abs/2207.01077
PPT: DepthCLIP.pdf
Presentation Video (15min): https://files.atypon.com/acm/37a0a7fe422cdb908bab81f1b2a88743
Authors: Renrui Zhang*, Ziyao Zeng*, Ziyu Guo, Yafeng Li
In this paper, we propose to apply CLIP for zero-shot monocular depth estimation, named DepthCLIP. We found that the patches of input image could respond to a certain semantic distance token and then be projected to a quantified bin to obtain the depth prediction. Without any training, our DepthCLIP surpasses existing unsupervised methods and even approaches the fully-supervised networks.
To our best knowledge, we are the first to conduct zero-shot adaptation from the semantic language knowledge to quantified downstream tasks and perform zero-shot monocular depth estimation. We hope our work could cast a light on the research of bridging semantic vision language knowledge to quantitative tasks.
👆Visualization for monocular depth estimation of our DepthCLIP. Note that we require no training and directly transfer CLIP for zero-shot prediction.
Please refer to ASTransformer to set up environment and prepare NYU Depth V2 Dataset.
Place entire "official_splits" file under ./datasets/NYU_Depth_V2/
Evaluate with our DepthCLIP:
sh tesh.sh
CLIP pre-trained model will be download automatically during testing (if no bug...)
Please see ./monoclip.py for our main model.
——our main contribution lies here, see class: MonoCLIP
Please see ./eval.py for our main function.
——help to understand the pipeline
Please see ./calculate_error.py for evaluation.
——similar to original ASTransformer, not change too much
Please see ./clip/model.py to see our usage of CLIP.
——similar to original CLIP, not change too much
Our code borrows a lot from:
