This is the project page of the paper "Geometric Pretraining for Monocular Depth Estimation''.
GeometricPritraining is a pretrain task designed specifically for depth estimation. The pretrain task requires only uncalibrated images from existing datasets or the internet. After the pretraining stage, the backbone network can be transferred to depth estimation tasks. Using unlimited images from the internet, we demonstrate that the geometric-pretrained networks perform better than ImageNet-pretrained networks by a large margin. At the time of the submission, the proposed geometric pretrained networks achieve the new state-of-the-art performance using only existing training methods.
This project page contains:
the implementation of the pretrain task,
the scripts to extract training images from internet videos.
the geometric pretrained networks and the corresponding transferred depth estimation networks.
All components will be open source once the paper is accepted.
2020-10-02: Release some of the pretrained backbones and monodepth weights.
2020-06-22: The open-source procedure is delayed by the unexpected COV-19 situation. We will gradually release models in the following week.
2019-11-02: We pretrained encoder networks using the new dataset and achieves new state-of-the-art results. These pretrained methods will be released shortly.
2019-11-02: We built a new dataset using methods described in the paper that contains only Youtube wild frames. The dataset contrains 151k image pairs for training and 16k images for testing. We are willing to release the extracted frames as a open-source dataset for the community. However, due to the Youtube license, we are afride that the dataset cannot be released without the creators' approvement. We are now contacting the creators.
The proposed method
The proposed method uses a conditional encoder-decoder network to reconstruct the optical flow between two images. With a narrow bottleneck, the encoder network is forced to learn motion-invariant structure information. After the pretraining, the encoder can be transferred to depth estimation using existing methods (e.g. Monodepth2). The system is illustrated in the following figure.
After the pretrain, the transferred network show better accuracy, generalization ability, and few-shot learning.
- The KITTI-trained network tested on KITTI dataset:
GeoPt is geometric pretrained backbone. Click each photo for full resolution.
- The KITTI-trained network tested on CityScapes:
- The KITTI-trained network tested on YouTube videos:
The proposed dataset DrivingVideos
There are countless of images on the internet. In this project, we build the pretrain dataset, DrivingVideos, using videos from the internet. Due to the current size of the pretrain dataset, we mixed the pretraining dataset using KITTI, CityScapes, and DrivingVideos. We are still enlarging the dataset. In the future, we will use only DrivingVideos for the pretraining as experiments show that this leads to the best transferred performance. For more details please check the paper.
A tipical sample from DrivingVideos is a 3-frame sequential image with no calibration informatiom:
The pretrained networks and transferred depth networks
Here, we provide the pretrained networks and transferred depth networks (Monodepth2 is used for the transferring). For details, please check the paper. We provide some of the weights on Dropbox and you can download via the link.
The naming styple follows:
k: KITTI RAW,
d: Drive videos from youtube,
youtube: A large dataset I collected from youtube after the paper submission,
m: The monodepth is trained using only monocular sequences,
ms: The monodepth is trained using both stereo and monocular sequences.
|Model||Layer Num.||Resolution||KITTI||CityScapes||DrivingVideos_small||DrivingVideos_big||Youtube New|
Transferred Networks (evaluation using the code from Monodepth2):
|Backbone||Training Mode||Abs Rel||Sq Rel||RMSE||RMSE log||delta < 1.25|