SC-SfMLearner

This codebase implements the system described in the paper:

Unsupervised Scale-consistent Depth and Ego-motion Learning from Monocular Video

Jia-Wang Bian, Zhichao Li, Naiyan Wang, Huangying Zhan, Chunhua Shen, Ming-Ming Cheng, Ian Reid

NeurIPS 2019

See our [paper] and [project webpage] for more details.

Video demo of dense reconstruction using estimated depth

Highlighted Features

1. A geometry consistency loss for enforcing the scale-consistency of predictions between consecutive frames.
2. A self-discovered mask for detecting moving objects and occlusions.
3. Enabling the unsupervised estimator (learned from monocular videos) to do visual odometry on a long video.

Preamble

This codebase was developed and tested with python 3.6, Pytorch 1.0.1, and CUDA 10.0 on Ubuntu 16.04. It is based on Clement Pinard's SfMLearner implementation, in which we make little modification and add our proposed losses.

Updates

Note that this is an updated and improved version, see the original version in 'Release / NeurIPS Version' for reproducing results in paper. Compared with NerIPS version, we (1) Change networks by using Resnet18 and Resnet50 pretrained model for encoder. (2) We add 'auto_mask' by Monodepth2.

Prerequisite

pip3 install -r requirements.txt

or install manually the following packages :

torch >= 1.0.1
imageio
matplotlib
scipy
argparse
tensorboardX
blessings
progressbar2
path.py
evo

It is also advised to have python3 bindings for opencv for tensorboard visualizations

Preparing training data

See "scripts/run_prepare_data.sh" for examples, including KITTI Raw, Cityscapes, and KITTI Odometry.

For KITTI Raw dataset, download the dataset using this script provided on the official website.

For Cityscapes, download the following packages: 1) leftImg8bit_sequence_trainvaltest.zip, 2) camera_trainvaltest.zip. You will probably need to contact the administrators to be able to get it.

For KITTI Odometry dataset download the dataset with color images.

Training

The "scripts" folder provides several examples for training and testing.

You can train the depth model on KITTI Raw by running

sh scripts/train_resnet18_depth_256.sh

or train the pose model on KITTI Odometry by running

sh scripts/train_resnet50_pose_256.sh

Then you can start a tensorboard session in this folder by

tensorboard --logdir=checkpoints/

and visualize the training progress by opening https://localhost:6006 on your browser.

Evaluation

You can evaluate depth using Eigen's split by running

sh scripts/test_kitti_depth.sh

and test visual odometry by running

sh scripts/test_kitti_vo.sh

You can evaluate visual odometry results using python code at this repo

Besides, you can evaluate 5-frame pose as SfMLearner by running

sh scripts/test_kitti_vo.sh

Pretrained Models

Avalaible here

Both NeurIPS models and Updated Models are provided. For evaluating NeurIPS models, please download the code from 'Release/NeurIPS version'.

Depth Results (NeurIPS version on KITTI Eigen's splits)

Models	Abs Rel	Sq Rel	RMSE	RMSE(log)	Acc.1	Acc.2	Acc.3
k_depth	0.137	1.089	5.439	0.217	0.830	0.942	0.975
cs+k_depth	0.128	1.047	5.234	0.208	0.846	0.947	0.976

Depth Results (Updated version on KITTI Eigen's splits)

Models	Abs Rel	Sq Rel	RMSE	RMSE(log)	Acc.1	Acc.2	Acc.3
resnet18	0.119	0.858	4.949	0.197	0.863	0.957	0.981
resnet50	0.115	0.814	4.705	0.191	0.873	0.960	0.982

Visual Odometry Results (NeurIPS version, Trained on KITTI 00-08)

Models		Seq. 09	Seq. 10
k_pose	t_err (%)	11.2	10.1
	r_err (degree/100m)	3.35	4.96
cs+k_pose	t_err (%)	8.24	10.7
	r_err (degree/100m)	2.19	4.58

If you use this work, please cite our paper

@inproceedings{bian2019depth,
  title={Unsupervised Scale-consistent Depth and Ego-motion Learning from Monocular Video},
  author={Bian, Jia-Wang and Li, Zhichao and Wang, Naiyan and Zhan, Huangying and Shen, Chunhua and Cheng, Ming-Ming and Reid, Ian},
  booktitle= {Thirty-third Conference on Neural Information Processing Systems (NeurIPS)},
  year={2019}
}

Related projects

• SfMLearner-Pytorch (CVPR 2017, our baseline framework.)

• Depth-VO-Feat (CVPR 2018, trained on stereo videos for depth and visual odometry)

• DF-VO (ICRA 2020, use scale-consistent depth with optical flow for more accurate visual odometry)

• Kitti-Odom-Eval-Python (python code for kitti odometry evaluation)