Lifelong-MonoDepth: Lifelong Learning for Multi-Domain Monocular Metric Depth Estimation

We present an efficient multi-head framework that enables lifelong, cross-domain, and scare-aware monocular depth learning. Depth maps in the real world are significantly different across domains; their quality and scales are domain dependent. Therefore, the model has to assemble multiple prediction branches for multi-domain metric depth inference. To this end, we present a framework that consists of a domain-shared encoder and multiple domain-specific predictors. The framework allows robust metric depth learning across multi-domains. For more detailed information, please check our paper.

Results

We show through experiments that the proposed method can

• enable lifelong learning for scale-aware depth estimation,
• cope with significant domain shift,
• infer a depth map in real-time.

Qualitative comparisons:

Datasets

We provided data index of RGB and depth pairs for both training and test set of all three datasets in csv file, you can find them in ./datasets/. Then, download the data required for training and test.

• NYU-v2 (0~10 meters), download via the link
  
• KITTI (0~80 meters), download via the official homepage
  
• ScanNet (0-6 meters), download via the official homepage
  

Trained Models

We provide all trained models as follows. We name the model in the order of learning.

Models	Description
N.pth.tar	Trained model on the NYU-v2
K.pth.tar	Trained model on the KITTI
S.pth.tar	Trained model on the ScanNet
NK.pth.tar	Trained model on the NYU-v2 and KITTI
KN.pth.tar	Trained model on the KITTI and NYU-v2
NKS.pth.tar	Trained model on the NYU-v2, KITTI, and ScanNet
NSK.pth.tar	Trained model on the NYU-v2, ScanNet, and KITTI
SNK.pth.tar	Trained model on the ScanNet, NYU-v2, and KITTI
SKN.pth.tar	Trained model on the ScanNet, KITTI, and NYU-v2
KSN.pth.tar	Trained model on the KITTI, ScanNet, and NYU-v2
KNS.pth.tar	Trained model on the KITTI, NYU-v2, and ScanNet

Running

• Test
  

  Download a trained model that you want to test and put it into ./runs/. The default model is NKS.pth.tar.

  1. Testing with domain prior that assumes learning orders are given as a prior: python test.py
     
  2. Testing without domain prior that requires automatically select the domain-specific predictor during inference: python test_inference.py
     

• Train
  

  The following shows examples of learning in the order of NYU-v2 → KITTI → ScanNet

  1. Training on a single-domain: python train_N.py
     
  2. Training on two domains: python train_NK.py
     
  3. Training on three domains: python train_NKS.py
     

Citation

@ARTICLE{10293000,
  author={Hu, Junjie and Fan, Chenyou and Zhou, Liguang and Gao, Qing and Liu, Honghai and Lam, Tin Lun},
  journal={IEEE Transactions on Neural Networks and Learning Systems}, 
  title={Lifelong-MonoDepth: Lifelong Learning for Multidomain Monocular Metric Depth Estimation}, 
  year={2023},
  volume={},
  number={},
  pages={1-11},
  doi={10.1109/TNNLS.2023.3323487}}