GAM-Depth

PyTorch implementation of our paper (2024 ICRA):

GAM-Depth: Self-Supervised Indoor Depth Estimation Leveraging a Gradient-Aware Mask and Semantic Constraints

Anqi Cheng, Zhiyuan Yang, Haiyue Zhu, Kezhi Mao

Getting Started

Installation

Step1: Creating a virtual environment

conda create -n gam_depth python=3.6
conda activate gam_depth
conda install pytorch==1.10.1 torchvision==0.11.2 torchaudio==0.10.1 cudatoolkit=11.3 -c pytorch -c conda-forge

Step2: Downloading the modified scikit_image package following StructDepth

unzip scikit-image-0.17.2.zip
cd scikit-image-0.17.2
python setup.py build_ext -i
pip install -e .

Step3: Installing other packages

pip install -r requirements.txt

Download pretrained model

Please download pretrained models and unzip them to MODEL_PATH

Inference single image

python inference_single_image.py --image_path=/home/image_path --load_weights_folder=MODEL_PATH

Evaluation

Download test dataset

Please download test datasets of NYUv2, ScanNet, and InteriorNet and unzip them to VAL_PATH

Evaluate NYUv2/InteriorNet/ScanNet depth

Modify the evaluation script in eval.sh to evaluate NYUv2/ScanNet/InteriorNet depth separately

python evaluation/nyuv2_eval_depth.py \
  --data_path VAL_PATH \
  --load_weights_folder MODEL_PATH \

Trainning

Download NYU V2 dataset

The raw NYU dataset is about 400G and has 590 videos. You can download the raw datasets from there

Download main directions and proxy semantic labels

Please download main directions with a random flip and proxy semantic labels from there and unzip them to VPS_PATH and SEG_PATH

Proxy semantic labels are generated from Light-Weight RefineNet

Training

Modify the training script train.sh for PATH or different trainning settings.

python train.py \
  --data_path DATA_PATH \
  --val_path VAL_PATH \
  --vps_path VPS_PATH \
  --seg_path SEG_PATH \
  --log_dir ../logs/ \
  --model_name 1 \
  --batch_size 4 \
  --num_epochs 50 \
  --start_epoch 0 \
  --load_weights_folder MODEL_PATH/pretrain/ \
  --using_GAM GAM \
  --using_seg

Acknowledgement

We borrowed a lot of codes from scikit-image, monodepth2, P²Net, StructDepth, Light-Weight RefineNet, and LEGO. Thanks for their excellent works!

Citation

@article{cheng2024gam,
  title={GAM-Depth: Self-Supervised Indoor Depth Estimation Leveraging a Gradient-Aware Mask and Semantic Constraints},
  author={Cheng, Anqi and Yang, Zhiyuan and Zhu, Haiyue and Mao, Kezhi},
  journal={arXiv preprint arXiv:2402.14354},
  year={2024}
}