Skip to content

Latest commit

 

History

History
274 lines (187 loc) · 12.5 KB

README.md

File metadata and controls

274 lines (187 loc) · 12.5 KB
Raw

Struct-MDC

video

journal arxiv

(click the above buttons for redirection!)

Official page of "Struct-MDC: Mesh-Refined Unsupervised Depth Completion Leveraging Structural Regularities from Visual SLAM", which is accepted in IEEE RA-L'22 & IROS'22

  • Depth completion from Visual(-inertial) SLAM using point & line features.

Code & Dataset verion info.

  • Code (including source code, utility code for visualization) & Dataset will be finalized & released soon!
  • version info
    • (04/20) docker image has been uploaded.
    • (04/21) Dataset has been uploaded.
    • (04/21) Visusal-SLAM module (modified UV-SLAM) has been uploaded.
    • (11/01) data generation tools has been uploaded.
    • (12/14) Visusal-SLAM module has been modified.
    • (12/22) Struct-MDC PyTorch code has been uploaded.



Results

  • 3D Depth estimation results
    • VOID (left three columns) and NYUv2 (right three columns)
    • detected features (top row), estimation from baseline (middle row) and ours (bottom row)

  • 2D Depth estimation results
Ground truth Baseline Struct-MDC (Ours)



Installation

1. Prerequisites (we've validated our code in the following environment!)

(You can skip the Visual-SLAM module part, if you just want to use NYUv2, VOID, and PLAD datsets)

  • Common
    • Ubuntu 18.04
  • Visual-SLAM module
    • ROS melodic
    • OpenCV 3.2.0 (under 3.4.1)
    • Ceres Solver-1.14.0
    • Eigen-3.3.9
    • CDT library 
      git clone https://github.com/artem-ogre/CDT.git
cd CDT
mkdir build && cd build
cmake -DCDT_USE_AS_COMPILED_LIBRARY=ON -DCDT_USE_BOOST=ON ..
cmake --build . && cmake --install .
sudo make install
  • Depth completion module
    • Python 3.7.7
    • PyTorch 1.5.0 (you can easily reproduce equivalent environment using our docker image)

2. Build

(You can skip the Visual-SLAM module part, if you just want to use NYUv2, VOID, and PLAD datsets)

  • Visual-SLAM module

    • As visual-SLAM, we modified the UV-SLAM, which is implemented in ROS environment.
    • make sure that your catkin workspace has following cmake args: -DCMAKE_BUILD_TYPE=Release
    cd ~/$(PATH_TO_YOUR_ROS_WORKSPACE)/src
git clone --recursive https://github.com/url-kaist/Struct-MDC
cd ..
catkin build
source ~/$(PATH_TO_YOUR_ROS_WORKSPACE)/devel/setup.bash

  • Depth completion module

    • Our depth compeltion module is based on the popular Deep-Learning framework, PyTorch.
    • For your convenience, we share our environment as Docker image. We assume that you have already installed the Docker. For Docker installation, please refer here
    # pull our docker image into your local machine
docker pull zinuok/nvidia-torch:latest

# run the image mounting our source
docker run -it --gpus "device=0" -v $(PATH_TO_YOUR_LOCAL_FOLER):/workspace zinuok/nvidia-torch:latest bash

3. Trouble shooting

  • any issues found will be updated in this section.
  • if you've found any other issues, please post it on Issues tab. We'll do our best to resolve your issues.



Download (processed-) Datasets & Pre-trained weights

  • Datasets

    • There are three datasets we used for verifying the performance of our proposed method.
    • We kindly share our modified datasets, which include also the line feature from UV-SLAM and already has been pre-processed. For testing our method, please use the modified one.

  • Pre-trained weights

    • We provide our pre-trained network, which is same as the one used in the paper.
    • Please download the weigths from the below table and put into the 'Struct-MDC_src/pretrained'.
    • There are two files for each dataset: pre-trained weights for
      • Depth network: used at training/evaluation time.
      • Pose network: used only at training time. (for supervision)

  • PLAD

    • This is our proposed dataset, which has point & line feature depth from UV-SLAM.
    • Each sequence was acquired in various indoor / outdoor man-made environment.
    • The PLAD dataset has been slightly modified from the paper's version. We are trying to optimize each sequence and reproduce the results in this dataset. Of course, this dataset is always available, but it will be continuously updated with version information. So please keep track of our updated instructions.

For more details on each dataset we used, please refer our paper

Dataset ref. link train/eval data ROS bag DepthNet weight PoseNet weight
VOID original void-download void-raw void-depth void-pose
NYUv2 pre-processed nyu-download - nyu-depth nyu-pose
PLAD proposed! plad-download plad-raw plad-depth plad-pose



Running Struct-MDC

Using our pre-trained network, you can simply run our network and verify the performance as same as our paper.

# move the pre-trained weights to the following folder:
cd Struct-MDC/Struct-MDC_src/
mv $(PATH_TO_PRETRAINED_WEIGHTS)  ./pretrained/{plad | nyu | void}/struct_MDC_model/

# link raw dataset
tar -xvzf {PLAD: PLAD_v2 | VOID: void_parsed_line | NYUV2: nyu_v2_line}.tar.gz
ln -s $(PATH_TO_DATASET_FOLDER) ./data/

# data preparation
python3 setup/setup_dataset_{nyu | void | plad}.py

# running
bash bash/run_structMDC_{nyu | void | plad}_pretrain.sh
  • Evaluation results
MAE RMSE < 1.05 < 1.10 < 1.25^3
paper 1170.303 1481.583 4.567 8.899 67.071
modified - - - - -



Training Struct-MDC

You can also train the network from the beginning using your own data.
However, in this case, you have to prepare the data as same as ours, from following procedure:

(Since our data structure, dataloader, and pre-processing code templetes follows our baseline: KBNet, you can also refer the author's link. We thanks the KBNet's authors.)

# data preparation
cd Struct-MDC_src
mkdir data
ln -s $(PATH_TO_{NYU | VOID | PLAD}_DATASET_ROOT) data/
python3 setup/setup_dataset_{nyu | void | plad}.py

# train
bash bash/train_{nyu | void | plad}.sh



Prepare Your own Dataset



Citation

If you use the algorithm in an academic context, please cite the following publication:

@article{jeon2022struct,
  title={Struct-MDC: Mesh-Refined Unsupervised Depth Completion Leveraging Structural Regularities From Visual SLAM},
  author={Jeon, Jinwoo and Lim, Hyunjun and Seo, Dong-Uk and Myung, Hyun},
  journal={IEEE Robotics and Automation Letters},
  volume={7},
  number={3},
  pages={6391--6398},
  year={2022},
  publisher={IEEE}
}

@article{lim2022uv,
  title={UV-SLAM: Unconstrained Line-based SLAM Using Vanishing Points for Structural Mapping},
  author={Lim, Hyunjun and Jeon, Jinwoo and Myung, Hyun},
  journal={IEEE Robotics and Automation Letters},
  year={2022},
  publisher={IEEE},
  volume={7},
  number={2},
  pages={1518-1525},
  doi={10.1109/LRA.2022.3140816}
}



Acknowledgements

  • Visual-SLAM module
    We use UV-SLAM, which is based on VINS-MONO, as our baseline code. Thanks for H. Lim and Dr. Qin Tong, Prof. Shen etc very much.
  • Depth completion module
    We use KBNet as our baseline code. Thanks for W. Alex and S. Stefano very much.



Licence

The source code is released under GPLv3 license. We are still working on improving the code reliability. For any technical issues, please contact Jinwoo Jeon (zinuok@kaist.ac.kr).