NDC-Scene: Boost Monocular 3D Semantic Scene Completion in Normalized Device Coordinates Space

Official PyTorch implementation for the ICCV 2023 paper.

NDC-Scene: Boost Monocular 3D Semantic Scene Completion in Normalized Device Coordinates Space
Jiawei Yao*, Chuming Li*, Keqiang Sun*, Yingjie Cai, Hao Li, Wanli Ouyang, Hongsheng Li

* equal contribution

NYUv2	SemanticKITTI

Requirements

1. Create conda environment:

$ conda create -y -n ndcscene python=3.7
$ conda activate ndcscene

2. This code is compatible with python 3.7, pytorch 1.7.1 and CUDA 10.2. Please install PyTorch:

$ conda install pytorch==1.7.1 torchvision==0.8.2 torchaudio==0.7.2 cudatoolkit=10.2 -c pytorch

3. Install the additional dependencies:

$ cd NDCScene/
$ pip install -r requirements.txt

4. Install tbb:

$ conda install -c bioconda tbb=2020.2

5. Downgrade torchmetrics to 0.6.0

$ pip install torchmetrics==0.6.0

6. Finally, install NDCScene:

$ pip install -e ./

Datasets

SemanticKITTI

1. You need to download

   • The Semantic Scene Completion dataset v1.1 (SemanticKITTI voxel data (700 MB)) from SemanticKITTI website
   • The KITTI Odometry Benchmark calibration data (Download odometry data set (calibration files, 1 MB)) and the RGB images (Download odometry data set (color, 65 GB)) from KITTI Odometry website.

2. Create a folder to store SemanticKITTI preprocess data at /path/to/kitti/preprocess/folder.

3. Store paths in environment variables for faster access (Note: folder 'dataset' is in /path/to/semantic_kitti):

$ export KITTI_PREPROCESS=/path/to/kitti/preprocess/folder
$ export KITTI_ROOT=/path/to/semantic_kitti 

4. Preprocess the data to generate labels at a lower scale, which are used to compute the ground truth relation matrices:

$ cd NDCScene/
$ python ndcscene/data/semantic_kitti/preprocess.py kitti_root=$KITTI_ROOT kitti_preprocess_root=$KITTI_PREPROCESS

NYUv2

1. Download the NYUv2 dataset.

2. Create a folder to store NYUv2 preprocess data at /path/to/NYU/preprocess/folder.

3. Store paths in environment variables for faster access:

$ export NYU_PREPROCESS=/path/to/NYU/preprocess/folder
$ export NYU_ROOT=/path/to/NYU/depthbin 

4. Preprocess the data to generate labels at a lower scale, which are used to compute the ground truth relation matrices:

$ cd NDCScene/
$ python ndcscene/data/NYU/preprocess.py NYU_root=$NYU_ROOT NYU_preprocess_root=$NYU_PREPROCESS

Pretrained models

Download NDCScene pretrained models on NYUv2 & SemanticKITTI.

Training

To train NDCScene with SemanticKITTI, type:

SemanticKITTI

1. Create folders to store training logs at /path/to/kitti/logdir.

2. Store in an environment variable:

$ export KITTI_LOG=/path/to/kitti/logdir

3. Train NDCScene using 4 GPUs with batch_size of 4 (1 item per GPU) on Semantic KITTI:

$ cd NDCScene/
$ python ndcscene/scripts/train_ndcscene.py \
    dataset=kitti \
    enable_log=true \
    kitti_root=$KITTI_ROOT \
    kitti_preprocess_root=$KITTI_PREPROCESS\
    kitti_logdir=$KITTI_LOG \
    n_gpus=4 batch_size=4    

NYUv2

1. Create folders to store training logs at /path/to/NYU/logdir.

2. Store in an environment variable:

$ export NYU_LOG=/path/to/NYU/logdir

3. Train NDCScene using 2 GPUs with batch_size of 4 (2 item per GPU) on NYUv2:

$ cd NDCScene/
$ python ndcscene/scripts/train_ndcscene.py \
    dataset=NYU \
    NYU_root=$NYU_ROOT \
    NYU_preprocess_root=$NYU_PREPROCESS \
    logdir=$NYU_LOG \
    n_gpus=2 batch_size=4

Evaluating

SemanticKITTI

To evaluate NDCScene on SemanticKITTI validation set, type:

$ cd NDCScene/
$ python ndcscene/scripts/eval_ndcscene.py \
    dataset=kitti \
    kitti_root=$KITTI_ROOT \
    kitti_preprocess_root=$KITTI_PREPROCESS \
    n_gpus=1 batch_size=1

NYUv2

To evaluate NDCScene on NYUv2 test set, type:

$ cd NDCScene/
$ python ndcscene/scripts/eval_ndcscene.py \
    dataset=NYU \
    NYU_root=$NYU_ROOT\
    NYU_preprocess_root=$NYU_PREPROCESS \
    n_gpus=1 batch_size=1

Inference

Please create folder /path/to/ndcscene/output to store the NDCScene outputs and store in environment variable:

export MONOSCENE_OUTPUT=/path/to/ndcscene/output

NYUv2

To generate the predictions on the NYUv2 test set, type:

$ cd NDCScene/
$ python ndcscene/scripts/generate_output.py \
    +output_path=$MONOSCENE_OUTPUT \
    dataset=NYU \
    NYU_root=$NYU_ROOT \
    NYU_preprocess_root=$NYU_PREPROCESS \
    n_gpus=1 batch_size=1

Semantic KITTI

To generate the predictions on the Semantic KITTI validation set, type:

$ cd NDCScene/
$ python ndcscene/scripts/generate_output.py \
    +output_path=$MONOSCENE_OUTPUT \
    dataset=kitti \
    kitti_root=$KITTI_ROOT \
    kitti_preprocess_root=$KITTI_PREPROCESS \
    n_gpus=1 batch_size=1

Visualization

We use mayavi to visualize the predictions. Please install mayavi following the official installation instruction. Then, use the following commands to visualize the outputs on respective datasets.

You also need to install some packages used by the visualization scripts using the commands:

pip install tqdm
pip install omegaconf
pip install hydra-core

NYUv2

$ cd NDCScene/
$ python ndcscene/scripts/visualization/NYU_vis_pred.py +file=/path/to/output/file.pkl

Semantic KITTI

$ cd NDCScene/
$ python ndcscene/scripts/visualization/kitti_vis_pred.py +file=/path/to/output/file.pkl +dataset=kitt

Citation

@InProceedings{Yao_2023_ICCV,
    author    = {Yao, Jiawei and Li, Chuming and Sun, Keqiang and Cai, Yingjie and Li, Hao and Ouyang, Wanli and Li, Hongsheng},
    title     = {NDC-Scene: Boost Monocular 3D Semantic Scene Completion in Normalized Device Coordinates Space},
    booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
    month     = {October},
    year      = {2023},
    pages     = {9455-9465}

}

Acknowledgement

This project is built based on MonoScene. We thank the contributors of the prior project for building such excellent codebase and repo. Please refer to this repo (https://github.com/astra-vision/MonoScene) for more documentations and details.