NeuralBKI

Welcome! This repository contains all software used to create the Convolutional Bayesian Kernel Inference Neural Network.

Table of Contents

• Network
• Use NeuralBKI
  • Dependencies
  • Datasets
  • Training/Testing
• Results
  • KittiOdometry
  • SemanticKITTI
  • Qualitative
• Acknowledgement

Network: NeuralBKI

Use NeuralBKI

Dependencies

• Pytorch - we tested on PyTorch 1.10 and 1.8.2
• ROS - we used ros noetic for map visualization

We also provide an environment.yml which you can use to create a conda environment (This is only tested on Ubuntu 20)

conda env create -f environment.yml
conda activate NeuralBKI

Datasets

• KittiOdometry
  • We preprocessed the Kitti Odometry following Yang et al.. We use ELAS to generate depth images from Kitti Odometry dataset's stereo images. The semantic segmentation labels of the images are from Dilation Network. From the depth images and semantic segmentation, 3D point clouds are generated.
  • You can download the preprocessed data here.
  • kitti_odometry.yml file is located in Config directory. data_dir should be the location of the KittiOdometry dataset.
• SemanticKitti
  • You can download the SemanticKitti ground truth and the semantic segmentation output from darknet53 with KNN
  • Move the semantic segmentation output into a directory named predictions_darknet for each seqeunce.
  • semantic_kitti.yml file is located in Config directory. data_dir should be the location of the SemanticKitti dataset.

Training Model

• Create or modify yaml file in Config directory (ConvBKI_* are specific model yaml file and kitti_odometry.yml, semantic_kitti.yml are the dataset yaml files)
• Change MODEL_NAME parameter to the name of the corresponding yaml file in train.py
• Run train.py

Testing Model

• Change MODEL_NAME parameter to the name of the corresponding yaml file in generate_result.py
• Run generate_results.py

YAML Parameters

• dataset - the name of the dataset yaml file
• save_dir - location where the filter weights are saved
• load_epoch - which filter weights to load during mapping
• visualize - whether to publish to rviz
• meas_result - whether to generate result
• gen_preds - whether to save the map

Results

KittiOdometry

Method	Building	Road	Vege.	Sidewalk	Car	Sign	Fence	Pole	Average
Segmentation	92.1	93.9	90.7	81.9	94.6	19.8	78.9	49.3	75.1
Yang et al.	95.6	90.4	92.8	70.0	94.4	0.1	84.5	49.5	72.2
BGKOctoMap-CRF	94.7	93.8	90.2	81.1	92.9	0.0	78.0	49.7	72.5
S-CSM	94.4	95.4	90.7	84.5	95.0	22.2	79.3	51.6	76.6
S-BKI	94.6	95.4	90.4	84.2	95.1	27.1	79.3	51.3	77.2
ConvBKI Single	92.7	94.8	90.9	84.7	95.1	22.1	80.2	52.1	76.6
ConvBKI Per Class	94.0	95.5	91.0	87.0	95.1	22.8	81.8	52.9	77.5
ConvBKI Compound	94.0	95.6	91.0	87.2	95.1	22.8	81.9	54.3	77.7

SemanticKitti

Data Split	Method	Car	Bicycle	Motocycle	Truck	Other Veh.	Person	Bicylist	Motorcyclist	Road	Parking	Sidewalk	Other Gr.	Building	Fence	Vegetation	Trunk	Terrain	Pole	Sign	Average
Val	Da.-kNN	91.0	25.0	47.1	40.7	25.5	45.2	62.9	0.0	93.8	46.5	81.9	0.2	85.8	54.2	84.2	52.9	72.7	53.2	40.0	52.8
	S-CSM	92.6	32.5	54.9	43.4	26.2	51.3	69.2	0.0	94.6	49.2	84.0	0.1	87.9	58.4	85.8	59.9	73.3	61.7	43.0	56.2
	S-BKI	93.5	33.5	57.3	44.5	27.2	52.9	72.1	0.0	94.4	49.6	84.0	0.0	88.7	59.6	86.9	62.5	75.3	63.6	45.1	57.4
	ConvBKI Sin.	92.0	29.8	57.4	44.4	25.2	53.1	72.1	0.0	93.1	45.8	80.9	0.1	88.2	57.8	86.1	61.2	74.0	59.7	44.4	56.1
	ConvBKI PC	92.6	34.5	59.2	34.6	39.4	58.6	73.5	0.0	93.0	47.2	80.9	0.1	88.4	58.3	86.4	61.7	74.2	58.4	47.4	57.3
	ConvBKI Com.	94.0	37.5	60.0	33.3	40.5	59.4	74.4	0.0	93.3	49.0	81.2	0.1	88.5	59.5	86.8	62.2	75.0	59.9	46.5	58.0
---	---------	----	------	------	------	------	------	------	-----	-----	------	------	------	------	------	------	------	------	----	----	------
Test	Da.-kNN	82.4	26.0	34.6	21.6	18.3	6.7	2.7	0.5	91.8	65.0	75.1	27.7	87.4	58.6	80.5	55.1	64.8	47.9	55.9	47.5
	S-BKI	83.8	30.6	43.0	26.0	19.6	8.5	3.4	0.0	92.6	65.3	77.4	30.1	89.7	63.7	83.4	64.3	67.4	58.6	67.1	51.3
	ConvBKI Com.	83.8	32.2	43.8	29.8	23.2	8.3	3.1	0.0	91.4	62.6	75.2	27.5	89.1	61.6	81.6	62.5	65.2	53.9	63.0	50.4

Qualitative

• Example map produced by ConvBKI Compound on the validation set of Semantic KITTI. Filtering out voxels with high variance improves the visual quality of the robotic map.

• Illustration of kernels learned by ConvBKI on the road and pole semantic classes. The network learns that the pole class is tall while the road class is wide.

Acknowledgement

We utilize data and code from:

• [1] Kitti Odometry
• [2] SemanticKITTI
• [3] ELAS
• [4] Dilation Network
• [5] darknet53 with KNN

Reference

If you find our work useful in your research work, consider citing our paper

@ARTICLE{wilson2022convolutional,
  title={Convolutional Bayesian Kernel Inference for 3D Semantic Mapping},
  author={Wilson, Joey and Fu, Yuewei and Zhang, Arthur and Song, Jingyu and Capodieci, Andrew and Jayakumar, Paramsothy and Barton, Kira and Ghaffari, Maani},
  journal={arXiv preprint arXiv:2209.10663},
  year={2022}
}

Bayesian Spatial Kernel Smoothing for Scalable Dense Semantic Mapping (PDF)

@ARTICLE{gan2019bayesian,
author={L. {Gan} and R. {Zhang} and J. W. {Grizzle} and R. M. {Eustice} and M. {Ghaffari}},
journal={IEEE Robotics and Automation Letters},
title={Bayesian Spatial Kernel Smoothing for Scalable Dense Semantic Mapping},
year={2020},
volume={5},
number={2},
pages={790-797},
keywords={Mapping;semantic scene understanding;range sensing;RGB-D perception},
doi={10.1109/LRA.2020.2965390},
ISSN={2377-3774},
month={April},}