Skip to content
Branch: release
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Type Name Latest commit message Commit time
Failed to load latest commit information.

Deep 3D Descriptor

This repository contains code to learn and apply a local feature descriptor for 3D LiDAR scans. We provide the scripts to train the model and a C++ library to interface the learned decriptor with PCL. Training data to learn the model as well as trained models are available.

Related Publication

Ayush Dewan, Tim Caselitz, Wolfram Burgard
Learning a Local Feature Descriptor for 3D LiDAR Scans
IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Madrid, Spain, 2018

1. License

This software is released under GPLv3. If you use it in academic work, please cite:

  author = {Ayush Dewan and Tim Caselitz and Wolfram Burgard},
  title = {Learning a Local Feature Descriptor for 3D LiDAR Scans},
  booktitle = {Proc.~of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
  address = {Madrid, Spain},
  year = {2018},
  url = {}

2. Training the Network

2.1. Prerequisites

  • Tensorflow
  • Pyhton 2.7
  • H5py

2.2. Dataset


This will download the dataset used for training and testing. The data is in the format NxCxHxW and is stored in hdf5 files. The first channel corresponds to the depth value and the second channel corresponds to the LiDAR intensity value. For every example there is a label and all the examples corresponding to the same keypoint have same label. These labels are not used directly for training but only to identify examples belonging to same keypoint.

2.3. Training the model

All the files required for training and testing the model is in python_scripts folder. To train the model following script has to be executed.


--path_to_store_models (default: learned_models/)
--batch_size (default: 32)
--epochs (default: 5)
--learning_rate (default: 0.0001)
--eta (default: 0.0005)
--growth_rate (default: 4)
--number_of_models_stored (default: 2)

We recommend the following training procedure. Train the network with the default parameters. Then retrain the network with learning rate set to 0.00001 and weights initialized using the last saved model from the first training. The path to the trained model can be set using the paramter --fine_tune_model_name.

2.3.1. Example commands for completing the above mentioned training procedure:

python --model_name  my_model --path_to_training_data ../dataset/training_data.hdf5  --path_to_testing_data  ../dataset/testing_data.hdf5
python --model_name  my_model_retrain --path_to_training_data ../dataset/training_data.hdf5  --path_to_testing_data  ../dataset/testing_data.hdf5 --learning_rate 0.00001 --fine_tune_model_name learned_models/my_model_110062

2.4. Testing the model

To test the model we provide the code for calculating the FPR-95 error. The model is tested on 50,000 positive and negative image patches from the testing data. This script prints the FPR-95 error, plots the curve between TPR and FPR, and stores the data used for plotting the curve.



2.4.1. Example command for testing a trained model

python --path_to_saved_model learned_models/my_model_retrain_55031  --path_to_testing_data ../dataset/testing_data.hdf5

3. C++ PCL Interface

3.1. Prerequisites

Thrift is required for both C++ and Python.

3.2. Installing

In the project directory

mkdir build
cd build
cmake ..

In case PCL 1.8 is not found, use -DPCL_DIR variable to specify the path of PCL installation.

cmake .. -DPCL_DIR:STRING=PATH_TO_PCLConfig.cmake

3.3. Downloading the test pointcloud


This will download the test pointcloud files used in the alignment experiment in the paper. The name format for the files is seq_scan_trackID_object.pcd. 'seq' corresponds to the sequence number from KITTI tracking benchmark. 'scan' is the scan used from the given sequence. 'trackID' is the object ID provided by the benchmark. For instance, '0011_126_14_object.pcd' and '0011_127_14_object.pcd' is the same object in two consecutive scans.

3.4. Downloading the models


This will download the trained model files. We provide the model for a feature descriptor learned simulataneously with a metric for matching as well as a feature descriptor learned using hinge loss. 'deep_3d_descriptor_matching' is the model for the descriptor using the learned metric, 'deep_3d_descriptor_hinge_loss' for the descriptor trained using hinge loss.

3.5. Using the learned descriptor with PCL

We provide a service and client API for using the learned feature descriptor with PCL.

All the Thrift related code and the python service file is in the folder python_cpp.

The service has to be started within the tensorflow environment.



We provide two test files, the first one for computing a feature descriptor and the second one for matching the descriptors.

For computing feature descriptor


--feature_neighborhood_radius (default: 1.6)
--sampling_radius (default: 0.4)

For visualizing the correspondences and using them to align the pointclouds (--use_ransac for inlier correspondences only)



3.5.1. Examples for visualizing the correspondences and the aligned pointcloud Estimate the correspondences using the learned metric
python --model_name ../models/deep_3d_descriptor_matching --use_hinge_loss 0

./visualize_deep_3d_descriptor_correspondences --path_to_source_pcd_file ../test_pcd/0011_1_2_object.pcd --sampling_radius_source 0.2 --path_to_target_pcd_file ../test_pcd/0011_2_2_object.pcd --sampling_radius_target 0.1 --feature_neighborhood_radius 1.6 --use_learned_metric 1 --use_ransac 0

Matched Keypoints Aligned Scans Estimate the correspondences using Euclidean distance
python --model_name ../models/deep_3d_descriptor_hinge_loss --use_hinge_loss 1

./visualize_deep_3d_descriptor_correspondences --path_to_source_pcd_file ../test_pcd/0011_1_2_object.pcd --sampling_radius_source 0.2 --path_to_target_pcd_file ../test_pcd/0011_2_2_object.pcd --sampling_radius_target 0.1 --feature_neighborhood_radius 1.6 --use_learned_metric 0 --use_ransac 0

Matched Keypoints Aligned Scans
You can’t perform that action at this time.