Semantic3D semantic segmentation with Open3D and PointNet++
Demo project for Semantic3D (semantic-8) segmentation with Open3D and PointNet++. The purpose of this project is to showcase the usage of Open3D in deep learning pipelines and provide a clean baseline implementation for semantic segmentation on Semantic3D dataset. Here's our entry on the semantic-8 test benchmark page.
Open3D is an open-source library that supports rapid development of software that deals with 3D data. The Open3D frontend exposes a set of carefully selected data structures and algorithms in both C++ and Python. The backend is highly optimized and is set up for parallelization. We welcome contributions from the open-source community.
In this project, Open3D was used for
- Point cloud data loading, writing, and visualization. Open3D provides efficient implementations of various point cloud manipulation methods.
- Data pre-processing, in particular, voxel-based down-sampling.
- Point cloud interpolation, in particular, fast nearest neighbor search for label interpolation.
- And more.
Download the dataset Semantic3D and extract it by running the following commands:
Open3D-PointNet2-Semantic3D/dataset/semantic_raw ├── bildstein_station1_xyz_intensity_rgb.labels ├── bildstein_station1_xyz_intensity_rgb.txt ├── bildstein_station3_xyz_intensity_rgb.labels ├── bildstein_station3_xyz_intensity_rgb.txt ├── ...
2. Convert txt to pcd file
Open3D is able to read
.pcd files much more efficiently.
Open3D-PointNet2-Semantic3D/dataset/semantic_raw ├── bildstein_station1_xyz_intensity_rgb.labels ├── bildstein_station1_xyz_intensity_rgb.pcd (new) ├── bildstein_station1_xyz_intensity_rgb.txt ├── bildstein_station3_xyz_intensity_rgb.labels ├── bildstein_station3_xyz_intensity_rgb.pcd (new) ├── bildstein_station3_xyz_intensity_rgb.txt ├── ...
The downsampled dataset will be written to
dataset/semantic_downsampled. Points with
label 0 (unlabled) are excluded during downsampling.
Open3D-PointNet2-Semantic3D/dataset/semantic_downsampled ├── bildstein_station1_xyz_intensity_rgb.labels ├── bildstein_station1_xyz_intensity_rgb.pcd ├── bildstein_station3_xyz_intensity_rgb.labels ├── bildstein_station3_xyz_intensity_rgb.pcd ├── ...
4. Compile TF Ops
We need to build TF kernels in
tf_ops. First, activate the virtualenv and make
sure TF can be found with current python. The following line shall run without
python -c "import tensorflow as tf"
Then build TF ops. You'll need CUDA and CMake 3.8+.
cd tf_ops mkdir build cd build cmake .. make
After compilation the following
.so files shall be in the
Open3D-PointNet2-Semantic3D/tf_ops/build ├── libtf_grouping.so ├── libtf_interpolate.so ├── libtf_sampling.so ├── ...
Verify that that the TF kernels are working by running
cd .. # Now we're at Open3D-PointNet2-Semantic3D/tf_ops python test_tf_ops.py
By default, the training set will be used for training and the validation set
will be used for validation. To train with both training and validation set,
--train_set=train_full flag. Checkpoints will be output to
Pick a checkpoint and run the
predict.py script. The prediction dataset is
--set. Since PointNet2 only takes a few thousand points per
forward pass, we need to sample from the prediction dataset multiple times to
get a good coverage of the points. Each sample contains the few thousand points
required by PointNet2. To specify the number of such samples per scene, use the
python predict.py --ckpt log/semantic/best_model_epoch_040.ckpt \ --set=validation \ --num_samples=500
The prediction results will be written to
Open3D-PointNet2-Semantic3D/result/sparse ├── sg27_station4_intensity_rgb.labels ├── sg27_station4_intensity_rgb.pcd ├── sg27_station5_intensity_rgb.labels ├── sg27_station5_intensity_rgb.pcd ├── ...
The last step is to interpolate the sparse prediction to the full point cloud. We use Open3D's K-NN hybrid search with specified radius.
The prediction results will be written to
Open3D-PointNet2-Semantic3D/result/dense ├── sg27_station4_intensity_rgb.labels ├── sg27_station5_intensity_rgb.labels ├── ...
Finally, if you're submitting to Semantic3D benchmark, we've included a handy tools to rename the submission file names.
Summary of directories
dataset/semantic_raw: Raw Semantic3D data, .txt and .labels files. Also contains the .pcd file generated by
dataset/semantic_downsampled: Generated from
downsample.py. Downsampled data, contains .pcd and .labels files.
result/sparse: Generated from
predict.py. Sparse predictions, contains .pcd and .labels files.
result/dense: Dense predictions, contains .labels files.
result/dense_label_colorized: Dense predictions with points colored by label type.