- Label pointcloud from stereo cameras or lidar with Tensorflow trained neural nets graph.pb
- Build 3D Semantic voxel maps, 2D occupancy maps, 2D obstacle distance maps
The Youtube video of the Cassie Bipedal robot's perception and mapping system is powered by this library.

Here is a semantic map built on NCLT dataset

Two types of accepted inputs (from ROS topics):
- Raw RGBD images from depth cameras, or
- Lidar scans + RGB images.
C++: pcl, eigen, xmlrpcpp
Outputs (in ROS messages):
- Labeled images, each pixel with a distribution across classes
- Labeled PointCloud ROS message
- Semantic Octomap
- 2D occupancy maps
- 2D cost map, cost computed from distance to closest obstacles
Tested on
- AlienWare Laptop (running in realtime for the Cassie Blue experiement): 3FPS on i7-6820, 32GB RAM, GTX1070
- Dell XPS: 5FPS on i7-7700, 32GB RAM, GTX1070.
- Jetson Xavier: Around 4Hz
ROS: Tested on Noetic + Ubuntu 20.04
Neural Network:
tensorflow-gpu-2.5.0tensorrtcuda-11.4,cudnn-8.2.1
Python thirdparty:
python-pclpython-opencv
C++: pcl, eigen, OpenCV
ros thirdparty:
semantic octomap: Modified octomap, supporting Bayesian updates for semantic label fusionros_numpy- ros-noetic-octomap-rviz-plugins
Make sure that your semantic octomap is build. Then run
catkin_make install -DCMAKE_BUILD_TYPE=Release -Doctomap_DIR=${you_octomap_source_directory}/lib/cmake/octomap/ --pkg SegmentationMapping
- run on Cassie with a single stereo camera (Intel Realsense):
roslaunch SegmentationMapping cassie_stereo_py.launch - run nclt dataset:
roslaunch SegmentationMapping nclt_distribution_deeplab.launch - run on mulitple cameras with multiple segmentations at the same time:
roslaunch SegmentationMapping cassie_stereo_py_multi_camera.launch

