RGBD-3DGS-SLAM is a sophisticated SLAM system that employs 3D Gaussian Splatting (3DGS) from Guassian Splatting SLAM (MonoGS) for precise point cloud and visual odometry estimations. It leverages neural network Universal Monocular Metric Depth Estimation (UniDepthV2) to infer depth and camera intrinsics from RGB images, and can also utilize additional camera information and depth maps if available. The system outputs high-quality point clouds and visual odometry data, making RGBD-3DGS-SLAM a versatile tool for a wide range of applications in robotics and computer vision.
Real-Time MonoGS with UniDepthV2 for Depth and camera intrinsics with ROS 2
MonoGS with UniDepthV2 and ROS 2
Clone the repo and the submodules using
https://github.com/jagennath-hari/RGBD-3DGS-SLAM --recursive
Install the packages using
cd RGBD-3DGS-SLAM && chmod +x install.sh && source ./install.sh
Or build from source using these libraries.
- PyTorch (Official Link).
- MonoGS (Official Link).
- UniDepth (Official Link).
- RoboStack ROS 2 Humble (Offical Link).
There is also enviroment.yml file, you can install or use as a reference using
conda env create -f environment.yml
cd MonoGS && bash scripts/download_tum.sh
cd MonoGS Move to this directory.
You can run the system on the TUM dataset using the same method from the original repository.
python slam.py --config configs/mono/tum/fr3_office.yaml
The code has been refactored to not use the ground truth depth but the depth from UniDepthV2 instead. It can be executed similary provided by original repository. A new directory called neural_depth will get created and new Depth Maps from UniDepthV2 will be available in it.
python slam.py --config configs/rgbd/tum/fr3_office.yaml
Ground truth Depth Map from TUM dataset |
Neural Depth Map produced by UniDepthV2 |
Original MonoGS Result |
MonoGS with UniDepthV2 Result |
Final Cloud in RVIZ 2
An online Guassian Viewer can be used to view the cloud in the result directory.
Original MonoGS Cloud |
MonoGS with UniDepthV2 Cloud |
To run using any camera you can leverage ROS 2 publisher-subscriber (DDS) protocol. A new config file MonoGS/configs/live/ROS.yaml will allow you to use ROS 2.
You can change the topic names in the config file. An example given below.
ROS_topics:
camera_topic: '/zed2i/zed_node/rgb/image_rect_color'
camera_info_topic: '/zed2i/zed_node/rgb/camera_info'
depth_topic: '/zed2i/zed_node/depth/depth_registered'
depth_scale: 1
The camera topic is mandatory, but camera_info_topic and depth_topic are optional.
The other combinations are
- An uncalibrated camera with Depth Maps.
ROS_topics:
camera_topic: '/zed2i/zed_node/rgb/image_rect_color'
camera_info_topic: 'None'
depth_topic: '/zed2i/zed_node/depth/depth_registered'
depth_scale: 1
- A calibrated camera without Depth Maps.
ROS_topics:
camera_topic: '/zed2i/zed_node/rgb/image_rect_color'
camera_info_topic: '/zed2i/zed_node/rgb/camera_info'
depth_topic: 'None'
depth_scale: 1
- An uncalibrated camera without Depth Maps.
ROS_topics:
camera_topic: '/zed2i/zed_node/rgb/image_rect_color'
camera_info_topic: 'None'
depth_topic: 'None'
depth_scale: 1
UniDepthV2 will estimate both camera intrinsics and metric Depth Map. So an RGB-D Image will be produced regardless.
To execute the SLAM system
Move to MonoGS directory if not already cd MonoGS.
To start the system
python slam.py --config configs/live/ROS.yaml
Depth Maps can be of different scales, make sure to set the depth scale in the ROS topics infos.
UniDepthV2 is not perfect and the estimated intrinsics and Depth Maps may not be accurate, it is advised to use a calibrated camera and use Depth Maps if available.
MonoGS with UniDepthV2 and ROS 2
During operation the system will output two topics when a new keyframe is created:
- /monoGS/cloud (sensor_msgs/PointCloud2)
- /monoGS/trajectory (nav_msgs/Path)
If you found this code/work to be useful in your own research, please considering citing the following:
@inproceedings{Matsuki:Murai:etal:CVPR2024,
title={{G}aussian {S}platting {SLAM}},
author={Hidenobu Matsuki and Riku Murai and Paul H. J. Kelly and Andrew J. Davison},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
year={2024}
}@inproceedings{piccinelli2024unidepth,
title={UniDepth: Universal Monocular Metric Depth Estimation},
author = {Piccinelli, Luigi and Yang, Yung-Hsu and Sakaridis, Christos and Segu, Mattia and Li, Siyuan and Van Gool, Luc and Yu, Fisher},
booktitle = {IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
year={2024}
}This software is released under BSD-3-Clause license. You can view a license summary here. MonoGS and UniDepth have their own licenses respectively.
This work incorporates many open-source codes.