Timothy Chen1 · Aiden Swann1 · Javier Yu1 · Ola Shorinwa1 · Riku Murai2 · Monroe Kennedy III1 · Mac Schwager1
1Stanford University
2Imperial College London
Project Page | Paper | arXiv | Data
SAFER-Splat (Simultaneous Action Filtering and Environment Reconstruction) is a real-time, scalable, and minimally invasive action filter, based on control barrier functions, for safe robotic navigation in a detailed map constructed at runtime using Gaussian Splatting.
We propose a novel Control Barrier Function (CBF) that not only induces safety with respect to all Gaussian primitives in the scene, but when synthesized into a controller, is capable of processing hundreds of thousands of Gaussians while maintaining a minimal memory footprint and operating at 15 Hz during online Splat training. Of the total compute time, a small fraction of it consumes GPU resources, enabling uninterrupted training. The safety layer is minimally invasive, correcting robot actions only when they are unsafe. To showcase the safety filter, we also introduce SplatBridge, an open-source software package built with ROS for real-time GSplat mapping for robots. We demonstrate the safety and robustness of our pipeline first in simulation, where our method is 20-50x faster, safer, and less conservative than competing methods based on neural radiance fields. Further, we demonstrate simultaneous GSplat mapping and safety filtering on a drone hardware platform using only on-board perception. We verify that under teleoperation a human pilot cannot invoke a collision.
- We provide ROS 2 nodes in
rosbranch. This includes SplatBridge and the CBF functions. - You can now view the Gaussian Splat and the CBF trajectories in viser. See Visualizing the paths section.
- Provide interactive Colab notebook that allows users to interface with the safety layer on a trained Splat.
This repository is built off of Nerfstudio. Please first follow the installation instructions there before installing any of the dependencies specifically for this repository. Once you have Nerfstudio installed in your Conda environment, install the following dependencies in that environment.
- CLARABEL. This library is for solving the quadratic program.
Our codebase is tested with Python 3.10, Nerfstudio 1.1.5, viser 0.2.7, Clarabel 0.10.0, Numpy 1.26.4, PyTorch 2.1.2, and CUDA 11.8. You can get up and running the quickest following the following commands:
conda create -n safersplat python=3.10
conda activate safersplat
pip install --upgrade pip
git clone https://github.com/chengine/safer-splat.git
cd safer-splat
pip install torch==2.1.2+cu118 torchvision==0.16.2+cu118 --extra-index-url https://download.pytorch.org/whl/cu118
conda install -c "nvidia/label/cuda-11.8.0" cuda-toolkit
pip install nerfstudio==1.1.5
pip install -r requirements.txt
Our datasets are hosted on a Google Drive. The scenes used in the paper are flightgate, statues, stonehenge, adirondacks. The training data is in the data folder, while the model parameters are in outputs. You can drag and drop these folders into your working directory.
Here's an example:
SAFER-Splat
├── data
│ └── flight
│ └── images
│ └── transforms.json
│
├──outputs
│ └── flight
│ └── splatfacto
│ └── 2024-09-12_172434
│ └── nerfstudio_models
| └── config.yml
| └── dataparser_transforms.json # This file contains the transform that transforms from "Data" frame to "Nerfstudio" frame (which is typically a unit box)
├── run.py
After the dependencies and the data is setup, run
python run.py
The most important thing is to ensure that the path in GSplatLoader is pointing to the right model location.
The variants for the distance can be changed. run.py allows you to change the distance type within the script.
Run the visualize.py, setting your scene name and method type properly so that the script correctly extracts the Gaussian Splatting model and the saved trajectory data (that was created through run.py). This script will launch viser and load your trajectories into the visualizer. If you run into issues where viser informs you that add_gaussian_splats does not exist, this means you have an outdated version of viser (likely one that came packaged with Nerfstudio) and will need to upgrade viser. Afterwards, navigate to the specified port in the browser to visualize the trajectories and the ellipsoidal representation of the Gaussian Splat.
To use your own datasets, simply train a Nerfstudio splatfacto model and follow the folder structure as illustrated above. If your images contain pixels that are transparent like in the case of synthetic scenes (i.e. the alpha value is 0), it is recommended to use the instant-ngp-data flag (e.g. ns-train splatfacto --data {DATA} instant-ngp-data) rather than blender-data or the default.
If you found SAFER-Splat useful, consider citing us! Thanks!
@misc{chen2024safersplatcontrolbarrierfunction,
title={SAFER-Splat: A Control Barrier Function for Safe Navigation with Online Gaussian Splatting Maps},
author={Timothy Chen and Aiden Swann and Javier Yu and Ola Shorinwa and Riku Murai and Monroe Kennedy III au2 and Mac Schwager},
year={2024},
eprint={2409.09868},
archivePrefix={arXiv},
primaryClass={cs.RO},
url={https://arxiv.org/abs/2409.09868},
}