Robust Bayesian Scene Reconstruction with Retrieval-Augmented Priors for Precise Grasping and Planning

Herbert Wright, Weiming Zhi, Martin Matak, Matthew Johnson-Roberson, Tucker Hermans (RA-L 2025)

Our method, BRRP, (a), (b) takes an input segmented RGBD image, and (c) retrieves objects to use as a prior, which allow it to (d) reconstruct the scene as well as (e) capture principled uncertainty about object shape.

Here are some useful links for more information:

• Website
• Arxiv

Getting Started

From Source

Clone the repo: start by making sure you have the repo cloned:

git clone https://github.com/Herb-Wright/brrp.git
cd brrp

Dependencies: if you want to be able to use the code, you need to install the dependencies, the best way to do this is to use a conda environment. If you have a cuda-enabled GPU:

conda env create -f environment.yml
conda activate brrp

Of course, you can also use mamba instead. If you don't have a cuda-enabled GPU or want to run the code on the CPU, you can use the env_cpu.yml file instead (creates a brrp_cpu environment).

Installing Project:

pip install .

Running an Example: To run the example script, make sure you have downloaded the scenes and have them in the ~/data/brrp_real_world_scenes directory (or edit the directory in the script). You also need to have the ycb_prior in that same directory. Then, you can run the command:

python scripts/example.py

This should cause a pop-up visualization of the reconstructed meshes and point cloud (after BRRP method is run).

Note: for the other scripts, you may need to install additional dependencies. These should be all listed at the top of the script. You also may need to download datasets for them to run as well.

Pip install from Github

Alternatively, if you have an environment, you can simply pip install directly from Github:

pip install https://github.com/Herb-Wright/brrp.git

Then, you can use it in python:

from brrp.full_method import full_brrp_method

...

weights, hp_trans = full_brrp_method(
    rgb=rgb, 
    xyz=xyz, 
    mask=seg_map, 
    prior_path=abspath("~/data/ycb_prior"), 
    device_str=device_str
)

Docker/Ros Version

Clone and build:

git clone https://github.com/Herb-Wright/brrp.git
cd brrp
docker build -t brrp .

Run the container:

docker run -it   --net=host   --gpus all   --env="NVIDIA_DRIVER_CAPABILITIES=all"   --env="DISPLAY=$DISPLAY"   --env="QT_X11_NO_MITSHM=1"   --volume="/tmp/.X11-unix:/tmp/.X11-unix:rw"   --volume="$XAUTHORITY:/root/.Xauthority:rw"   brrp bash

You might need to set some ROS env variables if communicating across machines.

Then to run the node:

python scripts/run_ros.py
# OR for PointSDF:
python scripts/run_ros_pointsdf.py

Datasets

• YCB Prior - Google Drive Link
• PointSDF Weights - Google Drive Link

Citation

Please consider citing our work:

@article{wright2025robust,
  title={Robust Bayesian Scene Reconstruction With Retrieval-Augmented Priors for Precise Grasping and Planning},
  author={Wright, Herbert and Zhi, Weiming and Matak, Martin and Johnson-Roberson, Matthew and Hermans, Tucker},
  journal={IEEE Robotics and Automation Letters},
  volume={11},
  number={1},
  pages={49--56},
  year={2025},
  publisher={IEEE}
}