This repository is contains the code for the experiments shown in the paper "Robotic Nonprehensile Transportation with a Hanging Tray", to appear in the IEEE/ASME International Conference on Advanced Intelligent Mechatronics, 2026.
The YOLO model used to segment people with a raised hand can be trained using the self-contained code in the yolo directory.
This project was built and run on a laptop running Ubuntu 20.04 with ROS1 Noetic. It controls a Ridgeback mobile base using a Femto Bolt camera and a Vicon motion capture system.
Follow the instructions to setup mobile_manipulation_central and the ROS1 SDK for the Orbbec Femto Bolt camera.
Clone the repository into your catkin workspace:
cd catkin_ws/src
git clone https://github.com/learnsyslab/visual_servor
Install additional dependencies:
# apt packages
sudo apt install python3-opencv ros-noetic-cv-bridge ros-noetic-ros-numpy
# pip packages
cd visual_servor
pip install -r requirements.txt
Finally, build your catkin workspace.
Simulation experiments are run using scripts/run_simulation_experiment.py.
You first need to measure the rest position of the hanging tray relative to the
robot's end effector. Use the script scripts/calibrate_hanging_tray.py and
save the result in the config directory.
Trials of transporting objects with different motion profiles and trays are
done using the script scripts/run_hardware_experiment.py. Record the data to
a rosbag by running scripts/record_bag.py at the same time in a separate
terminal.
To calculate the friction coefficient for different tray and object
combinations, record a bag during which you slowly manually tilt the tray with
the object on it until the object starts to slide. Then use
scripts/compute_mu_from_vicon_data.py to calculate the friction coefficient
corresponding to the tilt angle. Note that friction coefficients are not
required for control.
In three terminals, respectively run:
roslaunch visual_servor visual_servor.launchrosrun visual_servor waiter_vision_node.py --displayrosrun visual_servor waiter_control_node.py
MIT. See the LICENSE file.