In this paper, we focus on the robust and accurate estimation of liquid height for robotic pouring. However, vision-based methods often fail in occluded conditions while audio-based methods cannot work well in a noisy environment. We instead propose a multimodal pouring network (MP-Net) that is able to robustly predict liquid height by conditioning on both audition and haptics input. MP-Net is trained on a self-collected multimodal pouring dataset. This dataset contains 300 robot pouring recordings with audio and force/torque measurements for three types of target containers. We also augment the audio data by inserting robot noise. We evaluated MP-Net on our collected dataset and a wide variety of robot experiments. Both network training results and robot experiments demonstrate that MP-Net is robust against noise and changes to the task and environment. Moreover, we further combine the predicted height and force data to estimate the shape of the target container.
- Project website: https://lianghongzhuo.github.io/project/MultimodalPouring
- Preprint: https://arxiv.org/abs/2003.00342
- Video: https://youtu.be/_U7zTyS338I
Please contact Hongzhuo Liang (liang[at]informatik.uni-hamburg.de) for the download link.
-
Install python packages in a new environment
conda upgrade --all conda create -n pouring python=[PYTHON_VERSION] numpy ipython matplotlib mayavi yaml lxml seaborn pyyaml conda activate pouring conda install -c conda-forge librosa trimesh pyglet conda install -c conda-forge "h5py>=2.9=mpi*" pip install rospkg numpy-stl tensorboardx pyquaternion pyassimp==4.1.3
Note: PYTHON_VERSION can be 3.7 if you do not need to use this package with ROS, otherwise use 2.7 or use python at
/usr/bin/python
-
Clone and install this repository:
git clone https://github.com/lianghongzhuo/MultimodalPouring.git cd MultimodalPouring MP_POURING_DIR=${PWD} cd audio_pouring python setup.py develop
-
Install dependencies:
- Install portaudio dependencies: according to this,
install packages in this order will not remove any other packages
sudo apt install libsndfile1-dev sudo apt install libjack-jackd2-dev sudo apt install portaudio19-dev
- Make sure your current user name is in
audio
group - Other dependencies (only for robot experiment):
cd ${MP_POURING_DIR} sh audio_pouring_install.sh
- Install portaudio dependencies: according to this,
install packages in this order will not remove any other packages
-
Install following required ROS packages:
-
Bring up audio publishing node:
roslaunch portaudio_transport publish.launch
-
Bring up the Force/Torque sensor
-
Bring up a scale to get the ground truth height, if you do not have a ROS based scale, directly go to step 4.
-
Run demo code
cd ${MP_POURING_DIR}/audio_pouring python demo.py --cuda --bottle=1 --cavity-height=50 --multi --scale
-
(In case when a ROS-based scale is not available) you can also use a normal scale and check the pouring result with the code below:
from audio_pouring.utils.utils import weight2height print(weight2height(cup_id="1", cur_weight=0.02))
-
Data preparation: generate a 4s segment from a whole pouring sequence (pickle files):
cd ${MP_POURING_DIR}/audio_pouring/model ./run_long_preprocess.sh
-
Network training
cd ${MP_POURING_DIR}/audio_pouring # run ./run_network_multi.sh # or ./run_network_audio.sh # args: # --multi : use multi modal date or audio only # --lstm : set to use lstm or gru # --bs : set batch size # --bottle-train: set bottle id, should set to robot_pouring # --snr_db : set snr_db value (1000 means no noise, -1000 means mixed noise from 0 to 20)
- create a bottle config csv file and put it at
${MP_POURING_DIR}/audio_pouring/config/bottles
- modify and run the code below:
cd ${MP_POURING_DIR}/audio_pouring/utils python generate_bottle_config.py
If you find this paper and code useful in your research, please consider citing:
@inproceedings{liang2020MultimodalPouring,
title = {Robust Robotic Pouring using Audition and Haptics},
author = {Liang, Hongzhuo and Zhou, Chuangchuang and Li, Shuang and Ma, Xiaojian and Hendrich, Norman and Gerkmann, Timo and Sun, Fuchun and Zhang, Jianwei},
booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
year = {2020},
pages = {10880-10887}
}