Skip to content

lianghongzhuo/MultimodalPouring

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

4 Commits
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Multimodal Pouring

In this paper, we focus on the robust and accurate estimation of liquid height for robotic pouring. However, vision-based methods often fail in occluded conditions while audio-based methods cannot work well in a noisy environment. We instead propose a multimodal pouring network (MP-Net) that is able to robustly predict liquid height by conditioning on both audition and haptics input. MP-Net is trained on a self-collected multimodal pouring dataset. This dataset contains 300 robot pouring recordings with audio and force/torque measurements for three types of target containers. We also augment the audio data by inserting robot noise. We evaluated MP-Net on our collected dataset and a wide variety of robot experiments. Both network training results and robot experiments demonstrate that MP-Net is robust against noise and changes to the task and environment. Moreover, we further combine the predicted height and force data to estimate the shape of the target container.

Overview

Pipeline

Target containers

containers

Dataset and experiment setup

setup

Please contact Hongzhuo Liang (liang[at]informatik.uni-hamburg.de) for the download link.

Network

network

Installation

  1. Install Anaconda and PyTorch:

  2. Install python packages in a new environment

    conda upgrade --all
    conda create -n pouring python=[PYTHON_VERSION] numpy ipython matplotlib mayavi yaml lxml seaborn pyyaml
    conda activate pouring
    conda install -c conda-forge librosa trimesh pyglet
    conda install -c conda-forge "h5py>=2.9=mpi*"
    pip install rospkg numpy-stl tensorboardx pyquaternion pyassimp==4.1.3

    Note: PYTHON_VERSION can be 3.7 if you do not need to use this package with ROS, otherwise use 2.7 or use python at /usr/bin/python

  3. Clone and install this repository:

    git clone https://github.com/lianghongzhuo/MultimodalPouring.git
    cd MultimodalPouring
    MP_POURING_DIR=${PWD}
    cd audio_pouring
    python setup.py develop
  4. Install dependencies:

    • Install portaudio dependencies: according to this, install packages in this order will not remove any other packages
      sudo apt install libsndfile1-dev
      sudo apt install libjack-jackd2-dev
      sudo apt install portaudio19-dev
    • Make sure your current user name is in audio group
    • Other dependencies (only for robot experiment):
      cd ${MP_POURING_DIR}
      sh audio_pouring_install.sh
  5. Install following required ROS packages:

Run demo

  1. Bring up audio publishing node:

    roslaunch portaudio_transport publish.launch
  2. Bring up the Force/Torque sensor

  3. Bring up a scale to get the ground truth height, if you do not have a ROS based scale, directly go to step 4.

  4. Run demo code

    cd ${MP_POURING_DIR}/audio_pouring
    python demo.py --cuda --bottle=1 --cavity-height=50 --multi --scale
  5. (In case when a ROS-based scale is not available) you can also use a normal scale and check the pouring result with the code below:

    from audio_pouring.utils.utils import weight2height
    print(weight2height(cup_id="1", cur_weight=0.02))

Network Training

  1. Data preparation: generate a 4s segment from a whole pouring sequence (pickle files):

    cd ${MP_POURING_DIR}/audio_pouring/model
    ./run_long_preprocess.sh
  2. Network training

    cd ${MP_POURING_DIR}/audio_pouring
    # run
    ./run_network_multi.sh
    # or 
    ./run_network_audio.sh
    # args:
    # --multi       : use multi modal date or audio only
    # --lstm        : set to use lstm or gru
    # --bs          : set batch size
    # --bottle-train: set bottle id, should set to robot_pouring
    # --snr_db      : set snr_db value (1000 means no noise, -1000 means mixed noise from 0 to 20)

Generate your own bottle config file

  • create a bottle config csv file and put it at ${MP_POURING_DIR}/audio_pouring/config/bottles
  • modify and run the code below:
    cd ${MP_POURING_DIR}/audio_pouring/utils
    python generate_bottle_config.py

Citation

If you find this paper and code useful in your research, please consider citing:

@inproceedings{liang2020MultimodalPouring,
  title     = {Robust Robotic Pouring using Audition and Haptics},
  author    = {Liang, Hongzhuo and Zhou, Chuangchuang and Li, Shuang and Ma, Xiaojian and Hendrich, Norman and Gerkmann, Timo and Sun, Fuchun and Zhang, Jianwei},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
  year      = {2020},
  pages     = {10880-10887}
}