Skip to content

Making Sense of Audio Vibration for Liquid Height Estimation in Robotic Pouring


Notifications You must be signed in to change notification settings


Folders and files

Last commit message
Last commit date

Latest commit



15 Commits

Repository files navigation

Audio Pouring

We propose to make use of audio vibration sensing with a deep neural network named PouringNet to predict the liquid height from the audio fragment during the robotic pouring task. PouringNet is trained on our collected real-world pouring dataset with multimodal sensing data, which contains more than 3000 recordings of audio, force feedback, video and trajectory data of the human hand that performs the pouring task. Each record represents a complete pouring procedure. We conduct several evaluations on PouringNet with our dataset and robotic hardware. The results demonstrate that our PouringNet generalizes well across different liquid containers, positions of the audio receiver, initial liquid heights and types of liquid, and facilitates a more robust and accurate audio-based perception for robotic pouring.




  • Pouring with different cups

    All cups

    The cups are marked as #1-6 from left to right. And only cup #1/2/3 present in the dataset, while others are not included.
  • Pouring with different initial heights on cup #3.

  • Pouring with different microphone positions.


  1. Install Anaconda and PyTorch:

    conda upgrade --all
    conda create -n pouring python=2.7 numpy ipython matplotlib mayavi yaml lxml seaborn
    conda activate pouring
    conda install -c conda-forge librosa trimesh pyglet
    pip install rospkg tensorboardx pyassimp==4.1.3
    # cpu:
    conda install tensorflow
    conda install pytorch-cpu torchvision-cpu -c pytorch
    # gpu:
    conda install tensorflow-gpu
    conda install pytorch torchvision cudatoolkit=9.0 -c pytorch
  2. Clone and install this repository:

    git clone
    cd AudioPouring
    cd audio_pouring
    python develop
  3. Install dependencies:

    • Install portaudio dependencies: according to this, install packages in this order will not remove any other packages
      sudo apt install libsndfile1-dev
      sudo apt install libjack-jackd2-dev
      sudo apt install portaudio19-dev
    • Make sure your current user name is in audio group
    • Other dependencies (only for robot experiment):
  4. Install following required ROS packages:

Run demo

  1. Bring up audio publishing node:

    roslaunch portaudio_transport publish.launch
  2. Bring up a scale to get the ground truth height, if you do not have a ROS based scale, directly go to step 4.

  3. Run demo code

    cd ${AUDIO_POURING_DIR}/audio_pouring
    python --cuda --bottle=1 --cavity-height=50
  4. (In case when a ROS-based scale is not available) you can also use a normal scale and check the pouring result with the code below:

    from audio_pouring.utils.utils import weight2height
    print(weight2height(cup_id="1", cur_weight=0.02))

Network Training

  1. Data preparation: generate a ~4s segment from a whole pouring sequence (pickle files):

    cd ${AUDIO_POURING_DIR}/audio_pouring/model
    python train mt
    python test mt
  2. Data preparation: generate npy file list from that segment

    cd ${AUDIO_POURING_DIR}/audio_pouring/utils
  3. Network training

    cd ${AUDIO_POURING_DIR}/audio_pouring
    python --fixed --cuda --gpu=0 --bottle-train=0 --lstm --bs=32
    #--fixed       : the input audio length is fixed (must set)
    #--lstm        : set to use lstm or gru
    #--bs          : set batch size
    #--bottle-train: set bottle id, if set to 0, then all the data are used

Generate your own bottle config file

  • create a bottle config csv file and put it at ${AUDIO_POURING_DIR}/audio_pouring/config/bottles
  • modify and run the code below:
    cd ${AUDIO_POURING_DIR}/audio_pouring/utils



  • Containing video, audio, force/torque and position information collected during human pouring.
  • We plan to release our dataset progressively. Now the audio part is available at Google Drive.


If you find this paper and code useful in your research, please consider citing:

  title={Making Sense of Audio Vibration for Liquid Height Estimation in Robotic Pouring},
  author={Liang, Hongzhuo and Li, Shuang and Ma, Xiaojian  and Hendrich, Norman and Gerkmann, Timo and Sun, Fuchun and Zhang, Jianwei},
  booktitle={IEEE International Conference on Intelligent Robots and Systems (IROS)},


Making Sense of Audio Vibration for Liquid Height Estimation in Robotic Pouring







No releases published


No packages published