Panoramic Video People Tracking

This work builds on top of Monocular Person Tracking and Identification framework, proposed by Koide at. al, to adapt it working with panoramic images. The main advantage of using panoramic video is that people cannot leave the field of view of the camera (if not occluded). This fact leads to a better tracking stability.

Example of our Panoramic People Tracking

We also provide a dataset of panoramic videos, labelled everi 10-20 frames with a target subject. The dataset is available here for download. The password is iaslabPVPT.

If you use our algorithms or our dataset in your reasearch, please cite our article:

@InProceedings{10.1007/978-3-031-22216-0_28,
author="Bacchin, Alberto
and Berno, Filippo
and Menegatti, Emanuele
and Pretto, Alberto",
editor="Petrovic, Ivan
and Menegatti, Emanuele
and Markovi{\'{c}}, Ivan",
title="People Tracking in Panoramic Video for Guiding Robots",
booktitle="Intelligent Autonomous Systems 17",
year="2023",
publisher="Springer Nature Switzerland",
pages="407--424",
isbn="978-3-031-22216-0"
}

The work was develepoed in collaboration with filber97.

Installation

ATTENTION: we are working on some installation and compatibility issue. At the moment, we tested it on ROS Melodic. For further details, see this source.

Update system

sudo apt-get update
sudo apt-get upgrade

Flann & OpenBlas

sudo apt-get install libflann-dev libopenblas-base libopenblas-dev

dlib

mkdir ~/libs && cd ~/libs

wget http://dlib.net/files/dlib-19.17.tar.bz2
tar xvf dlib-19.17.tar.bz2

echo "export DLIB_ROOT=~/dlib-19.17" >> ~/.bashrc
source ~/.bashrc

tensorflow

sudo apt install python-pip
sudo pip install -U pip setuptools numpy
sudo pip install -U --extra-index-url https://developer.download.nvidia.com/compute/redist/jp33 tensorflow-gpu

ROS

See http://wiki.ros.org/melodic/Installation/Ubuntu

!! Install tensorflow before ROS otherwise you see some dependency issues...

tf-pose-estimation

# llvmlite requires llvm >= 7.0
# for Xavier
sudo apt install llvm-7.0

# for TX2 with Jetpack 3.3
mkdir ~/libs && cd ~/libs
wget http://releases.llvm.org/7.0.1/llvm-7.0.1.src.tar.xz
tar xvf tar xvf llvm-7.0.1.src.tar.xz
cd llvm-7.0.1.src && mkdir build && cd build
cmake -DCMAKE_BUILD_TYPE=Release -DLLVM_USE_LINKER=gold -DCMAKE_INSTALL_PREFIX=/usr/local ..
make -j6
sudo make install

cd catkin_ws/src
git clone https://github.com/koide3/tf-pose-estimation

# Follow the official installation instruction
cd tf-pose-estimation
sudo pip install -r requirements.txt

# In case you meet scipy install issue
# sudo apt-get install gfortran

cd tf_pose/pafprocess
swig -python -c++ pafprocess.i && python setup.py build_ext --inplace

cd ../..
sudo python setup.py install

# You may need to comment out python package dependencies in setup.py like as follows:
# 21: #   'dill==0.2.7.1',

cd models/graph/cmu
bash download.sh

other packages

cd catkin_ws/src
git clone https://github.com/koide3/open_face_recognition.git
git clone https://github.com/koide3/ccf_person_identification.git

sudo apt-get install ros-kinetic-image-transport ros-kinetic-image-transport-plugins ros-melodic-image-proc

Principal packages

git clone https://github.com/bach05/PanoramicVideoPeopleTracking.git 
catkin build

Usage

Detection with ROI

roslaunch tracking_demo omnidirectional_roi.launch
roslaunch tracking_demo start_robot.launch webcam:=true publish_dummy_frames:=true

Detection with TILES

roslaunch tracking_demo omnidirectional_tiles.launch
roslaunch tracking_demo start_robot.launch webcam:=true publish_dummy_frames:=true