Skip to content

Latest commit

 

History

History
 
 

online_demo

Folders and files

NameName
Last commit message
Last commit date

parent directory

..

README.md

README.md

 
 

main.py

main.py

 
 

mobilenet_tsm_tvm_cuda.json

mobilenet_tsm_tvm_cuda.json

 
 

mobilenet_v2_tsm.py

mobilenet_v2_tsm.py

 
 

README.md

TSM Online Hand Gesture Recognition Demo

@inproceedings{lin2019tsm,
  title={TSM: Temporal Shift Module for Efficient Video Understanding},
  author={Lin, Ji and Gan, Chuang and Han, Song},
  booktitle={Proceedings of the IEEE International Conference on Computer Vision},
  year={2019}
}

tsm-demo

See the [full video] of our demo on NVIDIA Jetson Nano.

Overview

We show how to deploy an online hand gesture recognition system on NVIDIA Jetson Nano. The model is based on MobileNetV2 backbone with Temporal Shift Module (TSM) to model the temporal relationship. It is compiled with TVM [1] for acceleration.

The model can achieve real-time recognition. Without considering the data IO time, it can achieve >70 FPS on Nano GPU.

[1] Tianqi Chen et al., TVM: An automated end-to-end optimizing compiler for deep learning, in OSDI 2018

Model

We used an online version of Temporal Shift Module in this demo. The model design is shown below:

After compiled with TVM, our model can efficient run on low-power devices.

Step-by-step Tutorial

We show how to set up the environment on Jetson Nano, compile the PyTorch model with TVM, and perform the online demo from camera streaming.

  1. Get an NVIDIA Jeston Nano board (it is only $99!).
  2. Get a micro SD card and burn the Nano system image into it following here. Insert the card and boot the Nano. Note: you may want to get a power adaptor for a stable power supply.
  3. Build OpenCV 4.0.0 using this script, so that we can enable camera access (It may take a while due to the weak CPU). You also need add cv2 package to path import search path.
export PYTHONPATH=/usr/local/python
  1. Follow here to install PyTorch and torchvision.
  2. Build TVM with following commands
sudo apt install llvm # install llvm which is required by tvm
git clone https://github.com/apache/incubator-tvm.git
cd incubator-tvm
git submodule update --init
mkdir build
cp cmake/config.cmake build/
cd build
#[
#edit config.cmake to change
# 32 line: USE_CUDA OFF -> USE_CUDA ON
#104 line: USE_LLVM OFF -> USE_LLVM ON
#]
cmake ..
make -j4
cd ..
cd python; sudo python3 setup.py install; cd ..
cd topi/python; sudo python3 setup.py install; cd ../..
  1. Install ONNX
# install onnx
sudo apt-get install protobuf-compiler libprotoc-dev
pip3 install onnx
  1. export cuda toolkit binary to path
export PATH=$PATH:/usr/local/cuda/bin
  1. Finally, run the demo. The first run will compile the PyTorch TSM model into TVM binary first and then run it. Later run will directly execute the compiled TVM model.
python3 main.py

Press Q or Esc to quit. Press F to enter/exit full-screen.

Supported Gestures

  • No gesture
  • Stop Sign
  • Drumming Fingers
  • Thumb Up
  • Thumb Down
  • Zooming In With Full Hand
  • Zooming In With Two Fingers
  • Zooming Out With Full Hand
  • Zooming Out With Two Fingers
  • Swiping Down
  • Swiping Left
  • Swiping Right
  • Swiping Up
  • Sliding Two Fingers Down
  • Sliding Two Fingers Left
  • Sliding Two Fingers Right
  • Sliding Two Fingers Up
  • Pulling Hand In
  • Pulling Two Fingers In

Contact

For any problems, contact:

Ji Lin, jilin@mit.edu

Yaoyao Ding, yyding@mit.edu