Skip to content
Branch: master
Find file History
Latest commit 0f4356b Oct 4, 2019
Permalink
Type Name Latest commit message Commit time
..
Failed to load latest commit information.
README.md Add online demo. Sep 30, 2019
main.py Minor change. Oct 4, 2019
mobilenet_tsm_tvm_cuda.json Add online demo. Sep 30, 2019
mobilenet_v2_tsm.py Add online demo. Sep 30, 2019

README.md

TSM Online Hand Gesture Recognition Demo

@inproceedings{lin2019tsm,
  title={TSM: Temporal Shift Module for Efficient Video Understanding},
  author={Lin, Ji and Gan, Chuang and Han, Song},
  booktitle={Proceedings of the IEEE International Conference on Computer Vision},
  year={2019}
}

tsm-demo

See the [full video] of our demo on NVIDIA Jetson Nano.

Overview

We show how to deploy an online hand gesture recognition system on NVIDIA Jetson Nano. The model is based on MobileNetV2 backbone with Temporal Shift Module (TSM) to model the temporal relationship. It is compiled with TVM [1] for acceleration.

The model can achieve real-time recognition. Without considering the data IO time, it can achieve >70 FPS on Nano GPU.

[1] Tianqi Chen et al., TVM: An automated end-to-end optimizing compiler for deep learning, in OSDI 2018

Model

We used an online version of Temporal Shift Module in this demo. The model design is shown below:

After compiled with TVM, our model can efficient run on low-power devices.

Step-by-step Tutorial

We show how to set up the environment on Jetson Nano, compile the PyTorch model with TVM, and perform the online demo from camera streaming.

  1. Get an NVIDIA Jeston Nano board (it is only $99!).
  2. Get a micro SD card and burn the Nano system image into it following here. Insert the card and boot the Nano. Note: you may want to get a power adaptor for a stable power supply.
  3. Build OpenCV 4.0.0 using this script, so that we can enable camera access (It may take a while due to the weak CPU). You also need add cv2 package to path import search path.
export PYTHONPATH=/usr/local/python
  1. Follow here to install PyTorch and torchvision.
  2. Build TVM with following commands
sudo apt install llvm # install llvm which is required by tvm
git clone https://github.com/dmlc/tvm.git
cd tvm
git submodule update --init
mkdir build
cp cmake/config.cmake build/
cd build
#[
#edit config.cmake to change
# 32 line: USE_CUDA OFF -> USE_CUDA ON
#104 line: USE_LLVM OFF -> USE_LLVM ON
#]
cmake ..
make -j4
cd ..
cd python; sudo python3 setup.py install; cd ..
cd nnvm/python; sudo python3 setup.py install; cd ../..
cd topi/python; sudo python3 setup.py install; cd ../..
  1. Install ONNX
# install onnx
sudo apt-get install protobuf-compiler libprotoc-dev
pip3 install onnx
  1. export cuda toolkit binary to path
export PATH=$PATH:/usr/local/cuda/bin
  1. Finally, run the demo. The first run will compile the PyTorch TSM model into TVM binary first and then run it. Later run will directly execute the compiled TVM model.
python3 main.py

Press Q or Esc to quit. Press F to enter/exit full-screen.

Supported Gestures

  • No gesture
  • Stop Sign
  • Drumming Fingers
  • Thumb Up
  • Thumb Down
  • Zooming In With Full Hand
  • Zooming In With Two Fingers
  • Zooming Out With Full Hand
  • Zooming Out With Two Fingers
  • Swiping Down
  • Swiping Left
  • Swiping Right
  • Swiping Up
  • Sliding Two Fingers Down
  • Sliding Two Fingers Left
  • Sliding Two Fingers Right
  • Sliding Two Fingers Up
  • Pulling Hand In
  • Pulling Two Fingers In

Contact

For any problems, contact:

Ji Lin, jilin@mit.edu

Yaoyao Ding, yyding@mit.edu

You can’t perform that action at this time.