Intelligent Interactive Systems Project

What's this

This project aims to recognize 6 pre-defined gestures from videos. As following images showing, there are 'fist_dorsal', 'fist_palm', 'open_dorsal', 'open_palm', 'three_fingers_dorsal', 'three_fingers_palm'.

Quick start

1. Install Opencv

   You should install Opencv in your environment.

2. git clone this project

   git clone https://github.com/nanguoyu/Intelligent-Interactive-Systems-Project.git
   cd Intelligent-Interactive-Systems-Project
   pip install -r requirements.txt
   pip install .

3. Download pre-trained models

   python DownloadWeights.py
   

4. Start the API system and its extraction and recognition server:

   If your environment is windows 64, you should run start_app_win64.bat and do not close its window.

   If your environment is mac os/ linux, you should run start_app_macos.sh and do not close its window.

5. Run tests

   python tests/test_app.py
   

   You can check tests/test_app.py to send your image to recognize gestures.

   import base64
   import json
   import requests
   
   img_file = "open_dorsal.jpg"
   # URL = 'http://127.0.0.1:5000/gesture'
   URL = 'http://127.0.0.1:5000/end2end'
   """ post image and return the response """
   with open(img_file, 'rb') as f:
       img = base64.b64encode(f.read()).decode()
   response = requests.post(URL, data={"image": img})
   print(json.loads(response.content.decode('utf-8')))

Features

Bases on a unpublished data set, we are implementing these parts.

landmark detection and gesture recognition

• A landmark detection system
• A machine learning based gesture recognition system
• A Kotlin program to control Furhat Robot
• An API for communication between the Kotlin program and Python Backend

Specialization part.

• An end-to-end system (CNN)

Demo for the end2end detection model

Reference

• The Supervised Machine Learning book(An upcoming textbook)
• SemiFlow