Control mouse by moving objects in front of webcam using Tensorflow Object Detection API
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
docs
graphs
tensormouse
tests
LICENSE
README.md
__init__.py
environment-win.yml
environment.yml
tensormouse.py

README.md

TensorMouse

Control your mouse cursor by moving objects in front of webcam using Tensorflow Object Detection API.

alt text

TensorMouse allows you to control your cursor by moving random objects (like cups, apples or bananas) in front of webcam to move your cursor as a replacement for mouse or touchpad.

Tensorflow Object Detection API trained on the COCO dataset supports up to 90 different types of objects that can be used to move the cursor.

Project includes ssd_mobilenet_v1_coco_11_06_2017 Tensorflow frozen graph trained on the mobilenet deep neural network by default, but supports any other graph Tensorflow Object Detection API graph by supplying the optional argument --graphpath

TensorMouse also supports clicks currently triggered by clicking CTRL key and mouse drags triggered by ALT key.

Getting Started

Linux/MacOS

  1. Make sure you have anaconda installed
  2. Open a terminal and navigate to project root
  3. conda env create -f environment.yml
  4. source activate tensormouse

Windows

  1. Make sure you have anaconda installed
  2. Open a terminal and navigate to project root
  3. conda env create -f environment-win.yml
  4. activate tensormouse

Usage

python tensormouse.py

Optional arguments (default value):

  • --source=0 Device index of the camera
  • --object=cup Object name - name of object to track (see graphs/labels.json for available options)
  • --graphpath=ssd_mobilenet_v1_coco_11_06_2017 Path to frozen tensofrlow graph

Wait for ✓ TensorMouse started successfully! message to use the application.

Use CTRL for clicks and ALT for cursor draging. Exit by clicking CAPS_LOCK

Requirements

Technologies Used

Notes

  • Make sure you have good lighting when using TensorMouse
  • Frame rate will be very low on low performing machines
  • Object deteciton on screen edges is poor. (todo: scale cursor movement so that near edge object movements will move cursor to edge)

Copyright

See LICENSE for details. Copyright (c) 2017 Tadej Magajna.