Control your mouse cursor by moving objects in front of webcam using Tensorflow Object Detection API.
TensorMouse allows you to control your cursor by moving random objects (like cups, apples or bananas) in front of webcam to move your cursor as a replacement for mouse or touchpad.
Tensorflow Object Detection API trained on the COCO dataset supports up to 90 different types of objects that can be used to move the cursor.
Project includes ssd_mobilenet_v1_coco_11_06_2017 Tensorflow frozen graph trained on the mobilenet deep neural network by default, but supports any other graph Tensorflow Object Detection API graph by supplying the optional argument --graphpath
TensorMouse also supports clicks currently triggered by clicking CTRL key and mouse drags triggered by ALT key.
- Make sure you have anaconda installed
- Open a terminal and navigate to project root
conda env create -f environment.ymlsource activate tensormouse
- Make sure you have anaconda installed
- Open a terminal and navigate to project root
conda env create -f environment-win.ymlactivate tensormouse
python tensormouse.py
Optional arguments (default value):
--source=0Device index of the camera--object=cupObject name - name of object to track (see graphs/labels.json for available options)--graphpath=ssd_mobilenet_v1_coco_11_06_2017Path to frozen tensofrlow graph
Wait for ✓ TensorMouse started successfully! message to use the application.
Use CTRL for clicks and ALT for cursor draging. Exit by clicking CAPS_LOCK
- Tensorflow Object Detection API
- COCO dataset
- datitran/object_detector_app
- Make sure you have good lighting when using TensorMouse
- Frame rate will be very low on low performing machines
- Object deteciton on screen edges is poor. (todo: scale cursor movement so that near edge object movements will move cursor to edge)
See LICENSE for details. Copyright (c) 2017 Tadej Magajna.