Breakout-YOLO is a classic browser game that can be controlled by gestures. See poster and post on habr(russian) for details.

Tasks

• structure code
• add bounding box rendering in game
• move detection part to worker.js and use Worker API for parallelization of object detection and drawing
• add images without gestures to dataset(and train model on this) to decrease number of false positives
• add more gestures and game bonuses

Getting started

Firefox(version >= 76.0.1)(recommended) and Chromium(version >= 81.0.4044.138) are supported.

Press demo and game to immediately enjoy the process(loading may take some time).

If you want(for some reason) to run all things locally follow this instruction:

For now only Flask is required. Install it in a way appropriate for your OS. For Linux just run following comands:

Conda (Recommended)

conda env create -f breakout.yml  
conda activate breakout  

or

Pip

python -m venv breakout  
source ./breakout/bin/activate  
pip install -r requirements.txt

After installation run app.py:

python app.py

• go to http://127.0.0.1:5000/docs/demo.html in your browser to run demonstration
• go to http://127.0.0.1:5000/docs/index.html in your browser to play game

If you get

OSError: [Errno 98] Address already in use

so there is another process listening on the port 5000. Just find its PID and kill by:

lsof -i :5000
kill -9 PID

Performance

Device+CPU+GPU	FPS
ASUS Vivobook S15, i7 1.8GHz, MX250	37
Lenovo IdeaPad 520 i5 2.5GHz, 940MX	25
Acer Aspire V5, i3 1.8GHz, GT 740M	16

Game Controls

• use «circle» gesture to move cursor in main menu or to unpause game
• use «finger up» gesture to click on buttons in main menu or to pause game
• use «fist» gesture to move paddle
• use «pistol» gesture to detach ball from sticky paddle
  

Demonstration:

Known issues

• In Chromium cursor movement smoothing may not work

Acknowledgments

• Yolov3 Keras implementation
• AlexeyAB darknet repository
• Yolo v3 official paper
• TensorFlowJS
• Dude who made a big contribution to the game script code