Browser game where a vehicle is driven through the camera using the ResNet model (Deep Residual Network) to estimate the position of the hands.
The application is written on top of this codepen.
Install dependencies and prepare the build directory:
yarnTo watch files for changes, and launch a dev server:
yarn watchThis application is written in vanilla js with HTML and CSS without any framework, using yarn as dependency manager.
A few resources about HTML+CSS+JS: https://www.w3schools.com/ How to use Yarn: https://classic.yarnpkg.com/en/docs/managing-dependencies/
TensorFlow.js is a library for machine learning in JavaScript.
A few resourses to get started with Tensorflow.js.
Residual Networks, or ResNets, learn residual functions with reference to the layer inputs, instead of learning unreferenced functions. Instead of hoping each few stacked layers directly fit a desired underlying mapping, residual nets let these layers fit a residual mapping. They stack residual blocks ontop of each other to form network: e.g. a ResNet-50 has fifty layers using these blocks.
More info in this paper.
Using the PoseNet library of tensorflow.js we can obtain the positions of the different parts of the body during the streaming of the video in real time.
In this project, the position of the wrists is estimated, and then the angle formed by the segment between them is calculated to infer if the "steering wheel" moved to the left or right.
Finally we use the result to alter the movement of the vehicle on the road.