DepthJS is a browser extension (currently Chrome) that allows the Microsoft Kinect to talk to any web page. It provides the low-level raw access to the Kinect as well as high-level hand gesture events to simplify development.
Everything is VERY alpha. Hand detection and tracking are reasonably robust, but recognizing gestures is less reliable.
So far we can recognize the following:
- Presence of hand (registration)
- Removal of hand (unregistration)
- Hand movement
- Quick open and close of hand ("hand click")
- Large swipe up/down/left/right
We want to recognize:
- Two hands
- Pushing and pulling
- Z-depth in general
- Individual fingers
- Finger-based hand gestures (e.g. peace sign, etc)
BACKEND: Kinect ====(libfreekinect)====> OpenFramework+OpenCV =====(0MQ)====> Tornado
In addition to the obvious improvements to our gesture recognition, we need to make the install process easier for end-users.
We like that the backend is very modular, so that anything (not just DepthJS) can connect via 0MQ or WebSockets to the C++ Kinect+OpenCV backend.
However, this is not a user friendly setup. We want to eventually integrate the entire driver+backend into an NSAPI-based native code extension such that we can distribute the extension "batteries-included" as a one-click install. Eventually it should be hosted on the various web browser (Safari, Firefox, and Chome) extension galleries.