Home
- TensorFlow
- PoseNet pose estimation library
- VideoJS
- videojs-record
- Video Documentation
- Web Workers
- React
- Redux
- Express
- Sequelize
- PostgreSQL
The main challenge we faced when building Coreo was integrating video recording with pose detection. Our initial idea was to capture a dance video, pass it through a pose detector, and then grade its outputs compared to another video. This process ended up being a little more complicated.
When the dancer is recording a video, we were actually displaying three frames at a time: one from cloudinary, our video database, one from the webcam, and one from a canvas where skeletons are drawn over the recording. To give the dancer quick feedback we needed to generate those skeletons without sending the video to the server. This meant that on the dancer’s computer we needed to detect all the poses within it, compare those poses with the poses of the choreographer’s video, and draw the poses over the webcam.
Web Workers are JavaScript's version of multithreading. Because JavaScript was designed as a single-threaded language, workers are a bit more restricted than threads in most other languages. You can think of a worker as a hidden second browser window that can pass messages to and from the main window.
In Coreo, once a user starts recording, Videojs fires an event every 250ms which contains the state of the recording; on receiving the event, we extract the image data associated with the current frame and send it to our worker. The worker then runs the image data through PoseNet and stores the output in its local state. Once the recording finishes, we send a 'finished' message to our worker, which responds by giving us an array containing all the processed data so far. We run that array through our pairing algorithm which allows us to play back the recording with the choreographer’s skellie overlaid atop the dancer, as you saw.
With the help of the worker, we were able to reduce the post-recording wait time from about 5 seconds down to less than one.
- Video hosting: Cloudinary
- Video processing: ffmpeg
- Database: PostgreSQL
- Deployment: Heroku