Code for Chapter 10: AI in the Browser with TensorFlow.js and ml5.js
Every single individual who uses a computer or a smartphone uniformly has access to one software program—their browser. Reach all those users with browser-based deep learning libraries including TensorFlow.js and ml5.js. Guest author Zaid Alyafeai walks us through techniques and tasks such as body pose estimation, generative adversarial networks (GANs), image-to-image translation with Pix2Pix and more, running not on a server but in the browser itself. Bonus: Hear from TensorFlow.js and ml5.js teams on how the projects incubated. Read online here.
Go through the code in the following order:
- Running Pretrained Models using TensorFlow.js: We load a pretrained MobileNet model and run it in the browser.
- Teachable Machine: We take it up a notch and train our own models directly in the browser using input from the webcam.
- Running Pretrained Model using ml5.js: ml5.js is a higher abstraction of TensorFlow.js that makes it easy to use existing pre-trained deep learning models in a unified way, with a minimal number of lines of code. The package comes with a wide range of built-in models, ranging from image segmentation to sound classification to text generation, some of which we will use in this example.
- Running on a livestream from a webcam using p5js: p5.js is a library that works nicely in conjunction with ml5.js and makes it super easy to make model predictions in real-time using a live video stream.
- PoseNet: We explore the computer vision task of pose estimation.
- Pix2Pix: We dive in sci-fi and examine image translation.
benchmark/index.html is a script that benchmarks the system.
We will need to install some libraries that allow us to train and test in the browser. These include