notebooks/chapter03_notebook/07_webcam_py3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> This is one of the 100 recipes of the [IPython Cookbook](http://ipython-books.github.io/), the definitive guide to high-performance scientific computing and data science in Python.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3.7. Processing webcam images in real-time from the notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this recipe, we show how to communicate data in both directions from the notebook to the Python kernel, and conversely. Specifically, we will retrieve the webcam feed from the browser using HTML5's `<video>` element, and pass it to Python in real time using the interactive capabilities of the IPython notebook 2.0+. This way, we can process the image in Python with an edge detector (implemented in scikit-image), and display it in the notebook in real time."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Most of the code for this recipe comes from [Jason Grout's example](https://github.com/jasongrout/ipywidgets)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. We need to import quite a few modules."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from IPython.html.widgets import DOMWidget\n",
    "from IPython.utils.traitlets import Unicode, Bytes, Instance\n",
    "from IPython.display import display\n",
    "\n",
    "from skimage import io, filter, color\n",
    "import urllib\n",
    "import base64\n",
    "from PIL import Image\n",
    "from io import BytesIO\n",
    "import numpy as np\n",
    "from numpy import array, ndarray\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2. We define two functions to convert images from and to base64 strings. This conversion is a common way to pass binary data between processes (here, the browser and Python)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def to_b64(img):\n",
    "    imgdata = BytesIO()\n",
    "    pil = Image.fromarray(img)\n",
    "    pil.save(imgdata, format='PNG')\n",
    "    imgdata.seek(0)\n",
    "    return urllib.parse.quote(base64.b64encode(imgdata.getvalue()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def from_b64(b64):\n",
    "    im = Image.open(BytesIO(base64.b64decode(b64)))\n",
    "    return array(im)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3. We define a Python function that will process the webcam image in real time. It accepts and returns a NumPy array. This function applies an edge detector with the `roberts()` function in scikit-image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def process_image(image):\n",
    "    img = filter.roberts(image[:,:,0]/255.)\n",
    "    return (255-img*255).astype(np.uint8)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "4. Now, we create a custom widget to handle the bidirectional communication of the video flow from the browser to Python and reciprocally."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "class Camera(DOMWidget):\n",
    "    _view_name = Unicode('CameraView', sync=True)\n",
    "    \n",
    "    # This string contains the base64-encoded raw\n",
    "    # webcam image (browser -> Python).\n",
    "    imageurl = Unicode('', sync=True)\n",
    "    \n",
    "    # This string contains the base64-encoded processed \n",
    "    # webcam image(Python -> browser).\n",
    "    imageurl2 = Unicode('', sync=True)\n",
    "\n",
    "    # This function is called whenever the raw webcam\n",
    "    # image is changed.\n",
    "    def _imageurl_changed(self, name, new):\n",
    "        head, data = new.split(',', 1)\n",
    "        if not data:\n",
    "            return\n",
    "        \n",
    "        # We convert the base64-encoded string\n",
    "        # to a NumPy array.\n",
    "        image = from_b64(data)\n",
    "        \n",
    "        # We process the image.\n",
    "        image = process_image(image)\n",
    "        \n",
    "        # We convert the processed image\n",
    "        # to a base64-encoded string.\n",
    "        b64 = to_b64(image)\n",
    "        \n",
    "        self.imageurl2 = 'data:image/png;base64,' + b64"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "5. The next step is to write the Javascript code for the widget."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%%javascript\n",
    "\n",
    "var video        = $('<video>')[0];\n",
    "var canvas       = $('<canvas>')[0];\n",
    "var canvas2       = $('<img>')[0];\n",
    "var width = 320;\n",
    "var height = 0;\n",
    "\n",
    "require([\"widgets/js/widget\"], function(WidgetManager){\n",
    "    var CameraView = IPython.DOMWidgetView.extend({\n",
    "        render: function(){\n",
    "            var that = this;\n",
    "\n",
    "            // We append the HTML elements.\n",
    "            setTimeout(function() {\n",
    "                that.$el.append(video).\n",
    "                         append(canvas).\n",
    "                         append(canvas2);}, 200);\n",
    "            \n",
    "            // We initialize the webcam.\n",
    "            var streaming = false;\n",
    "            navigator.getMedia = ( navigator.getUserMedia ||\n",
    "                                 navigator.webkitGetUserMedia ||\n",
    "                                 navigator.mozGetUserMedia ||\n",
    "                                 navigator.msGetUserMedia);\n",
    "\n",
    "            navigator.getMedia({video: true, audio: false},\n",
    "                function(stream) {\n",
    "                  if (navigator.mozGetUserMedia) {\n",
    "                    video.mozSrcObject = stream;\n",
    "                  } else {\n",
    "                    var vendorURL = (window.URL || \n",
    "                                     window.webkitURL);\n",
    "                    video.src = vendorURL.createObjectURL(\n",
    "                        stream);\n",
    "                  }\n",
    "                    video.controls = true;\n",
    "                  video.play();\n",
    "                },\n",
    "                function(err) {\n",
    "                  console.log(\"An error occured! \" + err);\n",
    "                }\n",
    "            );\n",
    "            \n",
    "            // We initialize the size of the canvas.\n",
    "            video.addEventListener('canplay', function(ev){\n",
    "                if (!streaming) {\n",
    "                  height = video.videoHeight / (\n",
    "                      video.videoWidth/width);\n",
    "                  video.setAttribute('width', width);\n",
    "                  video.setAttribute('height', height);\n",
    "                  canvas.setAttribute('width', width);\n",
    "                  canvas.setAttribute('height', height);\n",
    "                  canvas2.setAttribute('width', width);\n",
    "                  canvas2.setAttribute('height', height);\n",
    "                    \n",
    "                  streaming = true;\n",
    "                }\n",
    "            }, false);\n",
    "            \n",
    "            // Play/Pause functionality.\n",
    "            var interval;\n",
    "            video.addEventListener('play', function(ev){\n",
    "                // We get the picture every 100ms.    \n",
    "                interval = setInterval(takepicture, 100);\n",
    "            })\n",
    "            video.addEventListener('pause', function(ev){\n",
    "                clearInterval(interval);\n",
    "            })\n",
    "            \n",
    "            // This function is called at each time step.\n",
    "            // It takes a picture and sends it to the model.\n",
    "            function takepicture() {\n",
    "                canvas.width = width; canvas.height = height;\n",
    "                canvas2.width = width; canvas2.height = height;\n",
    "                \n",
    "                video.style.display = 'none';\n",
    "                canvas.style.display = 'none';\n",
    "                \n",
    "                // We take a screenshot from the webcam feed and \n",
    "                // we put the image in the first canvas.\n",
    "                canvas.getContext('2d').drawImage(video, \n",
    "                    0, 0, width, height);\n",
    "                \n",
    "                // We export the canvas image to the model.\n",
    "                that.model.set('imageurl',\n",
    "                               canvas.toDataURL('image/png'));\n",
    "                that.touch();\n",
    "            }\n",
    "        },\n",
    "        \n",
    "        update: function(){\n",
    "            // This function is called whenever Python modifies\n",
    "            // the second (processed) image. We retrieve it and\n",
    "            // we display it in the second canvas.\n",
    "            var img = this.model.get('imageurl2');\n",
    "            canvas2.src = img;\n",
    "            return CameraView.__super__.update.apply(this);\n",
    "        }\n",
    "    });\n",
    "    \n",
    "    // Register the view with the widget manager.\n",
    "    WidgetManager.register_widget_view('CameraView', \n",
    "                                       CameraView);\n",
    "});"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "6. Finally, we create and display the widget."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "c = Camera()\n",
    "display(c)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> You'll find all the explanations, figures, references, and much more in the book (to be released later this summer).\n",
    "\n",
    "> [IPython Cookbook](http://ipython-books.github.io/), by [Cyrille Rossant](http://cyrille.rossant.net), Packt Publishing, 2014 (500 pages)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}