featured/06_vispy.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Featured Recipe #6: Getting started with Vispy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> This is a featured recipe from the [**IPython Cookbook**](http://ipython-books.github.io/), the definitive guide to **high-performance scientific computing** and **data science** in Python."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Most existing plotting or visualization libraries in Python can display small or medium datasets (containing no more than a few tens of thousands of points). In the *Big Data* era, it is sometimes necessary to display larger datasets."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Vispy](http://vispy.org) is a young 2D/3D high-performance visualization library that can display very large datasets. Vispy leverages the computational power of modern Graphics Processing Units (GPUs) through the OpenGL library. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Vispy offers a Pythonic object-oriented interface to OpenGL, useful to those who know OpenGL or who are willing to learn it. Higher-level graphical interfaces are also being developed at the time of this writing, and experimental versions are already available. These interfaces do not require any knowledge of OpenGL."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this recipe, we give a brief introduction to the fundamental concepts of OpenGL. There are two situations where you would need to know these concepts:\n",
    "\n",
    "* If you want to use Vispy today, before the availability of stable high-level plotting interfaces.\n",
    "* If you want to create custom, sophisticated, high-performance visualizations that are not yet implemented in Vispy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, we display a digital signal using Vispy's object-oriented interface to OpenGL."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Getting ready"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Vispy depends on NumPy. A backend library is necessary (PyQt4/PySide, wxPython, glfw, or other)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This recipe has been tested with the [development version of Vispy](https://github.com/vispy/vispy). You should clone the GitHub repository and install Vispy with `python setup.py install`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The API used in this recipe might change in future versions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How to do it..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Let's import NumPy, **vispy.app** (to display a canvas) and **vispy.gloo** (object-oriented interface to OpenGL)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from vispy import app\n",
    "from vispy import gloo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2. In order to display a window, we need to create a **Canvas**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "c = app.Canvas(keys='interactive')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3. When using `vispy.gloo`, we need to write **shaders**. These programs, written in a C-like language called GLSL, run on the GPU and give us full flexibility for our visualizations. Here, we create a trivial **vertex shader** that directly displays 2D data points (stored in the `a_position` variable) in the canvas. The function `main()` executes once per data point (also called **vertex**). The variable `a_position` contains the `(x, y)` coordinates of the current vertex. All this function does is to pass these coordinates to the next stage of processing in the rendering pipeline. We give more details in the *How it works* section below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "vertex = \"\"\"\n",
    "attribute vec2 a_position;\n",
    "void main (void)\n",
    "{\n",
    "    gl_Position = vec4(a_position, 0.0, 1.0);\n",
    "}\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "4. The other shader we need to create is the **fragment shader**. It lets us control the pixels' color. Here, we display all data points in black. This function runs once per generated pixel."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "fragment = \"\"\"\n",
    "void main()\n",
    "{\n",
    "    gl_FragColor = vec4(0.0, 0.0, 0.0, 1.0);\n",
    "}\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "5. Next, we create an **OpenGL Program**. This object contains the shaders and allows us to link the shader variables to Python/NumPy data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "program = gloo.Program(vertex, fragment)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "6. We link the variable `a_position` to a `(1000, 2)` NumPy array containing the coordinates of 1000 data points. In the default coordinate system, the coordinates of the four canvas corners are `(+/-1, +/-1)`. Here, we generate a random time-dependent signal in $[-1,1]$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "program['a_position'] = np.c_[\n",
    "        np.linspace(-1.0, +1.0, 1000),\n",
    "        np.random.uniform(-0.5, +0.5, 1000)].astype(np.float32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "7. We create a callback function called when the window is being resized. Updating the **OpenGL viewport** lets us ensure that Vispy uses the entire canvas."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "@c.connect\n",
    "def on_resize(event):\n",
    "    gloo.set_viewport(0, 0, *event.size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "8. We create a callback function called when the canvas needs to be refreshed. This `on_draw` function renders the entire scene. First, we clear the window in white (it is necessary to do that at every frame). Then, we draw a succession of line segments using our OpenGL program. The vertices used for this visual are those returned by the vertex shader."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "@c.connect\n",
    "def on_draw(event):\n",
    "    gloo.clear((1,1,1,1))\n",
    "    program.draw('line_strip')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "9. Finally, we show the canvas and we run the application."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "c.show()\n",
    "app.run();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following figure shows a screenshot:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Basic visualization example with Vispy](images/vispy.PNG)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How it works..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "OpenGL is an open standard for hardware-accelerated interactive visualization. It is widely used in video games, industry software (Computer-Aided Design, or CAD, virtual reality) and scientific applications (medical imaging, computer graphics, and so on)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "OpenGL is a mature technology created in the early 1990s. In the early 2000s, OpenGL 2.0 brought a major new feature: the possibility to customize fundamental steps of the **rendering pipeline**. This pipeline defines the way data is processed on the GPU for real-time rendering. Many OpenGL courses and tutorials cover the old, fixed pipeline. Vispy supports exclusively the modern, programmable pipeline."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, we introduce the fundamental concepts of the programmable pipeline used in this recipe. OpenGL is considerably more complex than what we will cover here. However, Vispy provides a vastly simplified API for the most common features of OpenGL."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "style": "tip"
   },
   "source": [
    "Vispy is based on OpenGL ES 2.0, a flavor of OpenGL that is supported on desktop computers, mobile devices, and modern Web browsers (through **WebGL**). Modern graphics cards may support additional features. Those features will be available in future versions of Vispy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are four major elements in the rendering pipeline of a given OpenGL program:\n",
    "\n",
    "1. **Data buffers** store numerical data on the GPU. The main types of buffers are **vertex buffers**, **index buffers** and **textures**.\n",
    "2. **Variables** are available in the shaders. There are four major types of variables: **attributes**, **uniforms**, **varyings** and **texture samplers**.\n",
    "3. **Shaders** are GPU programs written in a C-like language called **OpenGL Shading Language** (GLSL). The two main types of shaders are **vertex shaders** and **fragment shaders**.\n",
    "4. The **primitive type** defines the way data points are rendered. The main types are points, lines and triangles."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is how the rendering pipeline works:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Data is sent on the GPU and stored in buffers.\n",
    "2. The vertex shader processes the data in parallel and generates a number of 4D points in a normalized coordinate system `(+/-1, +/-1)`. The fourth dimension is a homogeneous coordinate (generally 1).\n",
    "3. Graphics primitives are generated from the data points returned by the vertex shader (**primitive assembly** and **rasterization**).\n",
    "4. The fragment shader processes all primitive pixels in parallel and returns each pixel's color as RGBA components."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this recipe's example, there is only one GPU variable: the attribute `a_position`. An attribute is a variable that takes one value per data point. Uniforms are global variables (shared by all data points), whereas varyings are used to pass values from the vertex shader to the fragment shader (with automatic linear interpolation for a pixel between 2 or 3 vertices)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In `vispy.gloo`, a Program is created with the vertex and fragment shaders. Then, the variables declared in the shaders can be set with the syntax `program['varname'] = value`. When `varname` is an attribute variable, the value can just be a NumPy 2D array. In this array, every line contains the components of every data point."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similarly, we could declare and set uniforms and textures in our program."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, `program.draw()` renders the data using the specified primitive type. Here, the `line_strip` primitive type tells the GPU to run through all vertices (as returned by the vertex buffer) and to draw a line segment from one point to the next. If there are *n* points, there will be *n-1* line segments."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Other primitive types include points and triangles, with several ways of generating lines or triangles from a list of vertices."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In addition, an index buffer may be provided. An index buffer contains indices pointing to the vertex buffers. Using an index buffer would allow us to reuse any vertex multiple times during the primitive assembly stage. For example, when rendering a cube with a `triangles` primitive type (one triangle is generated for every triplet of points), we could use a vertex buffer with 8 data points and an index buffer with 36 indices (3 points per triangle, 2 triangles per face, 6 faces)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## There's more..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The example shown here is extremely simple. The approach provided by OpenGL and Vispy is nevertheless particularly powerful. It gives us full control on the rendering pipeline, and it allows us to leverage the computational power of the GPU in a nearly optimal way."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "High performance is achieved by minimizing the number of data transfers to the GPU. When displaying static data (for example, a scatter plot), it is possible to send the data to the GPU at initialization time only. Yet, rendering dynamic data is reasonably fast; the order of magnitude of data transfers is roughy 1 GB/s."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Besides, it is critical to use as few OpenGL draw calls as possible. Every draw incurs a significant overhead. High performance is achieved by rendering all similar primitive types at once (**batch rendering**). GPUs are particularly efficient with batch rendering, even when the properties of the points are different (for example, points with various sizes and colors)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, geometric or pixel transformations can be executed on the GPU with very high performance using the shaders. The massively parallel architecture of GPUs, consisting of hundreds or thousands of computing units, is fully leveraged when transformations are implemented in the shaders."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "General-purpose computations can be done in the shaders in a context of visualization. There is one major drawback compared to proper GPGPU frameworks like CUDA or OpenCL, though. In the vertex shader, a given thread has access to one data point only. Similarly, in the fragment shader, a thread has only access to one pixel. There are ways to mitigate this issue, but they lead to a drop of performance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, it is possible to interoperate OpenGL with CUDA/OpenCL. Buffers can be shared between OpenGL and the GPGPU framework. Complex CUDA/OpenCL computations can be implemented on vertex buffers or textures in real-time, leading to highly efficient rendering of numerical simulations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Vispy for scientific visualization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we have seen in this recipe, Vispy requires the user to know OpenGL and GLSL. However, higher-level graphical interfaces are currently being developed. Those interfaces will bring to scientists the power of GPUs for high-performance interactive visualization."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Visuals** will provide reusable, reactive graphical components like shapes, polygons, 3D meshes, surface plots, network graphs, and others. These visuals will be fully customizable and may be used without knowledge of OpenGL. A **shader composition system** will allow advanced users to reuse snippets of GLSL functionality in a modular way."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Visuals will be organized within a **scene graph** implementing GPU-based **transformations**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Scientific plotting interfaces will be implemented. Vispy could also serve as a high-performance backend for existing plotting libraries such as matplotlib."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Vispy will also support full integration in the IPython notebook using WebGL."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Eventually, Vispy could implement many kinds of scientific visualizations:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Scatter plots can be rendered efficiently with **point sprites**, using one vertex per data point. Panning and zooming can be implemented in the vertex shader, enabling fast interactive visualization of millions of points.\n",
    "* Digital signals, static or dynamic (real-time) can be displayed with polylines. High-quality rendering of curves can be achieved using an OpenGL implementation of the **Anti-grain Geometry** (agg) library.\n",
    "* Network graphs can be displayed by combining points and line segments.\n",
    "* 3D meshes can be displayed with triangles and index buffers. Geometric transformations and realistic lighting can be implemented in the vertex and fragment shader.\n",
    "* Real-time streams of images can be displayed efficiently with textures.\n",
    "* Axes, grids, ticks, text, and labels can be rendered efficiently in the fragment shader."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Many examples can be found in Vispy's gallery."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### References"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here are a few references:\n",
    "\n",
    "* [Vispy's gallery](http://vispy.org/gallery.html).\n",
    "* [A modern OpenGL tutorial by Nicolas P. Rougier](http://www.loria.fr/~rougier/teaching/opengl/).\n",
    "* [Hardware-accelerated interactive data visualization for neuroscience in Python](http://journal.frontiersin.org/Journal/10.3389/fninf.2013.00036/full).\n",
    "* [Vispy users mailing list](https://groups.google.com/forum/#!forum/vispy).\n",
    "* [Vispy-dev mailing list](https://groups.google.com/forum/#!forum/vispy-dev)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> This was a featured recipe from the [IPython Cookbook](http://ipython-books.github.io/), by [Cyrille Rossant](http://cyrille.rossant.net), Packt Publishing, 2014."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}