docs/source/index.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# IPython Notebook Validation for py.test - Documentation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One of the powerful uses of the IPython notebook is for documentation purposes, here we use a notebook to demonstrate the behaviour and usage of the IPython Notebook Validation plugin for py.test. The IPython notebook format `.ipynb` stores outputs as well as inputs. Validating the notebook means to rerun the notebook and make sure that it is generating the same output as has been stored.\n",
    "\n",
    "Therefore, the **user MUST make the following the distinction**:\n",
    "\n",
    "1. Running a notebook manually will likely change the output stored in the associated .ipynb file. These outputs will be used as references for the tests (i.e. the outputs from the last time you ran the notebook)\n",
    "2. Validating with py.test plugin - these tests run your notebook code seperately without storing the information, the outputs generated will be compared against those in the .ipynb file\n",
    "\n",
    "The purpose of the testing module is to ensure that the notebook is behaving as expected and that changes to underlying source code, haven't affected the results of an IPython notebook. For example, for documentation purposes - such as this."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Command line usage"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The py.test program doesn't usually collect notebooks for testing; by passing the `--nbval` flag at the command line, the IPython Notebook Validation plugin will collect and test notebook cells, comparing their outputs with those saved in the file.\n",
    "\n",
    "```\n",
    "$ py.test --nbval my_notebook.ipynb\n",
    "```\n",
    "\n",
    "There is also an option `--nbval-lax`, which collects notebooks and runs them, failing if there is an error. This mode does not check the output of cells unless they are marked with a special `#NBVAL_CHECK_OUTPUT` comment.\n",
    "\n",
    "```\n",
    "$ py.test --nbval-lax my_notebook.ipynb\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### REGEX Output sanitizing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since all output is captured by the IPython notebook, some pesky messages and prompts (with time-stamped messages, for example) may fail tests always, which might be expected. The plugin allows the user to specify a sanitizing file at the command prompt using the following flag:\n",
    "```\n",
    "$ py.test --nbval my_notebook.ipynb --sanitize-with my_sanitize_file\n",
    "```\n",
    "\n",
    "This sanitize file contains a number of REGEX replacements. It is recommended, when removing output for the tests, that you replace the removed output with some sort of marker, this helps with debugging. The following file is written to the folder of this notebook and can be used to sanitize its outputs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing doc_sanitize.cfg\n"
     ]
    }
   ],
   "source": [
    "%%writefile doc_sanitize.cfg\n",
    "[regex1]\n",
    "regex: \\d{1,2}/\\d{1,2}/\\d{2,4}\n",
    "replace: DATE-STAMP\n",
    "\n",
    "[regex2]\n",
    "regex: \\d{2}:\\d{2}:\\d{2}\n",
    "replace: TIME-STAMP"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first replacement finds dates in the given format replaces them with the label 'DATE-STAMP', likewise for strings that look like time. These will prevent the tests from failing due to time differences."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Validate this notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This documentation is written as a Notebook. You can validate this notebook yourself, as shown below; the outputs that you see here are stored in the ipynb file. If your system produces different outputs, the testing process will fail. Just use the following commands:\n",
    "```\n",
    "$ cd /path/to/repo/docs/source\n",
    "$ py.test --nbval index.ipynb --sanitize-with doc_sanitize.cfg\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Examples of plugin behaviour"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following examples demonstrate how the plugin behaves during testing. Test this notebook yourself to see the validation in action!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These two imports produce no output as standard, if any **warnings** are printed out the cell will fail. Under normal operating conditions they will pass."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If python doesn't consistently print 7, then something has gone terribly wrong. **Deterministic cells** are expected to pass everytime"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "7\n"
     ]
    }
   ],
   "source": [
    "print(5+2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Random outputs** will always fail."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.36133679016382714, 0.5043774697891126, 0.23281910875007927, 0.2713065513128683]\n",
      "[0.5512421277985322, 0.02592706358897756, 0.05036036771084684, 0.7515926759190724]\n"
     ]
    }
   ],
   "source": [
    "print([np.random.rand() for i in range(4)])\n",
    "print([np.random.rand() for i in range(4)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Inconsistent number of lines** of output will cause an error to be thrown."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n",
      "1\n",
      "1\n"
     ]
    }
   ],
   "source": [
    "for i in range(np.random.randint(1, 8)):\n",
    "    print(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because the **time and date** will change with each run, we would expect this cell to fail everytime. Using the sanitize file `doc_sanitize.cfg` (created above) you can clean up these outputs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The time is: 15:28:30\n",
      "Today's date is: 21/12/16\n"
     ]
    }
   ],
   "source": [
    "print('The time is: ' + time.strftime('%H:%M:%S'))\n",
    "print(\"Today's date is: \" + time.strftime('%d/%m/%y'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Avoid output comparison for specific cells"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In case we want to avoid the testing process in specific input cells, we can write the comment ** #NBVAL_IGNORE_OUTPUT ** at the\n",
    "beginning of the them:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This is not going to be tested\n",
      "12544\n"
     ]
    }
   ],
   "source": [
    "# NBVAL_IGNORE_OUTPUT\n",
    "print('This is not going to be tested')\n",
    "print(np.random.randint(1, 20000))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There's also a counterpart, to ensure the output is tested even when using `--nbval-lax` :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This will be tested\n",
      "42\n"
     ]
    }
   ],
   "source": [
    "# NBVAL_CHECK_OUTPUT\n",
    "print(\"This will be tested\")\n",
    "print(6 * 7)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that unexecuted cells will always skip its output check:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('This is not going to be tested when unrun')\n",
    "print(np.random.randint(1, 20000))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Skipping specific cells"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If, for some reason, a cell should not be executed during testing, the comment **# NBVAL_SKIP** can be used:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "# NBVAL_SKIP\n",
    "print(\"Entering infinite loop...\")\n",
    "while True:\n",
    "    pass\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Checking exceptions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sometimes, we might want to allow a notebook cell to raise an exception, and check that the traceback is as we expect.  By annotating the cell with the comment ** # NBVAL_RAISES_EXCEPTION ** you can indicate that the cell is expected to raise an exception.  The full traceback is not compared, but rather just that the raised exception is the same as the stored exception."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This exception will be tested\n"
     ]
    },
    {
     "ename": "RuntimeError",
     "evalue": "Foo",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mRuntimeError\u001b[0m                              Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-1-b97c0d501d6a>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;32mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"This exception will be tested\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0;32mraise\u001b[0m \u001b[0mRuntimeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Foo\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mRuntimeError\u001b[0m: Foo"
     ]
    }
   ],
   "source": [
    "# NBVAL_RAISES_EXCEPTION\n",
    "print(\"This exception will be tested\")\n",
    "raise RuntimeError(\"Foo\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This composes with the per-cell checking comments, so if you would like to avoid exceptions creating a test failure, but do not want to check the traceback, use `# NBVAL_IGNORE_OUTPUT`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "If the raised exception doesn't match the stored exception, we get a failure\n"
     ]
    },
    {
     "ename": "RuntimeError",
     "evalue": "Foo",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mRuntimeError\u001b[0m                              Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-3-32dcc1c70a4e>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;31m# NBVAL_RAISES_EXCEPTION\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      2\u001b[0m \u001b[0;32mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"If the raised exception doesn't match the stored exception, we get a failure\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 3\u001b[0;31m \u001b[0;32mraise\u001b[0m \u001b[0mRuntimeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Foo\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mRuntimeError\u001b[0m: Foo"
     ]
    }
   ],
   "source": [
    "# NBVAL_RAISES_EXCEPTION\n",
    "print(\"If the raised exception doesn't match the stored exception, we get a failure\")\n",
    "raise SyntaxError(\"Foo\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This exception will not be checked, but will not cause a failure.\n"
     ]
    },
    {
     "ename": "RuntimeError",
     "evalue": "Bar",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mRuntimeError\u001b[0m                              Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-2-bbee3f9e7de1>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      2\u001b[0m \u001b[0;31m# NBVAL_RAISES_EXCEPTION\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0;32mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"This exception will not be checked, but will not cause a failure.\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 4\u001b[0;31m \u001b[0;32mraise\u001b[0m \u001b[0mRuntimeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Bar\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mRuntimeError\u001b[0m: Bar"
     ]
    }
   ],
   "source": [
    "# NBVAL_IGNORE_OUTPUT\n",
    "# NBVAL_RAISES_EXCEPTION\n",
    "print(\"This exception will not be checked, but will not cause a failure.\")\n",
    "raise RuntimeError(\"Bar\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using tags instead of comments"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you do not want to put nbval comment annotations in your notebook, or your source language is not compatible with such annotations, you can use cell tags instead. Cell tags are strings that are added to the cell metadata under the label \"tags\", and can be added and remove using the \"Tags\" toolbar from Notebook version 5. The tags that Nbval recognizes are the same as the comment names, except lowercase, and with dashes ('-') instead of underscores ('\\_'). For instance, the comment \"`NBVAL_IGNORE_OUTPUT`\" becomes the tag \"`nbval-ignore-output`\".  However, for \"`NBVAL_RAISES_EXCEPTION`\", either \"`nbval-raises-exception`\" or the plain \"`raises-exception`\" tag can be used, since as of Notebook 5.1, the latter is a special tag that tells the Notebook cell executor to continue running normally after an exception is raised."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Figures"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Currently, only the matplotlib text output of the Figure is compared, but it is possible to modify the plugin to allow comparison of the image whole string."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x7f2cb3374198>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAREAAAEACAYAAACUHkKwAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAADtRJREFUeJzt3X2MXmWZx/HvDysmBkXApawUZoiCrFKCkiDqsp0NawSz\nsRujiy+JoJtgjCwm/rEYlixgNqv418qKMUQ0oCFdZROo77iy48uqyEJrW6FQE1qgQEksxEWMW8m1\nf8wBH2ee6UznPjPPU/l+kpO5n3OuOffVM+kv55y5m6aqkKSlOmTUDUg6uBkikpoYIpKaGCKSmhgi\nkpoYIpKaNIVIkiOS3Jrk3iTfTnL4PHVPJ7kryaYkN7fMKWm8pGWdSJKrgF9W1SeTXAIcUVUfHVL3\nq6p6cUOfksZUa4hsB9ZV1Z4kxwDTVXXykLr/raoXNfQpaUy1vhM5uqr2AFTVo8DR89S9IMlPk/wo\nyfrGOSWNkVULFST5DrB6cBdQwGVDyue7rZmoqkeSnADclmRLVd1/wN1KGjsLhkhVvWm+Y0n2JFk9\n8Djz2DzneKT7en+SaeA1wJwQSeI/5JFGqKpyoN+zYIgsYCNwAXAVcD5wy+yCJC8Bnqqq/0vyUuAN\nXf08Lm9saTlMA1Mj7mGWdQV/ecWou/hD/3XF2PV0Vv0F6644a9Rt/IHvXfGDsetpLadyXt61pO9t\nfSdyFfCmJPcCZwOfAEhyepJru5o/A/4nySbgu8DHq2p747ySxkTTnUhV7QX+asj+O4ELu/GPgVNb\n5pE0vlyxuiiTo25grsmpUXcw1xj2NDF1/KhbmGMce2phiCzK5KgbmOuEqVF3MNcY9jQ5NTHqFuYY\nx55aGCKSmhgikpoYIpKaGCKSmhgikpoYIpKaGCKSmhgikpoYIpKaGCKSmhgikpoYIpKaGCKSmhgi\nkpoYIpKaGCKSmhgikpoYIpKaGCKSmhgikpoYIpKaGCKSmhgikpoYIpKaGCKSmhgikpr0EiJJzkmy\nPcl9SS4ZcvzQJBuS7Ejy4yR/XP8ZqfQc1hwiSQ4BPg28GXg18K4kJ88q+ztgb1WdCPwr8MnWeSWN\nhz7uRM4AdlTVrqraB2wA1s+qWQ9c341vAs7uYV5JY6CPEDkWeHDg80PdvqE1VfU08ESSI3uYW9KI\njerFakY0r6SererhHLuBwRela7p9gx4CjgMeTvI84MVVtXf46aYHxpPdJqlvO6d3sWv6AQC2c9+S\nz9NHiNwBvCLJBPAI8E7gXbNqvgqcD9wOvAO4bf7TTfXQkqSFTE5NMDk1AcBaTuWmK/9jSedpDpGq\nejrJRcCtzDweXVdV9yS5Erijqr4GXAd8MckO4JfMBI2kPwJ93IlQVd8CXjlr3+UD498Cf9vHXJLG\niytWJTUxRCQ1MUQkNTFEJDUxRCQ1MUQkNTFEJDUxRCQ1MUQkNTFEJDUxRCQ1MUQkNTFEJDUxRCQ1\nMUQkNTFEJDUxRCQ1MUQkNTFEJDUxRCQ1MUQkNTFEJDUxRCQ1MUQkNTFEJDUxRCQ1MUQkNTFEJDUx\nRCQ16SVEkpyTZHuS+5JcMuT4+UkeS3JXt72/j3kljd6q1hMkOQT4NHA28DBwR5Jbqmr7rNINVXVx\n63ySxksfdyJnADuqaldV7QM2AOuH1KWHuSSNmT5C5FjgwYHPD3X7Zntbks1JvpxkTQ/zShoDzY8z\ni7QRuLGq9iW5ELiemcefudbV78eT62ByagXaO/hc9k//OOoWDgofu+bjo25hbE3fB9M7ZsY/P+qU\nJZ+njxDZDRw/8HlNt+9ZVfX4wMfPAZ+c92xTl/fQkqSFTJ00swFw4lo+9qVtSzpPH48zdwCvSDKR\n5FDgnczceTwryTEDH9cDd/cwr6Qx0HwnUlVPJ7kIuJWZULququ5JciVwR1V9Dbg4yVuBfcBe4ILW\neSWNh17eiVTVt4BXztp3+cD4UuDSPuaSNF5csSqpiSEiqYkhIqmJISKpiSEiqYkhIqmJISKpiSEi\nqYkhIqmJISKpiSEiqYkhIqmJISKpiSEiqYkhIqmJISKpiSEiqYkhIqmJISKpiSEiqYkhIqmJISKp\niSEiqYkhIqmJISKpiSEiqYkhIqmJISKpSS8hkuS6JHuSbNlPzdVJdiTZnOS0PuaVNHp93Yl8AXjz\nfAeTnAu8vKpOBD4AfLaneSWNWC8hUlU/BB7fT8l64Iau9nbg8CSr+5hb0mit1DuRY4EHBz7v7vZJ\nOsitGnUDc0xf+fvx5DqYnBpZK9Ifs+n7YHpH9+GorUs+z0qFyG7guIHPa7p9c01dvhL9SM95UyfN\nbACcuJaPfWnbks7T5+NMum2YjcB7AZKcCTxRVXt6nFvSiPRyJ5LkRmAKOCrJA8DlwKFAVdW1VfWN\nJG9J8gvg18D7+phX0uj1EiJV9e5F1FzUx1ySxosrViU1MUQkNTFEJDUxRCQ1MUQkNTFEJDUxRCQ1\nMUQkNTFEJDUxRCQ1MUQkNTFEJDUxRCQ1MUQkNTFEJDUxRCQ1MUQkNTFEJDUxRCQ1MUQkNTFEJDUx\nRCQ1MUQkNTFEJDUxRCQ1MUQkNTFEJDUxRCQ16SVEklyXZE+SLfMcX5fkiSR3ddtlfcwrafRW9XSe\nLwD/Btywn5rvV9Vbe5pP0pjo5U6kqn4IPL5AWfqYS9J4Wcl3Imcm2ZTk60letYLzSlpGfT3OLORO\nYKKqnkpyLnAzcNKwwrNq3bPjianjmZyaWJkODzJXXvPxUbdwULjy70fdwfja2W0AR5+ydcnnWZEQ\nqaonB8bfTPKZJEdW1d7ZteuuOGslWpKe8ya7DeCUtWv5yrZtSzpPn48zYZ73HklWD4zPADIsQCQd\nfHq5E0lyIzAFHJXkAeBy4FCgqupa4O1JPgjsA34DnNfHvJJGr5cQqap3L3D8GuCaPuaSNF5csSqp\niSEiqYkhIqmJISKpiSEiqYkhIqmJISKpiSEiqYkhIqmJISKpiSEiqYkhIqmJISKpiSEiqYkhIqmJ\nISKpiSEiqYkhIqmJISKpiSEiqYkhIqmJISKpiSEiqYkhIqmJISKpiSEiqYkhIqmJISKpSXOIJFmT\n5LYkP0+yNcnF89RdnWRHks1JTmudV9J4WNXDOX4HfKSqNic5DLgzya1Vtf2ZgiTnAi+vqhOTvA74\nLHBmD3NLGrHmO5GqerSqNnfjJ4F7gGNnla0HbuhqbgcOT7K6dW5Jo9frO5Ekk8BpwO2zDh0LPDjw\neTdzg0bSQai3EOkeZW4CPtzdkUh6DujjnQhJVjETIF+sqluGlOwGjhv4vKbbN8f3rvjBs+OJqeOZ\nnJroo0VJs+zsNoC7t25d8nl6CRHg88DdVfWpeY5vBD4E/HuSM4EnqmrPsMJ1V5zVU0uS9mey2wBO\nWbuWr2zbtqTzNIdIkjcC7wG2JtkEFHApMAFUVV1bVd9I8pYkvwB+DbyvdV5J46E5RKrqv4HnLaLu\nota5JI0fV6xKamKISGpiiEhqYohIamKISGpiiEhqYohIamKISGpiiEhqYohIamKISGpiiEhqYohI\namKISGpiiEhqYohIamKISGpiiEhqYohIamKISGpiiEhqYohIamKISGpiiEhqYohIamKISGpiiEhq\nYohIatIcIknWJLktyc+TbE1y8ZCadUmeSHJXt13WOq+k8dDHncjvgI9U1auB1wMfSnLykLrvV9Vr\nu+2fe5h3xeyc3jXqFuaY3jHqDuYax552jrqBIXaOuoGeNYdIVT1aVZu78ZPAPcCxQ0rTOteo7Jp+\nYNQtzPG9MfwLO4497Rx1A0PsHHUDPev1nUiSSeA04PYhh89MsinJ15O8qs95JY3Oqr5OlOQw4Cbg\nw90dyaA7gYmqeirJucDNwEnDzvMyXtZXS715ES8au74efeEx8Cfj1RMvfHjsejrsmIf505eNWU8P\nj19PR5xwwpK/N1XV3ECSVcDXgG9W1acWUX8/cHpV7Z21v70ZSUtWVQf82qGvO5HPA3fPFyBJVlfV\nnm58BjPhtXd23VL+AJJGqzlEkrwReA+wNckmoIBLgQmgqupa4O1JPgjsA34DnNc6r6Tx0MvjjKTn\nrpGuWE1yRJJbk9yb5NtJDp+n7ulukdqmJDcvUy/nJNme5L4klww5fmiSDUl2JPlxkuOXo48l9HV+\nkscGFvK9f5n7uS7JniRb9lNzdXedNic5bTn7WUxPo1jsuJhFmF3dil2rZVsYWlUj24CrgH/oxpcA\nn5in7lfL3MchwC+YeQR7PrAZOHlWzQeBz3Tj84ANK3B9FtPX+cDVK/gz+3Nmfo2/ZZ7j5wJf78av\nA34yBj2tAzau1DXq5jwGOK0bHwbcO+Rnt6LXapE9HfC1GvW/nVkPXN+Nrwf+Zp665X7hegawo6p2\nVdU+YEPX26DBXm8Czl7mnhbbF6zgQr6q+iHw+H5K1gM3dLW3A4cnWT3inmCFFzvW4hZhrui1WmRP\ncIDXatQhcnR1v7WpqkeBo+epe0GSnyb5UZJhf4laHQs8OPD5IeZe3Gdrqupp4IkkRy5DLwfaF8Db\nutvhLydZs8w9LWR2z7sZ3vNKG9lix/0swhzZtepzYWhvi83mk+Q7wGC6hpnf4Ax71prvLe9EVT2S\n5ATgtiRbqur+nls9UOPy6+iNwI1VtS/JhczcLa3EXdLBZNGLHfu2wCLMkehrYegzlv1OpKreVFWn\nDmxru68bgT3P3L4lOQZ4bJ5zPNJ9vR+YBl7Tc5u7gcEXpWu6fYMeAo7ren0e8OIastZlpfuqqse7\nRx2AzwGnL3NPC9lNd506w67liqqqJ6vqqW78TeD5K3AX+cwizJuAL1bVLUNKVvxaLdTTUq7VqB9n\nNgIXdOPzgTl/qCQvSXJoN34p8Abg7p77uAN4RZKJbq53dr0N+mrXI8A7gNt67mFJfXXh+4z19H9t\nhgnz34ltBN4LkORM4IlnHllH1dPge4b9LXZcBvtdhMlortWCC0MHxou7Viv5xnrI2+Ijgf9k5i3x\nrcBLuv2nA9d249cDW4BNwM+AC5apl3O6PnYAH+32XQn8dTd+AfDl7vhPgMkVukYL9fUvwLbu+nwX\nOGmZ+7kReBj4LfAA8D7gA8CFAzWfZua3Sj8DXrsC12i/PQEfGrhGPwJetwI9vRF4mpnfqG0C7up+\nliO7VovpaSnXysVmkpqM+nFG0kHOEJHUxBCR1MQQkdTEEJHUxBCR1MQQkdTEEJHU5P8BicLjTxAC\nPVsAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f2cb564a550>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.imshow(np.array([[i + j for i in range(3)]\n",
    "                     for j in range(3)]),\n",
    "           interpolation='None'\n",
    "           )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Skipping certain output types"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In case nbval is comparing some cell outputs you do not care about, like:\n",
    "\n",
    "```Traceback:missing key: TESTING dict_keys(['stderr']) != REFERENCE dict_keys(['application/javascript', 'stderr'])```\n",
    "\n",
    "There is a workaround. Add the following to your conftest.py:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def pytest_collectstart(collector):\n",
    "    if collector.fspath and collector.fspath.ext == '.ipynb':\n",
    "        collector.skip_compare += 'text/html', 'application/javascript', 'stderr',"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}