lectures/IITesting.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Interfaces, Implementations, and Testing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Design of a program\n",
    "\n",
    "From the Practice of Programming:\n",
    "\n",
    ">The essence of design is to balance competing goals and constraints. Although there may be many tradeoffs when one is writing a small self-contained system, the ramifications of particular choices remain within the system and affect only the individual programmer. But when code is to be used by others, decisions have wider repercussions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "What are the issues we need to be cognizant of?\n",
    "\n",
    "- **Interfaces**: your program is being designed to be used by someone: either an end user, another programmer (you are writing a library), or even yourself (we are talking about some layer in your program). This **interface** is a contract between you and the user: with preconditions, postconditions, etc Eg: your binary search, the C-library\n",
    "- There is **information** hiding between layers (a higher up layer can be more abstract, more lossy about all the information). Encapsulation,  abstraction,  modularization, are some of the techniques used here. Eg. web access to database.\n",
    "- There are **resource management** issues: who allocates storage for data structures.. (generally we want resource allocation/deallocation to happen in the same layer) Eg. `strdup` vs `strcpy`.\n",
    "- How to **deal with errors**: do we return special values, throw exceptions? who handles them? (generally we want to catch even the lowest level error and give the client the chance to handle it, possibly lossily)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Interface principles\n",
    "\n",
    "Interfaces should:\n",
    "\n",
    "- hide implementation details\n",
    "- have a small set of operations exposed, the smallest possible, and these should be orthogonal. Be stingy with the user.\n",
    "- but be transparent with the user in what goes on behind the scenes (calls the NSA)\n",
    "- be consistent internally: library functions should have similar signature, classes similar methods, and externally: programs should have the same cli flags, same ops.\n",
    "\n",
    "** Testing should deal with ALL of the issues above, and each layer ought to be tested separately **. \n",
    "\n",
    "This gives rise to :"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Different kinds of tests\n",
    "\n",
    "- **acceptance tests** verify that a program meets a customer's expectations. In a sense these are a test of the *interface* to the customer: does the program do everything you promised the customer it would do? You might use test harnesees for cli programs and selenium for this. The test of a library interface could also be thought of as an acceptance test\n",
    "\n",
    "- **unit tests** are tests which test a unit of the program, for use by another unit. These could test the interface for a client, but must also be testing internal functions which you want to use.\n",
    "\n",
    "Exploratory testing, regression testing, and integration testing are done in both of these categories, with the latter trying to combine layers and subsystems, not necessarily at the level of an entire application. \n",
    "\n",
    "One can also performance test, random and exploratorily test, and stress test a system (to create adversarial situations)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Testing of a program\n",
    "\n",
    "Test as you write your program.\n",
    "\n",
    "This is so important that I repeat it.\n",
    "\n",
    "**Test as you go**.\n",
    "\n",
    "You will cry otherwise. I have in the past.\n",
    "\n",
    "From The Practice of Programming:\n",
    "\n",
    "\n",
    ">The  effort  of  testing as  you  go  is  minimal  and  pays off  handsomely.  Thinking about testing as you  write a program will  lead to better code, because that's when you know  best  what the code should do.  **If  instead  you  wait  until  something breaks, you will  probably  have forgotten how  the code works**.  Working under  pressure, you  will need  to figure it  out again, which  takes time, and  the fixes  will  be  less  thorough  and more fragile because your refreshed understanding is  likely to be incomplete. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Assertions and the process of testing\n",
    "\n",
    "The workhorse of testing is the `assert` statement. The same statement can also be used to assert preconditions and postconditions, and thus to test them if appropriate. In C, `assert` is a macro which can be conditionally compiled away."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "def myaverage(l:list)->float:\n",
    "    sumit = 0.0\n",
    "    for f in l:\n",
    "        sumit = sumit + f\n",
    "    average = sumit/len(l)\n",
    "    return average"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "assert myaverage([1,2])==1.5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "ename": "AssertionError",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mAssertionError\u001b[0m                            Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-3-effec7f1a9bf>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0;32massert\u001b[0m \u001b[0mmyaverage\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m2\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m==\u001b[0m\u001b[0;36m3\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mAssertionError\u001b[0m: "
     ]
    }
   ],
   "source": [
    "assert myaverage([1,2])==3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Principles of testing\n",
    "\n",
    "#### Test Simple Parts First"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "def test_average():\n",
    "    assert myaverage([1,2])==1.5, \"1 and 2 must average to 1.5\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "test_average()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "#now say you make a boneheaded change\n",
    "def myaverage(l:list)->float:\n",
    "    average = 0.0\n",
    "    for f in l:\n",
    "        average = average + 2*f\n",
    "    average = average/len(l)\n",
    "    return average"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "ename": "AssertionError",
     "evalue": "1 and 2 must average to 1.5",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mAssertionError\u001b[0m                            Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-8-a9de5b052499>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mtest_average\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m<ipython-input-5-5ec8461ea141>\u001b[0m in \u001b[0;36mtest_average\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mtest_average\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m     \u001b[0;32massert\u001b[0m \u001b[0mmyaverage\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m2\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m==\u001b[0m\u001b[0;36m1.5\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"1 and 2 must average to 1.5\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mAssertionError\u001b[0m: 1 and 2 must average to 1.5"
     ]
    }
   ],
   "source": [
    "test_average()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "def myaverage(l:list)->float:\n",
    "    n = len(l)\n",
    "    thesum = sum(l)\n",
    "    average = thesum/n\n",
    "    return average"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "test_average()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Test code at its boundaries\n",
    "\n",
    "The idea is that most errors happen at data boundaries such as empty input, single input item, exactly full array, wierd values, etc. If a piece of code works at the boundaries, its likely to work elsewhere...\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "ename": "ZeroDivisionError",
     "evalue": "division by zero",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mZeroDivisionError\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-11-d83504e73c10>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mmyaverage\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m<ipython-input-9-a4032b9131af>\u001b[0m in \u001b[0;36mmyaverage\u001b[0;34m(l)\u001b[0m\n\u001b[1;32m      2\u001b[0m     \u001b[0mn\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ml\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      3\u001b[0m     \u001b[0mthesum\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msum\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ml\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 4\u001b[0;31m     \u001b[0maverage\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mthesum\u001b[0m\u001b[0;34m/\u001b[0m\u001b[0mn\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      5\u001b[0m     \u001b[0;32mreturn\u001b[0m \u001b[0maverage\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mZeroDivisionError\u001b[0m: division by zero"
     ]
    }
   ],
   "source": [
    "myaverage([])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Program defensively \n",
    "\n",
    "The user is supposed to give us an array, non-zero. We could specify that as a precondition, but we might as well be defensive and test for that, since this function might be called deep in some kernel.\n",
    "\n",
    "Practice:\n",
    ">\"Program  defensively.  A  useful  technique is  to  add  code  to  handle  \"can't  happen\" cases,  situations  where  it  is  not  logically  possible  for  something  to  happen  but (because of  some failure elsewhere) it might anyway.  Adding a test for zero or nega- tive array lengths to avg  was one example.  As another example, a program  process- ing  grades might  expect  that  there  would  be  no  negative  or huge  values  but  should check anyway: "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    " From https://docs.python.org/3/library/exceptions.html:\n",
    " \n",
    " >exception **ValueError**\n",
    " \n",
    ">Raised when a built-in operation or function receives an argument that has the right type but an inappropriate value, and the situation is not described by a more precise exception such as IndexError.\n",
    "\n",
    "(But see this as to why you might want to possibly write your own exception..https://utcc.utoronto.ca/~cks/space/blog/python/NeverRaiseCoreExceptions . It does not apply to `ValueError` in my opinion, but is useful advice nonetheless.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "def myaverage(l:list)->float:\n",
    "    \"\"\"\n",
    "    Calculate the average of list l\n",
    "    \"\"\"\n",
    "    n = len(l)\n",
    "    if n==0:\n",
    "        raise ValueError(\"cant calculate mean of length 0 list\")\n",
    "    thesum = sum(l)\n",
    "    average = thesum/n\n",
    "    return average"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "test_average()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ERROR: <class 'ValueError'> ('cant calculate mean of length 0 list',)\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    myaverage([])\n",
    "except Exception as e:\n",
    "    print(\"ERROR:\",type(e), e.args)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "It is not a bad practice to raise an exception when appropriate. Not all edge cases need to be dealt with by patching the error...but you must send back an error that makes sense to the user of your code...\n",
    "\n",
    "Create a test so that this edge case is covered:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "def test_average_empty():\n",
    "    try:\n",
    "        myaverage([])\n",
    "    except Exception as e:\n",
    "        assert (type(e) == ValueError and e.args[0]=='cant calculate mean of length 0 list')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "test_average_empty()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Automate using a test harness\n",
    "\n",
    "In this document, we have been creating tests on an ad-hoc basis. Its best to use a **test harness** to do this.\n",
    "\n",
    "You have already seen an example. Doctests are one way to do this. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting mymath.py\n"
     ]
    }
   ],
   "source": [
    "%%file mymath.py\n",
    "\n",
    "\n",
    "def myaverage(l:list)->float:\n",
    "    \"\"\"\n",
    "    Calculate the average of list l\n",
    "    \n",
    "    Examples:\n",
    "    \n",
    "    >>> myaverage([1,2])\n",
    "    1.5\n",
    "    >>> myaverage([])\n",
    "    Traceback (most recent call last):\n",
    "        ...\n",
    "    ValueError: cant calculate mean of length 0 list\n",
    "    \"\"\"\n",
    "    n = len(l)\n",
    "    if n==0:\n",
    "        raise ValueError(\"cant calculate mean of length 0 list\")\n",
    "    thesum = sum(l)\n",
    "    average = thesum/n\n",
    "    return average"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Trying:\r\n",
      "    myaverage([1,2])\r\n",
      "Expecting:\r\n",
      "    1.5\r\n",
      "ok\r\n",
      "Trying:\r\n",
      "    myaverage([])\r\n",
      "Expecting:\r\n",
      "    Traceback (most recent call last):\r\n",
      "        ...\r\n",
      "    ValueError: cant calculate mean of length 0 list\r\n",
      "ok\r\n",
      "1 items had no tests:\r\n",
      "    mymath\r\n",
      "1 items passed all tests:\r\n",
      "   2 tests in mymath.myaverage\r\n",
      "2 tests in 2 items.\r\n",
      "2 passed and 0 failed.\r\n",
      "Test passed.\r\n"
     ]
    }
   ],
   "source": [
    "!python3 -m doctest mymath.py --verbose"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### The unittest framework\n",
    "\n",
    "But too many doctests clutter the documentation of a function.\n",
    "\n",
    "We want examples in a function to be evocative about how to use that function, and not to be covering every wierd case!\n",
    "\n",
    "One should only have those examples as doctests which describe the various ways a class or function can be used. Edge cases which must work, etc, ought to be represented in a separate test file. And once you commit to using a regular framework, some people prefer to have all their tests there.."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting test_mymath.py\n"
     ]
    }
   ],
   "source": [
    "%%file test_mymath.py\n",
    "\n",
    "\n",
    "import unittest\n",
    "\n",
    "from mymath import myaverage\n",
    "\n",
    "class MyTest(unittest.TestCase):\n",
    "    \n",
    "    def test_mymath(self):\n",
    "        self.assertEqual(myaverage([2,3]), 2.5)\n",
    "\n",
    "if __name__ == '__main__':\n",
    "    unittest.main()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ".\r\n",
      "----------------------------------------------------------------------\r\n",
      "Ran 1 test in 0.000s\r\n",
      "\r\n",
      "OK\r\n"
     ]
    }
   ],
   "source": [
    "!python3 test_mymath.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### When you get an error\n",
    "\n",
    "It could be that:\n",
    "\n",
    "- you messed up an implementation (we saw this above with the mult by two)\n",
    "- you did not handle a case\n",
    "- your test was messed up (be careful of this)\n",
    "\n",
    "If the error was not found in an existing test, create a new test that represents the problem BEFORE you do anything else. The test should capture the essence of the problem: this process itself is useful in uncovering bugs. Then this error may even suggest more tests. You fix, perhaps by writing defensive code.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "ename": "TypeError",
     "evalue": "unsupported operand type(s) for +: 'int' and 'str'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-23-ca4d377dd12b>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mmyaverage\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'a'\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m<ipython-input-12-67be5128c677>\u001b[0m in \u001b[0;36mmyaverage\u001b[0;34m(l)\u001b[0m\n\u001b[1;32m      6\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mn\u001b[0m\u001b[0;34m==\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      7\u001b[0m         \u001b[0;32mraise\u001b[0m \u001b[0mValueError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"cant calculate mean of length 0 list\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 8\u001b[0;31m     \u001b[0mthesum\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msum\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ml\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      9\u001b[0m     \u001b[0maverage\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mthesum\u001b[0m\u001b[0;34m/\u001b[0m\u001b[0mn\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     10\u001b[0m     \u001b[0;32mreturn\u001b[0m \u001b[0maverage\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mTypeError\u001b[0m: unsupported operand type(s) for +: 'int' and 'str'"
     ]
    }
   ],
   "source": [
    "myaverage(['a',1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Test Incrementally\n",
    "\n",
    "In this way you test incrementally, adding tests all the time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "...\n",
      "----------------------------------------------------------------------\n",
      "Ran 3 tests in 0.002s\n",
      "\n",
      "OK\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<unittest.runner.TextTestResult run=3 errors=0 failures=0>"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import unittest\n",
    "\n",
    "class MyTest(unittest.TestCase):\n",
    "    \n",
    "    def test_mymath(self):\n",
    "        self.assertEqual(myaverage([2,3]), 2.5)\n",
    "        \n",
    "    def test_char(self):\n",
    "        with self.assertRaises(TypeError):\n",
    "            myaverage(['a',3])\n",
    "            \n",
    "    def test_zerol(self):\n",
    "        with self.assertRaises(ValueError):\n",
    "            myaverage([])\n",
    "\n",
    "\n",
    "suite = unittest.TestLoader().loadTestsFromModule(MyTest())\n",
    "unittest.TextTestRunner().run(suite)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Test pre-conditions and post-conditions\n",
    "\n",
    "While we could take the position that bad pre-conditions lead to undefined behavior, we are good citizens if we test preconditions for the user...and this helps during algorithm development as well. Here we test 2 pre-conditions: no zero length, and numerical array. The latter would be too expensive at the beginning, so we do it by wrapping `sum` in a `try..except` block. We could test postcondition by asserting that the average is a number."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "def myaverage(l:list)->float:\n",
    "    \"\"\"\n",
    "    Calculate the average of list l\n",
    "    \n",
    "    Examples:\n",
    "    \n",
    "    >>> myaverage([1,2])\n",
    "    1.5\n",
    "    \n",
    "    \"\"\"\n",
    "    n = len(l)\n",
    "    if n==0:\n",
    "        raise ValueError(\"cant calculate mean of length 0 list\")\n",
    "    try:\n",
    "        thesum = sum(l)\n",
    "    except:\n",
    "        raise TypeError(\"Cannot sum things of different types\")\n",
    "    average = thesum/n\n",
    "    return average"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "ename": "TypeError",
     "evalue": "Cannot sum things of different types",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-25-fc8c436283a6>\u001b[0m in \u001b[0;36mmyaverage\u001b[0;34m(l)\u001b[0m\n\u001b[1;32m     14\u001b[0m     \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 15\u001b[0;31m         \u001b[0mthesum\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msum\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ml\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     16\u001b[0m     \u001b[0;32mexcept\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mTypeError\u001b[0m: unsupported operand type(s) for +: 'int' and 'str'",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[0;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-26-ca4d377dd12b>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mmyaverage\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'a'\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m<ipython-input-25-fc8c436283a6>\u001b[0m in \u001b[0;36mmyaverage\u001b[0;34m(l)\u001b[0m\n\u001b[1;32m     15\u001b[0m         \u001b[0mthesum\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msum\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ml\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     16\u001b[0m     \u001b[0;32mexcept\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 17\u001b[0;31m         \u001b[0;32mraise\u001b[0m \u001b[0mTypeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Cannot sum things of different types\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     18\u001b[0m     \u001b[0maverage\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mthesum\u001b[0m\u001b[0;34m/\u001b[0m\u001b[0mn\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     19\u001b[0m     \u001b[0;32mreturn\u001b[0m \u001b[0maverage\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mTypeError\u001b[0m: Cannot sum things of different types"
     ]
    }
   ],
   "source": [
    "myaverage(['a',1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "... Working the `unittest` module ...\n",
    "\n",
    "First we write our code into a file..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting mymath.py\n"
     ]
    }
   ],
   "source": [
    "%%file mymath.py\n",
    "def myaverage(l:list)->float:\n",
    "    \"\"\"\n",
    "    Calculate the average of list l\n",
    "    \n",
    "    Examples:\n",
    "    \n",
    "    >>> myaverage([1,2])\n",
    "    1.5\n",
    "    \n",
    "    \"\"\"\n",
    "    n = len(l)\n",
    "    if n==0:\n",
    "        raise ValueError(\"cant calculate mean of length 0 list\")\n",
    "    try:\n",
    "        thesum = sum(l)\n",
    "    except:\n",
    "        raise TypeError(\"Cannot sum things of different types\")\n",
    "    average = thesum/n\n",
    "    return average"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Fraction(1, 2)"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from fractions import Fraction\n",
    "myaverage([Fraction(1,3), Fraction(2,3)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "After adding some more incremental testing, we yhen we write a test file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting test_mymath.py\n"
     ]
    }
   ],
   "source": [
    "%%file test_mymath.py\n",
    "from mymath import myaverage\n",
    "import unittest\n",
    "import numbers\n",
    "from fractions import Fraction\n",
    "\n",
    "class MyTest(unittest.TestCase):\n",
    "    \n",
    "    def test_mymath(self):\n",
    "        self.assertEqual(myaverage([2,3]), 2.5)\n",
    "        \n",
    "    def test_mymath_result(self):\n",
    "        self.assertTrue(isinstance(myaverage([Fraction(1,3), Fraction(2,3)]), numbers.Real))\n",
    "\n",
    "    def test_char(self):\n",
    "        with self.assertRaises(TypeError):\n",
    "            myaverage(['a',3])\n",
    "            \n",
    "    def test_zerol(self):\n",
    "        with self.assertRaises(ValueError):\n",
    "            myaverage([])\n",
    "\n",
    "\n",
    "if __name__ == '__main__':\n",
    "    unittest.main()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Now we test it...unittest will pick up all \"tests\" in a folder or in a given file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "....\r\n",
      "----------------------------------------------------------------------\r\n",
      "Ran 4 tests in 0.000s\r\n",
      "\r\n",
      "OK\r\n"
     ]
    }
   ],
   "source": [
    "!python3 -m unittest"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "....\r\n",
      "----------------------------------------------------------------------\r\n",
      "Ran 4 tests in 0.000s\r\n",
      "\r\n",
      "OK\r\n"
     ]
    }
   ],
   "source": [
    "!python3 -m unittest test_mymath"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### A test harness: The `py.test` program\n",
    "\n",
    "As a group you should choose a test harness. I like `py.test`: it will run some `nosetes`s, `unittest`s, as well as its own set. Here we talk about `py.test`, but the principles are the same for any such framework."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Install thus. Make sure you are in the `py35` virtual environment.\n",
    "\n",
    "`pip install pytest`\n",
    "\n",
    "`pip install pytest-cov`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1m============================= test session starts ==============================\u001b[0m\r\n",
      "platform darwin -- Python 3.5.2, pytest-2.8.5, py-1.4.31, pluggy-0.3.1\r\n",
      "rootdir: /Users/rahul/Projects/private/cs207/FALL/lectures, inifile: \r\n",
      "plugins: cov-2.2.1\r\n",
      "\u001b[1m\r",
      "collecting 0 items\u001b[0m\u001b[1m\r",
      "collecting 4 items\u001b[0m\u001b[1m\r",
      "collecting 4 items\u001b[0m\u001b[1m\r",
      "collected 4 items \r\n",
      "\u001b[0m\r\n",
      "test_mymath.py ....\r\n",
      "\r\n",
      "\u001b[32m\u001b[1m=========================== 4 passed in 0.03 seconds ===========================\u001b[0m\r\n"
     ]
    }
   ],
   "source": [
    "!py.test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1m============================= test session starts ==============================\u001b[0m\n",
      "platform darwin -- Python 3.5.2, pytest-2.8.5, py-1.4.31, pluggy-0.3.1\n",
      "rootdir: /Users/rahul/Projects/private/cs207/FALL/lectures, inifile: \n",
      "plugins: cov-2.2.1\n",
      "collected 6 items \n",
      "\u001b[0m\n",
      "binsearch.py .\n",
      "mymath.py .\n",
      "test_mymath.py ....\n",
      "\n",
      "\u001b[1m\u001b[32m=========================== 6 passed in 0.25 seconds ===========================\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!py.test --doctest-modules "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Testing and coverage"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "In some sense, it would be nice to somehow check that every line in a program has been covered by a test, so that you know that line has not contributed to making something wrong. But this is hard to do: it would be hard to use normal input data to force a program to go through particular statements. So we settle for testing the important lines. The `pytest-cov` module makes sure that this works.\n",
    "\n",
    "Coverage does not mean that every edge case has been tried, but rather, every critical statement has been.\n",
    "\n",
    "To illustrate coverage, we add a median function..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "def mymedian(l:list)->float:\n",
    "    \"\"\"\n",
    "    Calculate the average of list l\n",
    "    \n",
    "    Examples:\n",
    "    \n",
    "    >>> mymedian([1,2,3])\n",
    "    2\n",
    "    \n",
    "    >>> mymedian([1,2,3,4])\n",
    "    2.5\n",
    "    \"\"\"\n",
    "    lsorted = sorted(l)\n",
    "    mididx = len(lsorted)//2\n",
    "    if len(lsorted) % 2 == 0: #even\n",
    "        return (lsorted[mididx-1] + lsorted[mididx])/2\n",
    "    else:\n",
    "        return lsorted[mididx]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting mymath.py\n"
     ]
    }
   ],
   "source": [
    "%%file mymath.py\n",
    "\n",
    "def myaverage(l:list)->float:\n",
    "    \"\"\"\n",
    "    Calculate the average of list l\n",
    "    \n",
    "    Examples:\n",
    "    \n",
    "    >>> myaverage([1,2])\n",
    "    1.5\n",
    "    \n",
    "    \"\"\"\n",
    "    n = len(l)\n",
    "    if n==0:\n",
    "        raise ValueError(\"cant calculate mean of length 0 list\")\n",
    "    try:\n",
    "        thesum = sum(l)\n",
    "    except:\n",
    "        raise TypeError(\"Cannot sum things of different types\")\n",
    "    average = thesum/n\n",
    "    return average\n",
    "\n",
    "def mymedian(l:list)->float:\n",
    "    \"\"\"\n",
    "    Calculate the average of list l\n",
    "    \n",
    "    Examples:\n",
    "    \n",
    "    >>> mymedian([1,2,3])\n",
    "    2\n",
    "    \n",
    "    >>> mymedian([1,2,3,4])\n",
    "    2.5\n",
    "    \"\"\"\n",
    "    lsorted = sorted(l)\n",
    "    mididx = len(lsorted)//2\n",
    "    if len(lsorted) % 2 == 0: #even\n",
    "        return (lsorted[mididx-1] + lsorted[mididx])/2\n",
    "    else:\n",
    "        return lsorted[mididx]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Using `py.test` with coverage"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "     1\t\r\n",
      "     2\tdef myaverage(l:list)->float:\r\n",
      "     3\t    \"\"\"\r\n",
      "     4\t    Calculate the average of list l\r\n",
      "     5\t    \r\n",
      "     6\t    Examples:\r\n",
      "     7\t    \r\n",
      "     8\t    >>> myaverage([1,2])\r\n",
      "     9\t    1.5\r\n",
      "    10\t    \r\n",
      "    11\t    \"\"\"\r\n",
      "    12\t    n = len(l)\r\n",
      "    13\t    if n==0:\r\n",
      "    14\t        raise ValueError(\"cant calculate mean of length 0 list\")\r\n",
      "    15\t    try:\r\n",
      "    16\t        thesum = sum(l)\r\n",
      "    17\t    except:\r\n",
      "    18\t        raise TypeError(\"Cannot sum things of different types\")\r\n",
      "    19\t    average = thesum/n\r\n",
      "    20\t    return average\r\n",
      "    21\t\r\n",
      "    22\tdef mymedian(l:list)->float:\r\n",
      "    23\t    \"\"\"\r\n",
      "    24\t    Calculate the average of list l\r\n",
      "    25\t    \r\n",
      "    26\t    Examples:\r\n",
      "    27\t    \r\n",
      "    28\t    >>> mymedian([1,2,3])\r\n",
      "    29\t    2\r\n",
      "    30\t    \r\n",
      "    31\t    >>> mymedian([1,2,3,4])\r\n",
      "    32\t    2.5\r\n",
      "    33\t    \"\"\"\r\n",
      "    34\t    lsorted = sorted(l)\r\n",
      "    35\t    mididx = len(lsorted)//2\r\n",
      "    36\t    if len(lsorted) % 2 == 0: #even\r\n",
      "    37\t        return (lsorted[mididx-1] + lsorted[mididx])/2\r\n",
      "    38\t    else:\r\n",
      "    39\t        return lsorted[mididx]"
     ]
    }
   ],
   "source": [
    "!cat -n mymath.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "     1\tfrom mymath import myaverage\r\n",
      "     2\timport unittest\r\n",
      "     3\timport numbers\r\n",
      "     4\tfrom fractions import Fraction\r\n",
      "     5\t\r\n",
      "     6\tclass MyTest(unittest.TestCase):\r\n",
      "     7\t    \r\n",
      "     8\t    def test_mymath(self):\r\n",
      "     9\t        self.assertEqual(myaverage([2,3]), 2.5)\r\n",
      "    10\t        \r\n",
      "    11\t    def test_mymath_result(self):\r\n",
      "    12\t        self.assertTrue(isinstance(myaverage([Fraction(1,3), Fraction(2,3)]), numbers.Real))\r\n",
      "    13\t\r\n",
      "    14\t    def test_char(self):\r\n",
      "    15\t        with self.assertRaises(TypeError):\r\n",
      "    16\t            myaverage(['a',3])\r\n",
      "    17\t            \r\n",
      "    18\t    def test_zerol(self):\r\n",
      "    19\t        with self.assertRaises(ValueError):\r\n",
      "    20\t            myaverage([])\r\n",
      "    21\t\r\n",
      "    22\t\r\n",
      "    23\tif __name__ == '__main__':\r\n",
      "    24\t    unittest.main()"
     ]
    }
   ],
   "source": [
    "!cat -n test_mymath.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1m============================= test session starts ==============================\u001b[0m\n",
      "platform darwin -- Python 3.5.2, pytest-2.8.5, py-1.4.31, pluggy-0.3.1 -- //anaconda/envs/py35/bin/python\n",
      "cachedir: .cache\n",
      "rootdir: /Users/rahul/Projects/private/cs207/FALL/lectures, inifile: \n",
      "plugins: cov-2.2.1\n",
      "collected 4 items \n",
      "\u001b[0m\n",
      "test_mymath.py::MyTest::test_char \u001b[32mPASSED\u001b[0m\n",
      "test_mymath.py::MyTest::test_mymath \u001b[32mPASSED\u001b[0m\n",
      "test_mymath.py::MyTest::test_mymath_result \u001b[32mPASSED\u001b[0m\n",
      "test_mymath.py::MyTest::test_zerol \u001b[32mPASSED\u001b[0m\n",
      "--------------- coverage: platform darwin, python 3.5.2-final-0 ----------------\n",
      "Name             Stmts   Miss  Cover\n",
      "------------------------------------\n",
      "mymath.py           16      5    69%\n",
      "test_mymath.py      17      1    94%\n",
      "------------------------------------\n",
      "TOTAL               33      6    82%\n",
      "\n",
      "\u001b[1m\u001b[32m=========================== 4 passed in 0.06 seconds ===========================\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!py.test --cov --verbose "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "And you can ask for a coverage report with missing lines"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1m============================= test session starts ==============================\u001b[0m\n",
      "platform darwin -- Python 3.5.2, pytest-2.8.5, py-1.4.31, pluggy-0.3.1\n",
      "rootdir: /Users/rahul/Projects/private/cs207/FALL/lectures, inifile: \n",
      "plugins: cov-2.2.1\n",
      "collected 4 items \n",
      "\u001b[0m\n",
      "test_mymath.py ....\n",
      "--------------- coverage: platform darwin, python 3.5.2-final-0 ----------------\n",
      "Name        Stmts   Miss  Cover   Missing\n",
      "-----------------------------------------\n",
      "mymath.py      16      5    69%   34-39\n",
      "\n",
      "\u001b[32m\u001b[1m=========================== 4 passed in 0.06 seconds ===========================\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!py.test --cov=mymath --cov-report term-missing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1m============================= test session starts ==============================\u001b[0m\n",
      "platform darwin -- Python 3.5.2, pytest-2.8.5, py-1.4.31, pluggy-0.3.1\n",
      "rootdir: /Users/rahul/Projects/private/cs207/FALL/lectures, inifile: \n",
      "plugins: cov-2.2.1\n",
      "collected 7 items \n",
      "\u001b[0m\n",
      "binsearch.py .\n",
      "mymath.py ..\n",
      "test_mymath.py ....\n",
      "--------------- coverage: platform darwin, python 3.5.2-final-0 ----------------\n",
      "Name        Stmts   Miss  Cover   Missing\n",
      "-----------------------------------------\n",
      "mymath.py      16      0   100%   \n",
      "\n",
      "\u001b[1m\u001b[32m=========================== 7 passed in 0.26 seconds ===========================\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!py.test --doctest-modules --cov=mymath --cov-report term-missing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### `mymedian` has its problems so we add tests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "ename": "TypeError",
     "evalue": "unorderable types: int() < str()",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-107-5382d15ce334>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mmymedian\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'a'\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m<ipython-input-79-32e76a35022b>\u001b[0m in \u001b[0;36mmymedian\u001b[0;34m(l)\u001b[0m\n\u001b[1;32m     11\u001b[0m     \u001b[0;36m2.5\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     12\u001b[0m     \"\"\"\n\u001b[0;32m---> 13\u001b[0;31m     \u001b[0mlsorted\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msorted\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ml\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     14\u001b[0m     \u001b[0mmididx\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlsorted\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m//\u001b[0m\u001b[0;36m2\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     15\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlsorted\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m%\u001b[0m \u001b[0;36m2\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;31m#even\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mTypeError\u001b[0m: unorderable types: int() < str()"
     ]
    }
   ],
   "source": [
    "mymedian(['a',1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {
    "collapsed": false,
    "scrolled": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "ename": "IndexError",
     "evalue": "list index out of range",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mIndexError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-108-3469736f214c>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mmymedian\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m<ipython-input-79-32e76a35022b>\u001b[0m in \u001b[0;36mmymedian\u001b[0;34m(l)\u001b[0m\n\u001b[1;32m     14\u001b[0m     \u001b[0mmididx\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlsorted\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m//\u001b[0m\u001b[0;36m2\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     15\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlsorted\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m%\u001b[0m \u001b[0;36m2\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;31m#even\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 16\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mlsorted\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mmididx\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0mlsorted\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mmididx\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m/\u001b[0m\u001b[0;36m2\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     17\u001b[0m     \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     18\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mlsorted\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mmididx\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mIndexError\u001b[0m: list index out of range"
     ]
    }
   ],
   "source": [
    "mymedian([])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Lets fix by adressing preconditions in a similar way."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting mymath.py\n"
     ]
    }
   ],
   "source": [
    "%%file mymath.py\n",
    "\n",
    "def myaverage(l:list)->float:\n",
    "    \"\"\"\n",
    "    Calculate the average of list l\n",
    "    \n",
    "    Examples:\n",
    "    \n",
    "    >>> myaverage([1,2])\n",
    "    1.5\n",
    "    \n",
    "    \"\"\"\n",
    "    n = len(l)\n",
    "    if n==0:\n",
    "        raise ValueError(\"cant calculate mean of length 0 list\")\n",
    "    try:\n",
    "        thesum = sum(l)\n",
    "    except:\n",
    "        raise TypeError(\"Cannot sum things of different types\")\n",
    "    average = thesum/n\n",
    "    return average\n",
    "\n",
    "def mymedian(l:list)->float:\n",
    "    \"\"\"\n",
    "    Calculate the average of list l\n",
    "    \n",
    "    Examples:\n",
    "    \n",
    "    >>> mymedian([1,2,3])\n",
    "    2\n",
    "    \n",
    "    >>> mymedian([1,2,3,4])\n",
    "    2.5\n",
    "    \"\"\"\n",
    "    try:\n",
    "        lsorted = sorted(l)\n",
    "    except:\n",
    "        raise TypeError(\"Unable to sort array\")\n",
    "    n = len(lsorted)\n",
    "    if n==0:\n",
    "        raise ValueError(\"cant calculate median of length 0 list\")\n",
    "    mididx = len(lsorted)//2\n",
    "    if len(lsorted) % 2 == 0: #even\n",
    "        return (lsorted[mididx-1] + lsorted[mididx])/2\n",
    "    else:\n",
    "        return lsorted[mididx]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "     1\t\r\n",
      "     2\tdef myaverage(l:list)->float:\r\n",
      "     3\t    \"\"\"\r\n",
      "     4\t    Calculate the average of list l\r\n",
      "     5\t    \r\n",
      "     6\t    Examples:\r\n",
      "     7\t    \r\n",
      "     8\t    >>> myaverage([1,2])\r\n",
      "     9\t    1.5\r\n",
      "    10\t    \r\n",
      "    11\t    \"\"\"\r\n",
      "    12\t    n = len(l)\r\n",
      "    13\t    if n==0:\r\n",
      "    14\t        raise ValueError(\"cant calculate mean of length 0 list\")\r\n",
      "    15\t    try:\r\n",
      "    16\t        thesum = sum(l)\r\n",
      "    17\t    except:\r\n",
      "    18\t        raise TypeError(\"Cannot sum things of different types\")\r\n",
      "    19\t    average = thesum/n\r\n",
      "    20\t    return average\r\n",
      "    21\t\r\n",
      "    22\tdef mymedian(l:list)->float:\r\n",
      "    23\t    \"\"\"\r\n",
      "    24\t    Calculate the average of list l\r\n",
      "    25\t    \r\n",
      "    26\t    Examples:\r\n",
      "    27\t    \r\n",
      "    28\t    >>> mymedian([1,2,3])\r\n",
      "    29\t    2\r\n",
      "    30\t    \r\n",
      "    31\t    >>> mymedian([1,2,3,4])\r\n",
      "    32\t    2.5\r\n",
      "    33\t    \"\"\"\r\n",
      "    34\t    try:\r\n",
      "    35\t        lsorted = sorted(l)\r\n",
      "    36\t    except:\r\n",
      "    37\t        raise TypeError(\"Unable to sort array\")\r\n",
      "    38\t    n = len(lsorted)\r\n",
      "    39\t    if n==0:\r\n",
      "    40\t        raise ValueError(\"cant calculate median of length 0 list\")\r\n",
      "    41\t    mididx = len(lsorted)//2\r\n",
      "    42\t    if len(lsorted) % 2 == 0: #even\r\n",
      "    43\t        return (lsorted[mididx-1] + lsorted[mididx])/2\r\n",
      "    44\t    else:\r\n",
      "    45\t        return lsorted[mididx]"
     ]
    }
   ],
   "source": [
    "!cat -n mymath.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing test_mymath2.py\n"
     ]
    }
   ],
   "source": [
    "%%file test_mymath2.py\n",
    "\n",
    "from pytest import raises\n",
    "from mymath import myaverage, mymedian\n",
    "\n",
    "def test_mymath_mean():\n",
    "    assert myaverage([9,3]) == 6\n",
    "\n",
    "def test_char():\n",
    "    with raises(TypeError):\n",
    "        myaverage(['a',3])\n",
    "\n",
    "def test_mymath():\n",
    "    assert mymedian([9,3, 6]) == 6\n",
    "    \n",
    "def test_zero_median():\n",
    "    with raises(ValueError):\n",
    "        mymedian([])\n",
    "        \n",
    "def test_char_median():\n",
    "    with raises(TypeError):\n",
    "        mymedian(['a', 3])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### About py.test...\n",
    "\n",
    "- pytest runs doctests and unittests.\n",
    "- any function prefixed with `test_` is a test.\n",
    "- will try nosetests\n",
    "\n",
    "(you dont have to use py.test, and can use nosetests if your team prefers it)\n",
    "\n",
    "You can continue to use `unittest`. `py.test` has the advantage that simple asserts are transformed the way stuff like `assertEqual` works. Not only that, `py.test` will give you some useful message as to why the assertion failed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1m============================= test session starts ==============================\u001b[0m\n",
      "platform darwin -- Python 3.5.2, pytest-2.8.5, py-1.4.31, pluggy-0.3.1\n",
      "rootdir: /Users/rahul/Projects/private/cs207/FALL/lectures, inifile: \n",
      "plugins: cov-2.2.1\n",
      "collected 9 items \n",
      "\u001b[0m\n",
      "test_mymath.py ....\n",
      "test_mymath2.py .....\n",
      "--------------- coverage: platform darwin, python 3.5.2-final-0 ----------------\n",
      "Name        Stmts   Miss  Cover   Missing\n",
      "-----------------------------------------\n",
      "mymath.py      22      1    95%   43\n",
      "\n",
      "\u001b[1m\u001b[32m=========================== 9 passed in 0.06 seconds ===========================\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!py.test --cov=mymath --cov-report term-missing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### TDD\n",
    "\n",
    "This is **Test Driven Development**, or the idea that you write the test first.\n",
    "\n",
    "From Younker's Agile Python Development:\n",
    "\n",
    ">TDD uses very small development cycles. Tests aren’t written for entire functions. They are written incrementally as the functions are composed. If the chunks get too large, a test- driven developer can always back down to a smaller chunk.\n",
    "The cycles have a distinct four-part rhythm. A test is written, and then it is executed to verify that it fails. A test that succeeds at this point tells you nothing about your new code. (Every day I encounter one that works when I don’t expect it to.) After the test fails, the associ- ated code is written, and then the test is run again. This time it should pass. If it passes, then the process begins anew.\n",
    "\n",
    "One advantage of this is that focusses you on the interface of your function/class, etc. The downside is that it precludes exploration and exploration tests, where you might try different interfaces. Another danger is write you might write tons of small units.\n",
    "\n",
    "Should you do it? TDD is certainly useful at times, but make sure you do it when you have a concrete idea where you are going..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Fixtures: setup common scaffolding for your tests\n",
    "\n",
    "Many times you need a common setup and teardown for multiple tests. You might need to populate some data, for example. This is done via fixtures.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing test_mymath3.py\n"
     ]
    }
   ],
   "source": [
    "%%file test_mymath3.py\n",
    "\n",
    "\n",
    "from pytest import fixture\n",
    "from mymath import myaverage, mymedian\n",
    "\n",
    "\n",
    "@fixture\n",
    "def input_data():\n",
    "    return dict(b=[4,5,6], a=['a', 1,2])\n",
    "\n",
    "def test_with_fixture(input_data):\n",
    "    assert myaverage(input_data['b']) == 5\n",
    "    assert mymedian(input_data['b']) == 5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1m============================= test session starts ==============================\u001b[0m\n",
      "platform darwin -- Python 3.5.2, pytest-2.8.5, py-1.4.31, pluggy-0.3.1\n",
      "rootdir: /Users/rahul/Projects/private/cs207/FALL/lectures, inifile: \n",
      "plugins: cov-2.2.1\n",
      "collected 13 items \n",
      "\u001b[0m\n",
      "binsearch.py .\n",
      "mymath.py ..\n",
      "test_mymath.py ....\n",
      "test_mymath2.py .....\n",
      "test_mymath3.py .\n",
      "--------------- coverage: platform darwin, python 3.5.2-final-0 ----------------\n",
      "Name        Stmts   Miss  Cover   Missing\n",
      "-----------------------------------------\n",
      "mymath.py      22      0   100%   \n",
      "\n",
      "\u001b[1m\u001b[32m========================== 13 passed in 0.29 seconds ===========================\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!py.test --doctest-modules --cov=mymath --cov-report term-missing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The scope above is the scope of your test function. If you want multiple functions to use your fixture, you must use module scope..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing test_mymath4.py\n"
     ]
    }
   ],
   "source": [
    "%%file test_mymath4.py\n",
    "\n",
    "\n",
    "from pytest import fixture\n",
    "from mymath import mymedian, myaverage\n",
    "\n",
    "\n",
    "@fixture(scope=\"module\")\n",
    "def input_data():\n",
    "    return dict(b=range(1000))\n",
    "\n",
    "def test_first(input_data):\n",
    "    assert mymedian(input_data['b'])  == 499.5\n",
    "\n",
    "def test_second(input_data):\n",
    "    assert myaverage(input_data['b']) == 499.5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1m============================= test session starts ==============================\u001b[0m\n",
      "platform darwin -- Python 3.5.2, pytest-2.8.5, py-1.4.31, pluggy-0.3.1\n",
      "rootdir: /Users/rahul/Projects/private/cs207/FALL/lectures, inifile: \n",
      "plugins: cov-2.2.1\n",
      "collected 2 items \n",
      "\u001b[0m\n",
      "test_mymath4.py ..\n",
      "\n",
      "\u001b[32m\u001b[1m=========================== 2 passed in 0.04 seconds ===========================\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!py.test --doctest-modules test_mymath4.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "You can add code for teardown as well...."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing test_mymath5.py\n"
     ]
    }
   ],
   "source": [
    "%%file test_mymath5.py\n",
    "from pytest import fixture\n",
    "import io\n",
    "#from docs\n",
    "@fixture\n",
    "def file_data(request): # The fixture MUST have a 'request' argument\n",
    "    text = open(\"test_mymath5.py\")\n",
    "\n",
    "    @request.addfinalizer\n",
    "    def teardown():\n",
    "        text.close()\n",
    "    return text\n",
    "\n",
    "def test_data_type(file_data):\n",
    "    assert isinstance(file_data, io.TextIOWrapper)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1m============================= test session starts ==============================\u001b[0m\r\n",
      "platform darwin -- Python 3.5.2, pytest-2.8.5, py-1.4.31, pluggy-0.3.1\r\n",
      "rootdir: /Users/rahul/Projects/private/cs207/FALL/lectures, inifile: \r\n",
      "plugins: cov-2.2.1\r\n",
      "\u001b[1m\r",
      "collecting 0 items\u001b[0m\u001b[1m\r",
      "collecting 0 items\u001b[0m\u001b[1m\r",
      "collecting 1 items\u001b[0m\u001b[1m\r",
      "collected 1 items \r\n",
      "\u001b[0m\r\n",
      "test_mymath5.py .\r\n",
      "\r\n",
      "\u001b[32m\u001b[1m=========================== 1 passed in 0.02 seconds ===========================\u001b[0m\r\n"
     ]
    }
   ],
   "source": [
    "!py.test --doctest-modules test_mymath5.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "You can use fixtures in `unittest` as well..but they look a little bit different."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing test_mymath6.py\n"
     ]
    }
   ],
   "source": [
    "%%file test_mymath6.py\n",
    "\n",
    "from mymath import myaverage, mymedian\n",
    "\n",
    "import unittest\n",
    "\n",
    "class MyTest(unittest.TestCase):\n",
    "    \n",
    "    def setUp(self):\n",
    "        self.b = range(1000)\n",
    "        \n",
    "    def tearDown(self):\n",
    "        del self.b\n",
    "        \n",
    "    def test_mymath(self):\n",
    "        self.assertEqual(myaverage(self.b), 499.5)\n",
    "        \n",
    "    def test_char(self):\n",
    "        with self.assertRaises(TypeError):\n",
    "            myaverage(['a',3])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1m============================= test session starts ==============================\u001b[0m\n",
      "platform darwin -- Python 3.5.2, pytest-2.8.5, py-1.4.31, pluggy-0.3.1\n",
      "rootdir: /Users/rahul/Projects/private/cs207/FALL/lectures, inifile: \n",
      "plugins: cov-2.2.1\n",
      "collected 18 items \n",
      "\u001b[0m\n",
      "binsearch.py .\n",
      "mymath.py ..\n",
      "test_mymath.py ....\n",
      "test_mymath2.py .....\n",
      "test_mymath3.py .\n",
      "test_mymath4.py ..\n",
      "test_mymath5.py .\n",
      "test_mymath6.py ..\n",
      "--------------- coverage: platform darwin, python 3.5.2-final-0 ----------------\n",
      "Name        Stmts   Miss  Cover   Missing\n",
      "-----------------------------------------\n",
      "mymath.py      22      0   100%   \n",
      "\n",
      "\u001b[32m\u001b[1m========================== 18 passed in 0.28 seconds ===========================\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!py.test --doctest-modules --cov=mymath --cov-report term-missing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Fakes and Mocks: make your tests self-contained\n",
    "\n",
    "But in general you shouldnt even be dealing with a file or db. You should **mock** it out. Your tests should nor rely on a database being there, or a network connection being possible. Otherwise, you dont know where your failure came from.\n",
    "\n",
    "This is not to say that you shouldnt test against a real network or real database. This is indeed the domain of acceptance tests. But unit tests where you are testing a layer or just a small unit should be isolated and not concern itself outside the unit.\n",
    "\n",
    "Typically i tend to \"test\" against the real deal, and then write a fake which simulates it for formal tests. You can see the procedure below for a mail client.\n",
    "\n",
    "(But first, you can set up fixtures in a `conftest.py`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing conftest.py\n"
     ]
    }
   ],
   "source": [
    "%%file conftest.py\n",
    "import pytest\n",
    "import smtplib\n",
    "\n",
    "@pytest.fixture(scope=\"module\")\n",
    "def smtp():\n",
    "    return smtplib.SMTP(\"smtp.gmail.com\", 587)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "This is what really happens:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(250,\n",
       " b'smtp.gmail.com at your service, [50.177.146.107]\\nSIZE 35882577\\n8BITMIME\\nSTARTTLS\\nENHANCEDSTATUSCODES\\nPIPELINING\\nCHUNKING\\nSMTPUTF8')"
      ]
     },
     "execution_count": 124,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import smtplib\n",
    "xxx=smtplib.SMTP(\"smtp.gmail.com\", 587)\n",
    "xxx.ehlo()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(250, b'2.0.0 OK v33sm10032802qtc.46 - gsmtp')"
      ]
     },
     "execution_count": 125,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xxx.noop()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "So, lets write a test file which uses the fixture defined in conftest and the smtp idea above"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing test_smtp.py\n"
     ]
    }
   ],
   "source": [
    "%%file test_smtp.py\n",
    "\n",
    "def test_ehlo(smtp):\n",
    "    response, msg = smtp.ehlo()\n",
    "    assert response == 250\n",
    "    assert b\"smtp.gmail.com\" in msg\n",
    "\n",
    "def test_noop(smtp):\n",
    "    response, msg = smtp.noop()\n",
    "    assert response == 250"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1m============================= test session starts ==============================\u001b[0m\n",
      "platform darwin -- Python 3.5.2, pytest-2.8.5, py-1.4.31, pluggy-0.3.1\n",
      "rootdir: /Users/rahul/Projects/private/cs207/FALL/lectures, inifile: \n",
      "plugins: cov-2.2.1\n",
      "collected 2 items \n",
      "\u001b[0m\n",
      "test_smtp.py ..\n",
      "\n",
      "\u001b[1m\u001b[32m=========================== 2 passed in 0.19 seconds ===========================\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!py.test test_smtp.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Ok, but we shouldnt be stuck if the smtp server is down....so we look at the output from the previous and \"fake\" it for the `ehlo` and `noop` ops.\n",
    "\n",
    "We \"mock\" the original object by **monkey-patching in** the `ehlo` and `noop` methods. The `monkeypatch` object in `py.test` will handle setting the original object's methods to the new function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting conftest.py\n"
     ]
    }
   ],
   "source": [
    "%%file conftest.py\n",
    "import pytest\n",
    "import smtplib\n",
    "\n",
    "def ehlo(smtpi):\n",
    "    return (250, b'smtp.gmail.com BLA BLA')\n",
    "def noop(smtpi):\n",
    "    return (250, b'BLA BLA gsmtp')\n",
    "\n",
    "@pytest.fixture(autouse=True)\n",
    "def smtp(monkeypatch):\n",
    "    #smtp_instance = smtplib.SMTP()\n",
    "    monkeypatch.setattr(smtplib.SMTP, \"ehlo\", ehlo)\n",
    "    monkeypatch.setattr(smtplib.SMTP, \"noop\", noop)\n",
    "    return smtplib.SMTP()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1m============================= test session starts ==============================\u001b[0m\n",
      "platform darwin -- Python 3.5.2, pytest-2.8.5, py-1.4.31, pluggy-0.3.1\n",
      "rootdir: /Users/rahul/Projects/private/cs207/FALL/lectures, inifile: \n",
      "plugins: cov-2.2.1\n",
      "collected 2 items \n",
      "\u001b[0m\n",
      "test_smtp.py ..\n",
      "\n",
      "\u001b[32m\u001b[1m=========================== 2 passed in 0.38 seconds ===========================\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!py.test test_smtp.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "This can also be done by using `mock` and `unittest`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing test_smtp2.py\n"
     ]
    }
   ],
   "source": [
    "%%file test_smtp2.py\n",
    "\n",
    "import unittest\n",
    "import unittest.mock as mock\n",
    "import smtplib\n",
    "\n",
    "def ehlo():\n",
    "    return (250, b'smtp.gmail.com BLA BLA')\n",
    "def noop():\n",
    "    return (250, b'BLA BLA gsmtp')\n",
    "\n",
    "class MyTest(unittest.TestCase):\n",
    "    \n",
    "    def setUp(self):\n",
    "        self.patcher = mock.patch(\"smtplib.SMTP\")\n",
    "        self.smtp = self.patcher.start()\n",
    "        self.smtp.ehlo=ehlo\n",
    "        self.smtp.noop=noop\n",
    "        \n",
    "    def tearDown(self):\n",
    "        self.patcher.stop()\n",
    "\n",
    "    def test_ehlo(self):\n",
    "        response, msg = self.smtp.ehlo()\n",
    "        assert response == 250\n",
    "        assert b\"smtp.gmail.com\" in msg\n",
    "\n",
    "    def test_noop(self):\n",
    "        response, msg = self.smtp.noop()\n",
    "        assert response == 250"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1m============================= test session starts ==============================\u001b[0m\n",
      "platform darwin -- Python 3.5.2, pytest-2.8.5, py-1.4.31, pluggy-0.3.1\n",
      "rootdir: /Users/rahul/Projects/private/cs207/FALL/lectures, inifile: \n",
      "plugins: cov-2.2.1\n",
      "collected 2 items \n",
      "\u001b[0m\n",
      "test_smtp2.py ..\n",
      "\n",
      "\u001b[32m\u001b[1m=========================== 2 passed in 0.35 seconds ===========================\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!py.test test_smtp2.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "In your project, you will want to mock out your database connections in your unit tests. There are other libraries you can use as well, such as `minimock`."
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}