03-modularization-documentation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Intermediate Python Lesson 2: Modularization and Documentation\n",
    "\n",
    "Now that we've covered some of the basic syntax and libraries in Python we can start to tackle our data analysis problem.\n",
    "We are interested in understanding the relationship between the weather and the number of mosquitos so that we can plan mosquito control measures.\n",
    "Since we want to apply these mosquito control measures at a number of different sites we need to understand how the relationship varies across sites.\n",
    "Remember that we have a series of CSV files with each file containing the data for a single location.\n",
    "\n",
    "## Learning Objectives\n",
    "\n",
    "* Write code for people, not computers\n",
    "* Break a program into chunks\n",
    "* Write and use functions in Python\n",
    "* Write useful documentation\n",
    "\n",
    "## Starting small\n",
    "\n",
    "\n",
    "When approaching computational tasks like this one it is typically best to start small,\n",
    "check each piece of code as you go,\n",
    "and make incremental changes.\n",
    "This helps avoid marathon debugging sessions\n",
    "because it's much easier to debug one small piece of the code at a time than to write 100 lines of code and\n",
    "then try to figure out all of the different bugs in it.\n",
    "\n",
    "Let's start by reading in the data from a single file and conducting a simple regression analysis on it.\n",
    "In fact, I would actually start by just importing the data and making sure that everything is coming in OK."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "attributes": {
     "classes": [
      " {.python"
     ],
     "id": ""
    },
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "d = pd.read_csv('data/A2_mosquito_data.csv')\n",
    "d"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The import seems to be working properly, so that's good news,\n",
    "but does anyone have anyone see anything about the code that they don't like?\n",
    "\n",
    "That's right.\n",
    "The variable name I've chosen for the data doesn't really communicate any information to anyone about what it's holding,\n",
    "which means that when I come back to my code next month to change something I'm going to have a more difficult time understanding what the code is actually doing.\n",
    "This brings us to one of our first major lessons for the morning,\n",
    "which is that in order to understand what our code is doing so that we can quickly make changes in the future,\n",
    "we need to *write code for people, not computers*,\n",
    "and an important first step is to *use meaningful varible names*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "attributes": {
     "classes": [
      " {.python"
     ],
     "id": ""
    },
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "data = pd.read_csv('data/A2_mosquito_data.csv')\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `.head()` method lets us just look at the first few rows of the data.\n",
    "A method is a function attached to an object that operates on that object.\n",
    "So in this case we can think of it as being equivalent to `head(data)`.\n",
    "\n",
    "Everything looks good,\n",
    "but either global warming has gotten *really* out of control or the temperatures are in degrees Fahrenheit.\n",
    "Let's convert them to Celsius before we get started.\n",
    "\n",
    "We don't need to reimport the data in our new cell because all of the executed cells in IPython Notebook share the same workspace.\n",
    "However, it's worth noting that if we close the notebook and then open it again it is necessary to rerun all of the individual blocks of code that a code block relies on before continuing.\n",
    "To rerun all of the cells in a notebook you can select `Cell -> Run All` from the menu."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "attributes": {
     "classes": [
      " {.python"
     ],
     "id": ""
    },
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "data['temperature'] = (data['temperature'] - 32) * 5 / 9.0\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's better.\n",
    "Now let's go ahead and conduct a regression on the data.\n",
    "We'll use the `statsmodels` library to conduct the regression."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "attributes": {
     "classes": [
      " {.python"
     ],
     "id": ""
    },
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import statsmodels.api as sm\n",
    "\n",
    "regr_results = sm.OLS.from_formula('mosquitos ~ temperature + rainfall', data).fit()\n",
    "regr_results.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see `statsmodels` lets us use the names of the columns in our dataframe\n",
    "to clearly specify the form of the statistical model we want to fit.\n",
    "This also makes the code more readable since the model we are fitting is written in a nice,\n",
    "human readable, manner.\n",
    "The `summary` method gives us a visual representation of the results.\n",
    "This summary is nice to look at, but it isn't really useful for doing more computation,\n",
    "so we can look up particular values related to the regression using the `regr_results` attributes.\n",
    "These are variables that are attached to `regr_results`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "regr_results.params"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "attributes": {
     "classes": [
      " {.python"
     ],
     "id": ""
    },
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "regr_results.rsquared"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we want to hold onto these values for later we can assign them to variables:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "attributes": {
     "classes": [
      " {.python"
     ],
     "id": ""
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "parameters = regr_results.params\n",
    "rsquared = regr_results.rsquared"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And then we can plot the observed data against the values predicted by our regression to visualize the results.\n",
    "First, remember to tell the notebook that we want our plots to appear in the notebook itself."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "attributes": {
     "classes": [
      " {.python"
     ],
     "id": ""
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "attributes": {
     "classes": [
      " {.python"
     ],
     "id": ""
    },
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "predicted = parameters[0] + parameters[1] * data['temperature'] + parameters[2] * data['rainfall']\n",
    "plt.plot(predicted, data['mosquitos'], 'ro')\n",
    "min_mosquitos, max_mosquitos = min(data['mosquitos']), max(data['mosquitos'])\n",
    "plt.plot([min_mosquitos, max_mosquitos], [min_mosquitos, max_mosquitos], 'k-')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "OK, great.\n",
    "So putting this all together we now have a piece of code that imports the modules we need,\n",
    "loads the data into memory, fits a regression to the data,\n",
    "and stores the parameters and fit of data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "attributes": {
     "classes": [
      " {.python"
     ],
     "id": ""
    },
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import statsmodels.api as sm\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "data = pd.read_csv('data/A2_mosquito_data.csv')\n",
    "data['temperature'] = (data['temperature'] - 32) * 5 / 9.0\n",
    "regr_results = sm.OLS.from_formula('mosquitos ~ temperature + rainfall', data).fit()\n",
    "parameters = regr_results.params\n",
    "rsquared = regr_results.rsquared\n",
    "predicted = parameters[0] + parameters[1] * data['temperature'] + parameters[2] * data['rainfall']\n",
    "plt.plot(predicted, data['mosquitos'], 'ro')\n",
    "min_mosquitos, max_mosquitos = min(data['mosquitos']), max(data['mosquitos'])\n",
    "plt.plot([min_mosquitos, max_mosquitos], [min_mosquitos, max_mosquitos], 'k-')\n",
    "print(parameters)\n",
    "print(\"R^2 = \", rsquared)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Functions\n",
    "\n",
    "\n",
    "The next thing we need to do is loop over all of the possible data files,\n",
    "but in order to do that we're going to need to grow our code some more.\n",
    "Since our brain can only easily hold 5-7 pieces of information at once,\n",
    "and our code already has more than that many pieces,\n",
    "we need to start breaking our code into manageable sized chunks.\n",
    "This will let us read and understand the code more easily and make it easier to reuse pieces of our code.\n",
    "We'll do this using functions.\n",
    "\n",
    "Functions in Python take the general form\n",
    "\n",
    "```python\n",
    "def function_name(inputs):\n",
    "    do stuff\n",
    "    return output\n",
    "```\n",
    "\n",
    "So, if we want to write a function that returns the value of a number squared we could use:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "attributes": {
     "classes": [
      " {.python"
     ],
     "id": ""
    },
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def square(x):\n",
    "    x_squared = x ** 2\n",
    "    return x_squared\n",
    "\n",
    "print(\"Four squared is\", square(4))\n",
    "print(\"Five squared is\", square(5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also just return the desired value directly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "attributes": {
     "classes": [
      " {.python"
     ],
     "id": ""
    },
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def square(x):\n",
    "    return x ** 2\n",
    "\n",
    "square(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And remember, if we want to use the result of the function later we need to store it somewhere."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "attributes": {
     "classes": [
      " {.python"
     ],
     "id": ""
    },
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "two_squared = square(2)\n",
    "two_squared"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Challenges\n",
    " \n",
    "> 1. Write a function that converts temperature from Fahrenheit to\n",
    ">    Celsius and use it to replace this line of code:\n",
    "> \n",
    ">    ```python\n",
    ">    data['temperature'] = (data['temperature'] - 32) * 5 / 9.0\n",
    ">    ```\n",
    "> \n",
    "> 2. Write a function called `analyze()` that takes `data` as an\n",
    ">    input, performs the regression, makes the observed-predicted\n",
    ">    plot, and returns `parameters`.\n",
    "> \n",
    "> *Walk through someone's result.\n",
    "> When discussing talk about different names.\n",
    "> E.g., fahr_to_celsius is better than temp_to_celsius since it is\n",
    "> explicit both the input and the output.\n",
    "> Talk about the fact that even though this doesn't save us any lines\n",
    "> of code it's still easier to read.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The call stack\n",
    "\n",
    "\n",
    "Let's take a closer look at what happens when we call a function. To make things clearer, we'll start by putting the initial value 32 in a variable and store the final result in one as well:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Don't worry if this fails\n",
    "%load_ext tutormagic"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "attributes": {
     "classes": [
      " {.python"
     ],
     "id": ""
    },
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#%%tutor --lang python3\n",
    "# Uncomment ^ that line if the previous cell ran OK\n",
    "def celsius_to_kelvin(tempC):\n",
    "    tempK = tempC + 273.15\n",
    "    return tempK\n",
    "\n",
    "original = 32.0\n",
    "final = celsius_to_kelvin(original)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Call Stack (Initial State)\n",
    "\n",
    "When the first three lines of this function are executed the function is created,\n",
    "but nothing happens.\n",
    "The function is like a recipe,\n",
    "it contains the information about how to do something,\n",
    "but it doesn't do so until you explicitly ask it to.\n",
    "We then create the variable `original` and assign the value 32.0 to it.\n",
    "The values `tempC` and `tempK` don't currently exist. \n",
    "\n",
    "#### Call Stack Immediately After Function Call\n",
    "\n",
    "When we call `celsius_to_kelvin`,\n",
    "Python creates another stack frame to hold the function's variables.\n",
    "Upon creation this stack frame only includes the inputs being passed to the function,\n",
    "so in our case `tempC`.\n",
    "As the function is executed variables created by the function are stored in the functions stack frame,\n",
    "so `tempC` is created in the `celsius_to_kelvin` stack frame.\n",
    "\n",
    "#### Call Stack At End Of Function Call\n",
    "\n",
    "When the call to `celsius_to_kelvin` returns a value,\n",
    "Python throws away `celsius_to_kelvin`'s stack frame,\n",
    "including all of the variables it contains,\n",
    "and creates a new variable\n",
    "in the original stack frame to hold the temperature in Celsius.\n",
    "\n",
    "#### Call Stack After End\n",
    "\n",
    "This global stack frame is always there;\n",
    "it holds the variables we defined outside the functions in our code.\n",
    "What it doesn't hold is the variables that were in the other stack frames.\n",
    "If we try to get the value of `tempC` or `tempK` after our functions have finished running,\n",
    "Python tells us that there's no such thing:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "attributes": {
     "classes": [
      " {.python"
     ],
     "id": ""
    },
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print(tempK)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The reason for this is encapsulation,\n",
    "and it's one of the key to writing correct, comprehensible programs.\n",
    "A function's job is to turn several operations into one so that we can think about\n",
    "a single function call instead of a dozen or a hundred statements each time we want to do something.\n",
    "That only works if functions don't interfere with each other by potentially changing the same variables;\n",
    "if they do, we have to pay attention to the details once again,\n",
    "which quickly overloads our short-term memory.\n",
    "\n",
    "## Testing Functions\n",
    "\n",
    "\n",
    "Once we start putting things into functions so that we can re-use them,\n",
    "we need to start testing that those functions are working correctly.\n",
    "The most basic thing we can do is some informal testing to make sure the function is doing what it is supposed to do.\n",
    "To see how to do this, let's write a function to center the values in a dataset prior to conducting statistical analysis. \n",
    "Centering means setting the mean of each variable to be the same value, typically zero."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "attributes": {
     "classes": [
      " {.python"
     ],
     "id": ""
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def center(data):\n",
    "    return data - data.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We could test this on our actual data,\n",
    "but since we don't know what the values ought to be,\n",
    "it will be hard to tell if the result was correct.\n",
    "Instead, let's create a made up data frame where we know what the result should look like."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "attributes": {
     "classes": [
      " {.python"
     ],
     "id": ""
    },
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "test_data = pd.DataFrame([[1, 1], [1, 2]])\n",
    "test_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we've made some test data we need to figure out what we think the result should be\n",
    "and we need to do this *before* we run the test.\n",
    "This is important because we are biased to believe that any result we get back is correct,\n",
    "and we want to avoid that bias.\n",
    "This also helps make sure that we are confident in what we want the code to do.\n",
    "So, what should the result of running `center(data)` be?\n",
    "\n",
    "OK, let's go ahead and run the function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "attributes": {
     "classes": [
      " {.python"
     ],
     "id": ""
    },
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "center(test_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That looks right,\n",
    "so let's try `center` on our real data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "attributes": {
     "classes": [
      " {.python"
     ],
     "id": ""
    },
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "data = pd.read_csv('data/A2_mosquito_data.csv')\n",
    "center(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's hard to tell from the default output whether the result is correct,\n",
    "but there are a few simple tests that will reassure us:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "attributes": {
     "classes": [
      " {.python"
     ],
     "id": ""
    },
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print('original mean:')\n",
    "print(data.mean())\n",
    "centered = center(data)\n",
    "print()\n",
    "print('mean of centered data:')\n",
    "print(centered.mean())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The mean of the centered data is very close to zero;\n",
    "it's not quite zero because of floating point precision issues.\n",
    "We can even go further and check that the standard deviation hasn't changed\n",
    "(which it shouldn't if we've just centered the data):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "attributes": {
     "classes": [
      " {.python"
     ],
     "id": ""
    },
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print('std dev before and after:')\n",
    "print(data.std())\n",
    "print()\n",
    "print(centered.std())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The standard deviations look the same.\n",
    "It's still possible that our function is wrong,\n",
    "but it seems unlikely enough that we we're probably in good shape for now.\n",
    "\n",
    "Testing is really important when writing scientific code.\n",
    "If you haven't checked that your code works properly, you can't be confident in your results.\n",
    "We'll talk more about testing tomorrow.\n",
    "\n",
    "## Documentation\n",
    "\n",
    "\n",
    "OK, the `center` function seems to be working fine.\n",
    "Does anyone else see anything that's missing before we move on?\n",
    "\n",
    "Yes, we should write some *documentation*\n",
    "to remind ourselves later what it's for and how to use it.\n",
    "This function may be fairly straightforward,\n",
    "but in most cases it won't be so easy to remember exactly what a function is doing in a few months.\n",
    "Just imagine looking at our `analyze` function a few months in the future\n",
    "and trying to remember exactly what it was doing just based on the code.\n",
    "\n",
    "The usual way to put documentation in code is to add *comments* like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "attributes": {
     "classes": [
      " {.python"
     ],
     "id": ""
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# center(data): return a new DataFrame containing the original data centered around zero.\n",
    "def center(data, desired):\n",
    "    return data - data.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There's a better way to do this in Python.\n",
    "If the first thing in a function is a string that isn't assigned to a variable,\n",
    "that string is attached to the function as its documentation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "attributes": {
     "classes": [
      " {.python"
     ],
     "id": ""
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def center(data, desired):\n",
    "    \"\"\"Return a new DataFrame containing the original data centered around zero.\"\"\"\n",
    "    return data - data.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is better because we can now ask Python's built-in help system to show us the documentation for the function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "attributes": {
     "classes": [
      " {.python"
     ],
     "id": ""
    },
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "help(center)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A string like this is called a *docstring*\n",
    "and there are also automatic documentation generators that use these docstrings to produce documentation for users.\n",
    "We use triple quotes because\n",
    "it allows us to include multiple lines of text and because it is considered good Python style."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "attributes": {
     "classes": [
      " {.python"
     ],
     "id": ""
    },
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def center(data):\n",
    "    \"\"\"Return a new array containing the original data centered on zero\n",
    "    \n",
    "    Example:\n",
    "    >>> import pandas\n",
    "    >>> data = pandas.DataFrame([[0, 1], [0, 2])\n",
    "    >>> center(data)\n",
    "       0    1\n",
    "    0  0 -0.5\n",
    "    1  0  0.5\n",
    "\n",
    "     \n",
    "    \"\"\"\n",
    "    return data - data.mean()\n",
    "\n",
    "help(center)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Challenge\n",
    "\n",
    "\n",
    "1. Test your temperature conversion function to make sure it's working\n",
    "   (think about some temperatures that you easily know the conversion for).\n",
    "2. Add documentation to both the temperature conversation function and the analysis function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Looping over files\n",
    "\n",
    "\n",
    "So now our code looks something like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "attributes": {
     "classes": [
      " {.python"
     ],
     "id": ""
    },
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import statsmodels.api as sm\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "def fahr_to_celsius(tempF):\n",
    "    \"\"\"Convert fahrenheit to celsius\"\"\"\n",
    "    tempC = (tempF - 32) * 5 / 9.0\n",
    "    return tempC\n",
    "\n",
    "def analyze(data):\n",
    "    \"\"\"Perform regression analysis on mosquito data\n",
    "    \n",
    "    Takes a dataframe as input that includes columns named 'temperature',\n",
    "    'rainfall', and 'mosquitos'.\n",
    "    \n",
    "    Performs a multiple regression to predict the number of mosquitos.\n",
    "    Creates an observed-predicted plot of the result and\n",
    "    returns the parameters of the regression.\n",
    "    \n",
    "    \"\"\"\n",
    "    regr_results = sm.OLS.from_formula('mosquitos ~ temperature + rainfall', data).fit()\n",
    "    parameters = regr_results.params\n",
    "    predicted = parameters[0] + parameters[1] * data['temperature'] + parameters[2] * data['rainfall']\n",
    "    plt.figure()\n",
    "    plt.plot(predicted, data['mosquitos'], 'ro')\n",
    "    min_mosquitos, max_mosquitos = min(data['mosquitos']), max(data['mosquitos'])\n",
    "    plt.plot([min_mosquitos, max_mosquitos], [min_mosquitos, max_mosquitos], 'k-')\n",
    "    return parameters\n",
    "\n",
    "data = pd.read_csv('data/A2_mosquito_data.csv')\n",
    "data['temperature'] = fahr_to_celsius(data['temperature'])\n",
    "regr_results = analyze(data)\n",
    "print(regr_results)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we want to loop over all of the possible data files,\n",
    "and to do that we need to know their names.\n",
    "If we only had a dozen files we could write them all down,\n",
    "but if we have hundreds of files or the filenames change then that won't really work.\n",
    "Fortunately Python has a built in library to help us find the files we want to work with called `glob`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "attributes": {
     "classes": [
      " {.python"
     ],
     "id": ""
    },
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import glob\n",
    "\n",
    "filenames = glob.glob('data/*.csv')\n",
    "filenames"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The object returned by `glob` is a list of strings.\n",
    "A list is a Python data type that holds a group of potentially heterogenous values.\n",
    "That means it can hold pretty much anything,\n",
    "including functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "attributes": {
     "classes": [
      " {.python"
     ],
     "id": ""
    },
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "mylist = [1, 'a', center]\n",
    "mylist"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case all of the values are strings\n",
    "that contain the names of all of the files that match the expression given to `glob`,\n",
    "so in this case all of the files with the `.csv` extension.\n",
    "\n",
    "Let's restrict the filenames a little more finely,\n",
    "so that we don't accidentally get any data we don't want,\n",
    "and print out the filenames one at a time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "attributes": {
     "classes": [
      " {.python"
     ],
     "id": ""
    },
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "filenames = glob.glob('data/*.csv')\n",
    "for filename in filenames:\n",
    "    print(filename)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Challenge\n",
    "\n",
    "\n",
    "Modify your code to loop over all of the files in your directory,\n",
    "making an observed-predicted plot for each file and printing the parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}