07. Machine Learning/Machine Learning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "ML1",
     "locked": false,
     "solution": false
    }
   },
   "source": [
    "# Machine Learning\n",
    "\n",
    "## Table of Contents\n",
    "\n",
    "- [Setup](#Setup)\n",
    "\n",
    "    - [Setup - Imports](#Setup---Imports)\n",
    "    - [Setup - Database connection](#Setup---Database-connection)\n",
    "    \n",
    "        - [Setup - Database connection - MySQL](#Setup---Database-connection---MySQL)\n",
    "        - [Setup - Database connection - PostgreSQL](#Setup---Database-connection---PostgreSQL)\n",
    "\n",
    "- [Motivation and Background](#Motivation-and-Background)\n",
    "- [Data Basics](#Data-Basics)\n",
    "- [Understanding the Data](#Understanding-the-Data)\n",
    "\n",
    "    - [_Exercise 1 - descriptive statistics_](#Exercise-1---descriptive-statistics)\n",
    "\n",
    "- [Cleaning and Subsetting Data](#Cleaning-and-Subsetting-Data)\n",
    "\n",
    "    - [_Exercise 2 - function `cleanData`_](#Exercise-2---function-cleanData)\n",
    "    - [Checkpoint - save `cleaned_data_frame`](#Checkpoint---save-cleaned_data_frame)\n",
    "\n",
    "- [Model Selection and Assessment](#Model-Selection-and-Assessment)\n",
    "\n",
    "    - [Cleanup - clear `cleaned_data_frame`](#Cleanup---clear-cleaned_data_frame)\n",
    "    - [Training a model](#Training-a-model)\n",
    "    - [1. Split data into training and testing data sets](#1.-Split-data-into-training-and-testing-data-sets)\n",
    "    - [Cleanup - clear X and y data frames](#Cleanup---clear-X-and-y-data-frames)\n",
    "    - [2. Model Build - Fit model](#2.-Model-Build---Fit-model)\n",
    "    - [3. Evaluate Performance - accuracy](#3.-Evaluate-Performance---accuracy)\n",
    "\n",
    "- [_Exercise 3 - Train, fit and evaluate your model_](#Exercise-3---Train,-fit-and-evaluate-your-model)\n",
    "\n",
    "- [References](#References)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Setup\n",
    "\n",
    "- Back to [Table of Contents](#Table-of-Contents)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup - Imports\n",
    "\n",
    "- Back to [Table of Contents](#Table-of-Contents)\n",
    "\n",
    "Before we begin, run the code cell below to initialize the libraries we'll be using in this assignment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import datetime\n",
    "import gc\n",
    "import math\n",
    "import numpy\n",
    "import pandas\n",
    "import sqlalchemy\n",
    "\n",
    "# IPython\n",
    "from IPython.core.display import Image\n",
    "\n",
    "# scikit-learn\n",
    "from sklearn.cross_validation import train_test_split\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.metrics import accuracy_score\n",
    "from sklearn import metrics\n",
    "\n",
    "# Database connection packages - one or the other will be imported below:\n",
    "#import pymysql\n",
    "#import psycopg2\n",
    "\n",
    "print( \"packages imported at \" + str( datetime.datetime.now() ) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup - Database connection\n",
    "\n",
    "- Back to [Table of Contents](#Table-of-Contents)\n",
    "\n",
    "`Pandas` uses a database engine to connect to databases (via the `SQLAlchemy` Python package).  In the code cell below for your database (either MySQL or PostgreSQL), place your database username and password in the variables '`db_username`' and '`db_password`', then run the cell to create a `SQLAlchemy` database engine connected to our class database server for Pandas to use."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setup - Database connection - MySQL\n",
    "\n",
    "- Back to [Table of Contents](#Table-of-Contents)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# ==> MySQL\n",
    "import pymysql\n",
    "\n",
    "# set up database credentials\n",
    "db_username = \"<username>\"\n",
    "db_password = \"<password>\"\n",
    "db_host = \"localhost\"\n",
    "db_port = \"3306\"\n",
    "db_database = \"homework\"\n",
    "db_encoding = \"utf8\"\n",
    "\n",
    "# Create database connection for pandas.\n",
    "connection_string = \"mysql+pymysql://\" + db_username + \":\" + db_password + \"@\" + db_host + \":\" + db_port + \"/\" + db_database + \"?charset=\" + db_encoding\n",
    "pandas_db = sqlalchemy.create_engine( connection_string )\n",
    "\n",
    "print( \"sqlalchemy mysql database engine created at \" + str( datetime.datetime.now() ) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setup - Database connection - PostgreSQL\n",
    "\n",
    "- Back to [Table of Contents](#Table-of-Contents)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# ==> PostgreSQL\n",
    "import psycopg2\n",
    "import psycopg2.extras\n",
    "\n",
    "# set up database credentials\n",
    "db_username = \"<username>\"\n",
    "db_password = \"<password>\"\n",
    "db_host = \"localhost\"\n",
    "db_port = \"5432\"\n",
    "db_database = \"homework\"\n",
    "db_encoding = \"utf8\"\n",
    "\n",
    "db_host = \"cuspdev.local\"\n",
    "db_username = \"jonathanmorgan\"\n",
    "db_password = \"today123\"\n",
    "\n",
    "# Create database connection for pandas.\n",
    "connection_string = \"postgresql+psycopg2://\" + db_username + \":\" + db_password + \"@\" + db_host + \":\" + db_port + \"/\" + db_database + \"?client_encoding=\" + db_encoding\n",
    "pandas_db = sqlalchemy.create_engine( connection_string )\n",
    "\n",
    "print( \"sqlalchemy postgresql database engine created at \" + str( datetime.datetime.now() ) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "ML2",
     "locked": false,
     "solution": false
    }
   },
   "source": [
    "# Motivation and Background\n",
    "\n",
    "- Back to [Table of Contents](#Table-of-Contents)\n",
    "\n",
    "Research in science policy often involves making use of publically available datasets.  Today, we will be diving into the National Institutes for Health (NIH) grant data.  NIH provides a myriad of information about its grants which you can access here: http://projectreporter.nih.gov/reporter.cfm. Unfortunately, since the information draws on disparate sources like eRA databases, Medline, PubMed Central, the NIH Intramural Database, and iEdison, it is often not complete. \n",
    "\n",
    "In this workbook, we will examine one application of machine learning that deals with predicting missing information. Often in sciece policy research, we are interested in knowing what areas of science are being funded.  There are diverse techniques to determine a grant's area of science, including assessing a grants topic using text analytics like topic modeling (covered in the Text Analysis workbook).\n",
    "\n",
    "The best option is an explicit and standardized identifier that is present on every record.  The NIH grant data doesn't have an area of science taxonomy.  It does, however, provide a place to store an expicit, though potentially not standardized or totally accurate, indicator of a grant's overall topic area - the academic department of the grant's primary investigator (PI).  Unfortunately, this variable is often left empty.  In this workbook, we will walk through the process of imputing values for a missing categorical variable using the example of predicting the academic department of a given grant's primary investigator."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "ML3",
     "locked": false,
     "solution": false
    }
   },
   "source": [
    "# Data Basics\n",
    "\n",
    "- Back to [Table of Contents](#Table-of-Contents)\n",
    "\n",
    "The data we need for this exercise is in the '`umetricsgrants`' database in MySQL on the class server.  For more details on this data, see the data schemas we've provided in class:\n",
    "\n",
    "- Database Schema PowerPoint (also includes USPTO and StarMetrics data) - [http://jpsmonline.umd.edu/mod/resource/view.php?id=2387](http://jpsmonline.umd.edu/mod/resource/view.php?id=2387)\n",
    "- UmetricsGrants Schema (image) - [http://jpsmonline.umd.edu/mod/resource/view.php?id=2384](http://jpsmonline.umd.edu/mod/resource/view.php?id=2384)\n",
    "\n",
    "In particular, in the '`nih_project`' table, there is a variable '`org_dept`' that details the department of the PI of the grant.  This will be our outcome variable of interest (the variable we will try to predict when it is missing).  Once you start trying your own models, you can use as predictors variables from any of the tables whose names start with '`nih_`'.  To get us started, however, we have taken a subset of columns from these tables and placed them in a single table named '`machine_learning`' in the '`homework`' database.\n",
    "\n",
    "In the past (in the Database Basics and Text Analysis assignment notebooks) we have interacted with the database using SQL and a direct python connection to the database.  In this lesson, we'll be using a different program - the **pandas** package ( site: [http://pandas.pydata.org/](http://pandas.pydata.org/); doc: [http://pandas.pydata.org/pandas-docs/stable/index.html](http://pandas.pydata.org/pandas-docs/stable/index.html) ) - to read in and manipulate data.  Pandas provides an alternative to reading data directly from MySQL that stores the data in special table format called a \"data frame\" that allows for easy statistical analysis and can be directly used for machine learning.\n",
    "\n",
    "Pandas uses a database engine to connect to databases (via the SQLAlchemy Python package).  We created this engine in the section [Setup - Database connection](#Setup---Database-connection) above.\n",
    "\n",
    "Next, we will use this database connection to have pandas read in the data stored in the '`machine_learning`' table.  Pandas has a set of Input/Output tools that let it read from and write to a large variety of tabular data formats ( [http://pandas.pydata.org/pandas-docs/stable/io.html](http://pandas.pydata.org/pandas-docs/stable/io.html) ), including CSV and Excel files, databases via SQL, JSON files, and SAS and Stata data files.  In the example below, we'll use the `pandas.read_sql()` function to read the results of an SQL query into a pandas data frame.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "data_frame = pandas.read_sql( 'Select * from machine_learning;', pandas_db )\n",
    "\n",
    "print( \"machine_learning DataFrame loaded at \" + str( datetime.datetime.now() ) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, lets look at what the data looks like.  The DataFrame method '`data_frame.head( number_of_rows )`' outputs the  first `number_of_rows` rows in a data frame.  Lets look at the first five rows in our data.\n",
    "\n",
    "In the code cell below, there are two ways to output this information.  If you just call the method, you'll get an HTML table output directly into the ipython notebook.  If you pass the results of the method to the \"`print()`\" function, you'll get text output that works outside of jupyter/ipython."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# to get a pretty tabular view, just call the method.\n",
    "data_frame.head( 5 )\n",
    "\n",
    "# to get a text-based view, print() the call to the method.\n",
    "#print( data_frame.head( 5 ) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Understanding the Data\n",
    "\n",
    "- Back to [Table of Contents](#Table-of-Contents)\n",
    "\n",
    "In pandas, our data is represented by a DataFrame. You can think of data frames as a giant spreadsheet which you can program, with the data for each column stored in its own list that pandas calls a Series (or vector of values), along with a set of methods (another name for functions that are tied to objects) that make managing data in pandas easy.\n",
    "\n",
    "A Series is a list of values each of which can also have a label, which pandas calls an \"index\", and which generally is used to store names of columns when you retrieve a Series that represents a row, and IDs of rows when you retrieve a Series that represents a column of data in a table.\n",
    "\n",
    "While DataFrames and Series are separate objects, they may share the same methods where those methods make sense in both a table and list context (`head()` and `tail()`, as used in examples in this notebook, for example).\n",
    "\n",
    "More details on pandas data structures:\n",
    "\n",
    "- Data Structures overview: [http://pandas.pydata.org/pandas-docs/stable/dsintro.html](http://pandas.pydata.org/pandas-docs/stable/dsintro.html)\n",
    "- Series specifics: [http://pandas.pydata.org/pandas-docs/stable/dsintro.html#series](http://pandas.pydata.org/pandas-docs/stable/dsintro.html#series)\n",
    "- DataFrame specifics: [http://pandas.pydata.org/pandas-docs/stable/dsintro.html#dataframe](http://pandas.pydata.org/pandas-docs/stable/dsintro.html#dataframe)\n",
    "\n",
    "For example, if you want to look at the last five values in the `org_dept` column, you can retrieve the Series that contains the `org_dept` column's data using square-bracket notation (like you'd use to access a value in a dictionary):\n",
    "\n",
    "    data_frame[ \"org_dept\" ]\n",
    "    \n",
    "then call the \"`tail( number_of_rows )`\" method (the opposite of \"`head()`\") on the Series to get `number_of_rows` values from the end of the column's data.  To get the last 5 values in the `org_dept` column:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# get vector of \"org_dept\" column values from data frame\n",
    "org_dept_column_series = data_frame[ \"org_dept\" ]\n",
    "\n",
    "# see the last 5 values in the vector.\n",
    "print( org_dept_column_series.tail( 5 ) )\n",
    "\n",
    "# It is also OK to chain together, but I did not above for clarity's sake, and in\n",
    "#    general, be wary of doing too many things on one line.\n",
    "# data_frame[ \"org_dept\" ].tail( 5 )\n",
    "\n",
    "# empty org_dept_column_series variable and garbage collect, to conserve memory\n",
    "org_dept_column_series = None\n",
    "gc.collect()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To see how the data is stored internally, we can reference the \"`data_frame.dtypes`\" variable, which contains a pandas Series object with the name of each column in your data frame the label for the type of the data in that column:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "data_frame.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Lets look at these database columns one by one:\n",
    "\n",
    "* **`application_id`** - Unique identifier for each grant\n",
    "* **`cfda_code`** - CFDA contains detailed program descriptions for 2,292 Federal assistance programs: [https://www.cfda.gov/](https://www.cfda.gov/)\n",
    "* **`year`** - Year in which grant was awarded\n",
    "* **`activity`** - A 3-character code identifying the grant, contract, or intramural activity through which a project is supported. Here is a list of activity codes: [http://grants.nih.gov/grants/funding/ac_search_results.htm](http://grants.nih.gov/grants/funding/ac_search_results.htm)\n",
    "* **`administering_ic`** - Administering Institute or Center - A two-character code to designate the agency, NIH Institute, or Center administering the grant. See [definitions]: [](http://grants.nih.gov/grants/glossary.htm#I14).\n",
    "* **`arra_funded`** - “Y” indicates a project supported by funds appropriated through the American Recovery and Reinvestment Act of 2009.\n",
    "* **`org_name`** - The name of the educational institution, research organization, business, or government agency receiving funding for the grant, contract, cooperative agreement, or intramural project.  \n",
    "* **`org_dept`** - The departmental affiliation of the contact principal investigator for a project, using a standardized categorization of departments.  Names are available only for medical school departments.\n",
    "* **`study_section`** - A designator of the legislatively-mandated panel of subject matter experts that reviewed the research grant application for scientific and technical merit.\n",
    "* **`total_cost`** - Total project funding from all NIH Institute and Centers for a given fiscal year.\n",
    "* **`topic_id`** - Using text analysis techniques, a topic_id was assigned to each grant. This topic_id is a key for that topic. You can see what the topic contains by looking in the `topiclda_text` table in `umetricsgrants` database.\n",
    "* **`ed_inst_type`** - Generic name for the grouping of components across an institution who has applied for or receives NIH funding.\n",
    "\n",
    "In the examples that follow, we'll only be using the columns in this table as predictors for models.  Once you get to the last exercise, however, when you are trying to make as accurate a model as you can, you will be free to use any of the columns in the table `nih_project` in the `umetricsgrants` database.  A complete description of all variables in that table: [http://exporter.nih.gov/about.aspx](http://exporter.nih.gov/about.aspx)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 1 - descriptive statistics\n",
    "\n",
    "- Return to [Table of Contents](#Table-of-Contents)\n",
    "\n",
    "Pandas provides some great functions for descriptive statistics ( [http://pandas.pydata.org/pandas-docs/stable/basics.html#descriptive-statistics](http://pandas.pydata.org/pandas-docs/stable/basics.html#descriptive-statistics) ).  Some examples:\n",
    "\n",
    "- **`describe()`** - \"computes a variety of summary statistics about a Series or the columns of a DataFrame (excluding NAs of course)\" ( from [http://pandas.pydata.org/pandas-docs/stable/basics.html#summarizing-data-describe](http://pandas.pydata.org/pandas-docs/stable/basics.html#summarizing-data-describe) )\n",
    "\n",
    "    - includes the count of values, mean, standard deviation, min, 25%, 50%, and 75% values, and the max.\n",
    "\n",
    "- **`head()` and `tail()`**, shown above - \"To view a small sample of a Series or DataFrame object, use the `head()` and `tail()` methods. The default number of elements to display is five, but you may pass a custom number.\" ( from [http://pandas.pydata.org/pandas-docs/stable/basics.html#head-and-tail](http://pandas.pydata.org/pandas-docs/stable/basics.html#head-and-tail) )\n",
    "- **`value_counts()`** - The `value_counts()` \"series method and top-level function computes a histogram of a one-dimensional array of values.\" ( from [http://pandas.pydata.org/pandas-docs/stable/basics.html#value-counts-histogramming-mode](http://pandas.pydata.org/pandas-docs/stable/basics.html#value-counts-histogramming-mode) ).  This method returns a Series of the counts of the number of times each unique value in the column is present in the column (also known as frequencies), from largest count to least, with the value itself the label for each row.\n",
    "\n",
    "For the first part of exercise 1, we will combine some of these methods for calculating descriptive statistics into a function that accepts a DataFrame of data from our `homework.MachineLearning2` table, then calculates and returns both the descriptives for the columns in the table and the top ten most frequently referenced departments.\n",
    "\n",
    "We will be making a function that returns multiple values to help you understand how this works in Python, since some of the machine learning functions used below return multiple values.\n",
    "\n",
    "In a Python function, if you want to return multiple values, you place each in the line with your return statement, separated by commas.  This is like a list of variables, but you don't need to put it in square brackets.  You just place the items after the `return` keyword.  So, if you wanted to write a function that returns the circumference and area of a circle when passed the radius of a circle (run the cell below):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def calculateCircleInfo( radius_IN ):\n",
    "    \n",
    "    '''\n",
    "    Accepts radius of circle.  Calculates circumference and area of circle,\n",
    "       returns them both.\n",
    "    '''\n",
    "    \n",
    "    # return references\n",
    "    circumference_OUT = -1\n",
    "    area_OUT = -1\n",
    "    \n",
    "    # got a radius?\n",
    "    if ( ( radius_IN is not None ) and ( radius_IN != \"\" ) and ( radius_IN > -1 ) ):\n",
    "        \n",
    "        # yes.  calculate circumference...\n",
    "        circumference_OUT = 2 * math.pi * radius_IN\n",
    "        \n",
    "        # ...and calculate area\n",
    "        area_OUT = math.pi * ( radius_IN ** 2 )\n",
    "        \n",
    "    #-- END check to see if radius is populated and not negative (we'll allow 0...). --#\n",
    "    \n",
    "    # return both diameter and area.\n",
    "    return circumference_OUT, area_OUT\n",
    "    \n",
    "#-- END function calculateCircleInfo() --#\n",
    "\n",
    "print( \"function calculateCircleInfo() defined at \" + str( datetime.datetime.now() ) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When a function or method returns more than one thing, it returns the items in a tuple.  When you call a function or method that returns more than one thing, to accept all of the items it returns, you can either:\n",
    "\n",
    "- assign the results to a single variable.  This variable will contain a tuple, in which you can then reference the individual values returned by the function or method using square bracket notation, by index (starting with 0).\n",
    "- assign the results of the function to a list of the same number of variables, separated by commas, to the left of the assignment operator (the equal sign - \"=\").\n",
    "\n",
    "For example, to calculate the circumference and area of a circle with radius of 3:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# declare variables\n",
    "radius = -1\n",
    "circumference = -1\n",
    "area = -1\n",
    "circle_info_tuple = None\n",
    "\n",
    "# set radius\n",
    "radius = 3\n",
    "\n",
    "# calculate circumference and area - individual values\n",
    "circumference, area = calculateCircleInfo( radius )\n",
    "\n",
    "# print the results\n",
    "print( \"[values] - For circle of radius \" + str( radius ) + \": - circumference = \" + str( circumference ) + \" - area = \" + str( area ) ) \n",
    "\n",
    "# calculate circumference and area - tuple\n",
    "circle_info_tuple = calculateCircleInfo( radius )\n",
    "\n",
    "# unpack value using square-bracket tuple notation.\n",
    "circumference = circle_info_tuple[ 0 ] # first\n",
    "area = circle_info_tuple[ 1 ] # second\n",
    "\n",
    "# print the results\n",
    "print( \"[tuple] - For circle of radius \" + str( radius ) + \": - circumference = \" + str( circumference ) + \" - area = \" + str( area ) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we'll make our own function that returns multiple values.  Complete the `printDescriptiveStats()` function below so that it uses the data frame of data from the `homework.MachineLearning2` table and uses it to create and return:\n",
    "\n",
    "- 1) - **`summary_OUT`** - the summary statistics for all variables in the dataset (the results of invoking `describe()`)\n",
    "- 2) - **`top_depts_OUT`** - a Series that contains the top 10 most referenced department names from the column \"`org_dept`\" and the count of mentions for each (use a combination of the `head()`, and `value_counts()` functions)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "nbgrader": {
     "grade": true,
     "grade_id": "ML_Ex1",
     "locked": false,
     "points": 1,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "def printDescriptiveStats( data_frame_IN ):\n",
    "\n",
    "    \"\"\"\n",
    "    Parameters\n",
    "    ----------\n",
    "    data_frame_IN : A pandas DataFrame\n",
    "    \n",
    "    Returns\n",
    "    -------\n",
    "    summary_OUT : A Pandas dataframe containing count, mean, standard deviation, \n",
    "              minimum, maximum, and the 25th, 50th and 75th percentile\n",
    "    top_depts_OUT : A pandas.core.series.Series containing the top 10 departments\n",
    "               and their frequencies\n",
    "    \"\"\"\n",
    "    ### BEGIN SOLUTION\n",
    "    \n",
    "    # use describe to summarize table\n",
    "    summary_OUT = data_frame_IN.describe()\n",
    "    \n",
    "    # get column \"org_dept\"\n",
    "    department_series = data_frame_IN[ \"org_dept\" ]\n",
    "\n",
    "    # calculate frequencies\n",
    "    department_frequencies = department_series.value_counts()\n",
    "    \n",
    "    # get top ten most frequent\n",
    "    top_depts_OUT = department_frequencies.head( 10 )\n",
    "    \n",
    "    return summary_OUT, top_depts_OUT \n",
    "\n",
    "    ### END SOLUTION\n",
    "    \n",
    "#-- END function printDescriptiveStats() --#\n",
    "\n",
    "print( \"function printDescriptiveStats() defined at \" + str( datetime.datetime.now() ) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the cell above that contains your function definition, then run the cell below to test it out and see what the data in data_frame looks like."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "nbgrader": {
     "grade": true,
     "grade_id": "ML_Ex1-autograde",
     "locked": true,
     "points": 1,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "# TEST - lets see what our data looks like\n",
    "output = printDescriptiveStats( data_frame )\n",
    "\n",
    "# TEST to make sure two things returned\n",
    "assert len( output ) == 2\n",
    "\n",
    "# get descriptives and top_departments\n",
    "descriptives = output[ 0 ]\n",
    "top_departments = output[ 1 ]\n",
    "\n",
    "print( \"Descriptives:\" )\n",
    "print( descriptives )\n",
    "print( \"Top Departments:\" )\n",
    "print( top_departments )\n",
    "\n",
    "# TEST to make sure frequencies are right.\n",
    "assert top_departments[ 0 ] == 16192"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Cleaning and Subsetting Data\n",
    "\n",
    "- Back to [Table of Contents](#Table-of-Contents)\n",
    "\n",
    "Looking at the data, it is clear that there are missing values in `org_dept` (look at how many rows have \"NONE\", and where that values ranks in the frequencies for `org_dept`).\n",
    "\n",
    "In order to train a model to predict this value, we'll need to separate those that have a value from those that do now, so that we can train and evaluate our model using a samples of just those grant rows that contain an `org_dept` value.\n",
    "\n",
    "To filter out rows with no `org_dept` value, we'll need to do some basic cleaning of the data.  For that, we first need to figure out which variables have missing values.\n",
    "\n",
    "The function `calc_null_frequencies()` accepts a DataFrame in which you'd like to count null values per column.  It returns a DataFrame that lists each column in the DataFrame and for each, counts of non-null (\"False\") and null (\"True\") values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def calc_null_frequencies( data_frame_IN ):\n",
    "\n",
    "    \"\"\"\n",
    "    for a given DataFrame, calculates how many values for \n",
    "    each variable are null and returns the resulting table.\n",
    "    \"\"\"\n",
    "    \n",
    "    # return reference\n",
    "    null_frequencies_OUT = None\n",
    "    \n",
    "    # declare variables\n",
    "    data_frame_long = None\n",
    "    variable_value_series = None\n",
    "    is_null_value_series = None\n",
    "    variable_name_series = None\n",
    "    \n",
    "    # Use DataFrame.melt() to convert DataFrame table into long format\n",
    "    #   where every column value is its own row: variable names are in\n",
    "    #   column \"variable\", values are in column \"value\".\n",
    "    data_frame_long = pandas.melt( data_frame_IN )\n",
    "    \n",
    "    # get Series that just contains column values\n",
    "    #value_series = data_frame_long.value # alternate dot (\".\") notation\n",
    "    variable_value_series = data_frame_long[ \"value\" ]\n",
    "    \n",
    "    # Make a Series where position of each column value in Series\n",
    "    #    contains True if value was NULL or False if not.\n",
    "    is_null_value_series = variable_value_series.isnull()\n",
    "\n",
    "    # get Series of column names to match each of the values\n",
    "    #   (not distinct - one for each value - lots!)\n",
    "    # variable_name_series = data_frame_long.variable # alternate dot (\".\") notation\n",
    "    variable_name_series = data_frame_long[ \"variable\" ]\n",
    "    \n",
    "    # ceate a dataframe that sums counts of non-null values (\"False\") to\n",
    "    #    null values (\"True\") for each column name.\n",
    "    null_frequencies_OUT = pandas.crosstab( variable_name_series, is_null_value_series )\n",
    "\n",
    "    return null_frequencies_OUT\n",
    "\n",
    "#-- END function calc_null_frequencies() --#\n",
    "\n",
    "print( \"function calc_null_frequencies() defined at \" + str( datetime.datetime.now() ) )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Lets see what our NULL values look like\n",
    "calc_null_frequencies( data_frame )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, that we have a better understanding of the issues our dataset has (in particular, `org_dept` has many NULL values and many rows where value is \"NONE\"), lets go ahead and write some code to deal with them."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 2 - function `cleanData`\n",
    "\n",
    "- Return to [Table of Contents](#Table-of-Contents)\n",
    "\n",
    "To facilitate our cleanup, we will create a function that uses pandas' subsetting functionality to filter out rows where a given column contains a given value.\n",
    "\n",
    "In pandas, if you want to filter a DataFrame, you use a special form of square bracket notation to specify a boolean test to be used on each row to decide if it should be included in the resulting DataFrame.  If the test evaluates to True, the row is included.  If the test evaluates to False, the row is not included.\n",
    "\n",
    "The syntax for this notation is to place this boolean test inside square brackets next to the name of the variable that contains the DataFrame you want to filter:\n",
    "\n",
    "    filtered_data_frame = data_frame[ <boolean_test> ]\n",
    "\n",
    "Inside this `<boolean_test>`, you can reference the value for a given column using the standard square bracket notation (`data_frame[ \"<column_name>\" ]`), you can call functions or methods on objects, and you can use logical operators to `and` and `or` multiple tests together.  Examples make it clearer:\n",
    "\n",
    "    # filter out rows where column \"cfda_code\" is not 123 (so keep rows where \"cfda_code\" is 123)\n",
    "    filtered_data_frame = data_frame[ data_frame[ \"cfda_code\" ] == 123 ]\n",
    "    \n",
    "    # filter out rows where column \"year\" is NULL (so keep rows where pandas.notnull( data_frame[ \"year\" ] == True )\n",
    "    filtered_data_frame = data_frame[ pandas.notnull( data_frame[ \"year\" ] ) == True ]\n",
    "    \n",
    "    # filter out rows where column \"org_name\" = \"ARBYS\" (so keep rows where \"org_name\" is not \"ARBYS\")\n",
    "    filtered_data_frame = data_frame[ data_frame[ \"org_name\" ] != \"ARBYS\" ]\n",
    "    \n",
    "    # filter out rows where column \"org_name\" = \"ARBYS\" and \"year\" is 1726 (Arby's didn't exist in 1726...)\n",
    "    filtered_data_frame = data_frame[ ( data_frame[ \"org_name\" ] != \"ARBYS\" ) and ( data_frame[ \"year\" ] != 1726 )]\n",
    "\n",
    "In the cleanData function below, using the DataFrame (**`data_frame_IN`**), name of a column in the dataframe (**`column_name_IN`**), and filter value passed in (**`filter_value_IN`**):\n",
    "\n",
    "- return a cleaned dataframe that is a subset of **`data_frame_IN`** from which you've removed rows that contain the specified value (**`filter_value_IN`**) in the specified column (**`column_name_IN`**).  In order to return this subset, store it in the return variable **`cleaned_data_OUT`**.\n",
    "- Your function should be equipped to deal with NULL - a special value that can't be filtered normally.  The simplest way to do this is to decide that a certain string value will tell you when you are filtering on NULL (any case of the string \"NULL\" - \"NULL\"/\"null\"/\"NuLl\", etc.).  Whenever you find this value has been passed in `filter_value_IN`, use the `pandas.notnull()` function to test whether you should keep a given row, rather than a boolean operator."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "nbgrader": {
     "grade": true,
     "grade_id": "ML_Ex2",
     "locked": false,
     "points": 1,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "def cleanData( data_frame_IN, column_name_IN, filter_value_IN ):\n",
    "\n",
    "    \"\"\"\n",
    "    Parameters\n",
    "    ----------\n",
    "    - data_frame_IN : A pandas DataFrame\n",
    "    - column_name_IN : Name of the column on the dataframe\n",
    "    - filter_value_IN : The value that causes rows to be filtered out of data_frame_IN \n",
    "           if it is present in the column named column_name_IN.\n",
    "    \n",
    "    Returns\n",
    "    -------\n",
    "    - cleaned_data_OUT : A Pandas DataFrame containing only rows that did not have the \n",
    "                specified value in the specified column\n",
    "    \"\"\"\n",
    "    \n",
    "    # return reference\n",
    "    cleaned_data_OUT = None\n",
    "\n",
    "    # check to make sure that column name is in data frame's list of column names.\n",
    "    if( column_name_IN not in list( data_frame_IN.columns.values ) ):\n",
    "    \n",
    "        print(\"ERROR : Column you specified not present in the dataframe\")\n",
    "        clean_data_OUT = None\n",
    "        \n",
    "    #-- END check to see if column is in data frame's list of column names. --#\n",
    "    \n",
    "    ### BEGIN SOLUTION\n",
    "\n",
    "    if filter_value_IN.upper() == \"NULL\":\n",
    "    \n",
    "        # keep rows where column passed is not NULL.\n",
    "        cleaned_data_OUT = data_frame_IN[ pandas.notnull( data_frame_IN[ column_name_IN ] ) == True ]\n",
    "\n",
    "    else:\n",
    "        \n",
    "        # keep rows where column passed in does not contain filter_out_value_IN.\n",
    "        cleaned_data_OUT = data_frame_IN[ data_frame_IN[ column_name_IN ] != filter_value_IN ]\n",
    "\n",
    "    ### END SOLUTION\n",
    "    \n",
    "    return cleaned_data_OUT\n",
    "    \n",
    "#-- END function cleanData() --#\n",
    "\n",
    "print( \"function cleanData() defined at \" + str( datetime.datetime.now() ) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Clearly, if the department name is NULL, we cannot use that data to train our classifier.  Fortunately, we can use our `cleanData()` function to remove all rows where \"`org_dept`\" is NULL."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "nbgrader": {
     "grade": true,
     "grade_id": "ML_Ex2-test",
     "locked": true,
     "points": 1,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "# TEST - remove all rows where \"`org_dept`\" is NULL.\n",
    "cleaned_data_frame = cleanData( data_frame, \"org_dept\", \"null\" )\n",
    "\n",
    "# generate null frequencies data frame again to see if it worked.\n",
    "null_frequencies_df = calc_null_frequencies( cleaned_data_frame )\n",
    "\n",
    "# TEST - should be 402091 rows left after clearing out NULL departments.\n",
    "assert len( cleaned_data_frame ) == 99998\n",
    "\n",
    "# output NULL frequencies table\n",
    "null_frequencies_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That did clean most of the NULL values for us, even across all the columns.  There might be other values in \"`org_dept`\" we want to clear out as well, though (\"NONE\"), so lets run the code cell below to print a frequency table for \"`org_dept`\" and see what values remain:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "cleaned_data_frame[\"org_dept\"].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The values of \"`NONE`\", \"`MISCELLANEOUS`\" and \"`NO CODE ASSIGNED`\" are also useless in terms of training or testing a machine learning model for predicting a grant's department.  Run the code cell below to use our `cleanData()` method to remove all rows where the \"`org_dept`\" column contains \"`NONE`\", \"`MISCELLANEOUS`\" or \"`NO CODE ASSIGNED`\":"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Getting rid of NONE and NO CODE ASSIGNED.\n",
    "cleaned_data_frame = cleanData( cleaned_data_frame, \"org_dept\", \"NONE\" )\n",
    "cleaned_data_frame = cleanData( cleaned_data_frame, \"org_dept\", \"NO CODE ASSIGNED\" )\n",
    "cleaned_data_frame = cleanData( cleaned_data_frame, \"org_dept\", \"MISCELLANEOUS\" )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to avoid confusion caused by mixed case, we should also make sure that the values of all categorical variables (variables of data type \"`numpy.object_`\") are converted to a uniform case.  Run the example code in the cell below to convert all categorical columns to upper case:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Converting to upper case\n",
    "\n",
    "# loop over column names\n",
    "column_name_series = data_frame.columns.values\n",
    "for column_name in list( column_name_series ):\n",
    "    \n",
    "    # get column data type\n",
    "    column_data_type = cleaned_data_frame[ column_name ].dtype\n",
    "    \n",
    "    # is it categorical (numpy.object_)?\n",
    "    if column_data_type == numpy.object_:\n",
    "        \n",
    "        # yes - use Series.str.upper() method to convert all values in column to upper case.\n",
    "        cleaned_data_frame[ column_name ] = cleaned_data_frame[ column_name ].str.upper()\n",
    "        \n",
    "    #-- END check to see if variable is categorical --#\n",
    "    \n",
    "#-- END loop over columns. --#\n",
    "\n",
    "# and calculate the null frequencies again to see where we're at\n",
    "calc_null_frequencies( cleaned_data_frame )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this example, \"`year`\", \"`study_section`\" and \"`cfda_code`\" are categorical variables that we aren't interested in predicting, so run the cell below to remove rows where these columns are NULL:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "cleaned_data_frame = cleanData( cleaned_data_frame, \"cfda_code\", \"NULL\" )\n",
    "cleaned_data_frame = cleanData( cleaned_data_frame, \"study_section\", \"NULL\" )\n",
    "cleaned_data_frame = cleanData( cleaned_data_frame, \"year\", \"NULL\" )\n",
    "\n",
    "# Lets see if we have any more null frequencies to deal with\n",
    "calc_null_frequencies( cleaned_data_frame )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We still have 203 missing values for \"`total_cost`\", which we'll want to address one way or another so we can use this column in our models.  We can get rid of them, but we could also use some sort of function to estimate a value (basic ones are centrality measures like mean, median or mode based on the other values in that column).  This can be complicated to do well.  Since there are a total of 87,068 records, losing 203 shouldn't impact the overall size of our data set, so for this exercise, we'll just delete these 203 records and move on:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# get rid of rows where \"TOTAL_COST\" is NULL.\n",
    "cleaned_data_frame = cleanData( cleaned_data_frame, \"total_cost\", \"NULL\" )\n",
    "\n",
    "# look at null frequencies now.\n",
    "calc_null_frequencies( cleaned_data_frame )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets take another look at our \"`org_dept`\" variable, and see if we can combine some departments that are very similar."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Look at the department values that remain, and how many total records we have:\n",
    "print( cleaned_data_frame[ \"org_dept\" ].value_counts() )\n",
    "print( \"Record Count: \" + str( len( cleaned_data_frame ) ) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It certainly looks like a lot of these departments can be combined into broader areas.  To reduce the number of categories to predict (and so make it easier for a model to make good predictions), run the cell below to combine some of these departments into broader areas of interest.\n",
    "\n",
    "The lines of code below that update \"`org_dept`\" use a similar syntax to those we used to filter out rows that contained a specified value in a given column.  The `.loc` reference is one of a number of special pandas ways to filter or index DataFrames and Series that has benefits over the base method indicated by using plain square brackets (these benefits are related to optimizations in pandas that are difficult to summarize - specifics on `.loc[]` and a discussion of optimizations: [http://pandas.pydata.org/pandas-docs/stable/indexing.html](http://pandas.pydata.org/pandas-docs/stable/indexing.html) ).\n",
    "\n",
    "Inside the brackets next to \"`.loc[]`\", there is a statement with two sections used to specify what row(s) - before the comma - and column(s) - after the comma - we are trying to reference.  Before the comma is an expression that indicates which rows to process that pandas calls a \"row_indexer\" - in this case, a boolean expression run on each row, where all rows that evaluate to `True` are included in the operation (looking for rows in which \"`org_dept`\" has a certain value).  After the comma is an expression that defines which columns to work with on the selected rows that pandas calls the \"column_indexer\" - in this case, it specifies a single column to update for rows included based on the row_indexer.\n",
    "\n",
    "So, in this example:\n",
    "\n",
    "    cleaned_data_frame.loc[ cleaned_data_frame[\"org_dept\"] == \"ANESTHESIOLOGY\", [ 'org_dept' ] ] = \"MEDICINE\"\n",
    "    \n",
    "- \"**`cleaned_data_frame[\"org_dept\"] == \"ANESTHESIOLOGY\"`**\" is the \"row_indexer\" test to select only rows where the value in \"`org_dept`\" is \"ANESTHESIOLOGY\".\n",
    "- \"**`, [ 'org_dept' ]`**\" is the \"column_indexer\" - we are working with column \"`org_dept`\", but only where it currently has a value of \"ANESTHESIOLOGY\".\n",
    "- then, after the reference to `.loc[]`, \"**` = \"MEDICINE\"`**\" tells the dataframe to set the value \"MEDICINE\" in the rows and columns selected by `.loc[]`.\n",
    "\n",
    "So, yeah.  That is what is going on.  Again, for more information, see [http://pandas.pydata.org/pandas-docs/stable/indexing.html](http://pandas.pydata.org/pandas-docs/stable/indexing.html), but feel free at this point to just run the cell below and move on.  =)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# MEDICINE\n",
    "cleaned_data_frame.loc[ cleaned_data_frame[\"org_dept\"] == \"ANESTHESIOLOGY\", [ 'org_dept' ] ] = \"MEDICINE\"\n",
    "cleaned_data_frame.loc[ cleaned_data_frame[\"org_dept\"] == \"DENTISTRY\", [ 'org_dept' ] ] = \"MEDICINE\"\n",
    "cleaned_data_frame.loc[ cleaned_data_frame[\"org_dept\"] == \"DERMATOLOGY\", [ 'org_dept' ] ] = \"MEDICINE\"\n",
    "cleaned_data_frame.loc[ cleaned_data_frame[\"org_dept\"] == \"EMERGENCY MEDICINE\", [ 'org_dept' ] ] = \"MEDICINE\"\n",
    "cleaned_data_frame.loc[ cleaned_data_frame[\"org_dept\"] == \"FAMILY MEDICINE\", [ 'org_dept' ] ] = \"MEDICINE\"\n",
    "cleaned_data_frame.loc[ cleaned_data_frame[\"org_dept\"] == \"INTERNAL MEDICINE/MEDICINE\", [ 'org_dept' ] ] = \"MEDICINE\"\n",
    "cleaned_data_frame.loc[ cleaned_data_frame[\"org_dept\"] == \"NEUROLOGY\", [ 'org_dept' ] ] = \"MEDICINE\"\n",
    "cleaned_data_frame.loc[ cleaned_data_frame[\"org_dept\"] == \"NEUROSURGERY\", [ 'org_dept' ] ] = \"MEDICINE\"\n",
    "cleaned_data_frame.loc[ cleaned_data_frame[\"org_dept\"] == \"OBSTETRICS &GYNECOLOGY\", [ 'org_dept' ] ] = \"MEDICINE\"\n",
    "cleaned_data_frame.loc[ cleaned_data_frame[\"org_dept\"] == \"OPHTHALMOLOGY\", [ 'org_dept' ] ] = \"MEDICINE\"\n",
    "cleaned_data_frame.loc[ cleaned_data_frame[\"org_dept\"] == \"ORTHOPEDICS\", [ 'org_dept' ] ] = \"MEDICINE\"\n",
    "cleaned_data_frame.loc[ cleaned_data_frame[\"org_dept\"] == \"OTOLARYNGOLOGY\", [ 'org_dept' ] ] = \"MEDICINE\"\n",
    "cleaned_data_frame.loc[ cleaned_data_frame[\"org_dept\"] == \"PATHOLOGY\",[ 'org_dept' ] ] = \"MEDICINE\"\n",
    "cleaned_data_frame.loc[ cleaned_data_frame[\"org_dept\"] == \"PHARMACOLOGY\", [ 'org_dept' ] ] = \"MEDICINE\"\n",
    "cleaned_data_frame.loc[ cleaned_data_frame[\"org_dept\"] == \"PHYSICAL MEDICINE &REHAB\", [ 'org_dept' ] ] = \"MEDICINE\"\n",
    "cleaned_data_frame.loc[ cleaned_data_frame[\"org_dept\"] == \"PEDIATRICS\", [ 'org_dept' ] ] = \"MEDICINE\"\n",
    "cleaned_data_frame.loc[ cleaned_data_frame[\"org_dept\"] == \"PLASTIC SURGERY\", [ 'org_dept' ] ] = \"MEDICINE\"\n",
    "cleaned_data_frame.loc[ cleaned_data_frame[\"org_dept\"] == \"RADIATION-DIAGNOSTIC/ONCOLOGY\", [ 'org_dept' ] ] = \"MEDICINE\"\n",
    "cleaned_data_frame.loc[ cleaned_data_frame[\"org_dept\"] == \"SURGERY\", [ 'org_dept' ] ] = \"MEDICINE\"\n",
    "cleaned_data_frame.loc[ cleaned_data_frame[\"org_dept\"] == \"UROLOGY\", [ 'org_dept' ] ] = \"MEDICINE\"\n",
    "\n",
    "# BIOLOGY\n",
    "cleaned_data_frame.loc[ cleaned_data_frame[\"org_dept\"] == \"ANATOMY/CELL BIOLOGY\", [ 'org_dept' ] ] = \"BIOLOGY\"\n",
    "cleaned_data_frame.loc[ cleaned_data_frame[\"org_dept\"] == \"PHYSIOLOGY\", [ 'org_dept' ] ] = \"BIOLOGY\"\n",
    "cleaned_data_frame.loc[ cleaned_data_frame[\"org_dept\"] == \"ZOOLOGY\", [ 'org_dept' ] ] = \"BIOLOGY\"\n",
    "\n",
    "# ENGINEERING\n",
    "cleaned_data_frame.loc[ cleaned_data_frame[\"org_dept\"] == \"BIOMEDICAL ENGINEERING\", [ 'org_dept' ] ] = \"ENGINEERING (ALL TYPES)\"\n",
    "\n",
    "# OTHER HEALTH PROFESSIONS\n",
    "cleaned_data_frame.loc[ cleaned_data_frame[\"org_dept\"] == \"PUBLIC HEALTH &PREV MEDICINE\", [ 'org_dept' ] ] = \"OTHER HEALTH PROFESSIONS\"\n",
    "\n",
    "# OTHER CLINICAL SCIENCES\n",
    "cleaned_data_frame.loc[ cleaned_data_frame[\"org_dept\"] == \"NUTRITION\", [ 'org_dept' ] ] = \"OTHER CLINICAL SCIENCES\"\n",
    "\n",
    "# We will also get rid of \"ADMINISTRATION\n",
    "cleaned_data_frame = cleanData( cleaned_data_frame, \"org_dept\", \"ADMINISTRATION\" )\n",
    "\n",
    "# check out distribution of categories now:\n",
    "print( cleaned_data_frame[ \"org_dept\" ].value_counts() )\n",
    "print( \"Record Count: \" + str( len( cleaned_data_frame ) ) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Checkpoint - save `cleaned_data_frame`\n",
    "\n",
    "- Back to [Table of Contents](#Table-of-Contents)\n",
    "\n",
    "At this point, our DataFrame referenced by `cleaned_data_frame` is as clean as we are going to make it for this assignment.  If you want to be able to pick up here in the future without having to run through all those steps again, you can write the DataFrame referenced by `cleaned_data_frame` to disk now, so if you need to start over, you can just pick up here without having to re-clean the data.  To write the dataframe to the file system:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# This might be a good time to persist your dataframe to the file system,\n",
    "#    so you don't need to clean it again if you leave off.\n",
    "\n",
    "# set name of file you want to contain your data frame.\n",
    "dataframe_file_name = \"asgt7_cleaned.msg\"\n",
    "\n",
    "# write your dataframe out to the msgpack (binary JSON) format.\n",
    "cleaned_data_frame.to_msgpack( dataframe_file_name )\n",
    "\n",
    "# to load subsequently:\n",
    "#cleaned_data_frame = pandas.read_msgpack( dataframe_file_name )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Model Selection and Assessment\n",
    "\n",
    "- Back to [Table of Contents](#Table-of-Contents)\n",
    "\n",
    "Now that we have a clean dataset, we can move on to the fun parts!! The python machine learning libraries do not accept categorical variables, so we need to convert all such variables to dummies first (boolean variables that capture the presence or absence in a given row of each of the potential values for each categorical variable). However, pandas makes it super easy! \n",
    "\n",
    "But before we do that, lets split our data variables into predictors (features, or dependent variables, or \"X\" variables) and variables to predict (independent variables, or \"Y\" variables).  For ease of reference, in subsequent examples, names of variables that pertain to predictors will start with \"`X_`\", and names of variables that pertain to variables we are to predict will start with \"`y_`\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Lets go ahead and split into predictors and predicted\n",
    "\n",
    "# make a list of the column names not in dependent column name list (currently just \"org_dept\")\n",
    "# one line - predictor_column_list = [ column_name for column_name in list( cleaned_data_frame.columns.values ) if column_name not in [ \"org_dept\" ] ]\n",
    "X_column_list = []\n",
    "y_column_list = [ \"org_dept\" ]\n",
    "\n",
    "# loop over column names.\n",
    "column_name_list = cleaned_data_frame.columns.values\n",
    "for column_name in column_name_list:\n",
    "    \n",
    "    # if the name is not predicted_column_list, add it to predictor_column_list\n",
    "    if ( column_name not in y_column_list ):\n",
    "        \n",
    "        # add to the predictor_column_list\n",
    "        X_column_list.append( column_name )\n",
    "        \n",
    "    #-- END check to see if column is in predicted/IV/Y list --#\n",
    "    \n",
    "#-- END loop over columns. --#\n",
    "\n",
    "# split columns into two DataFrames, those we are to predict,\n",
    "#    and those that are predictors.\n",
    "X_data_frame = cleaned_data_frame[ X_column_list ]\n",
    "y_data_frame = cleaned_data_frame[ y_column_list ]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we can easily convert all categorical variables in `X_data_frame` into dummy/binary variables using the `pandas.get_dummies()` function ( [http://pandas.pydata.org/pandas-docs/stable/generated/pandas.get_dummies.html](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.get_dummies.html) )."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Python's sckikit algorithms dont work on categorical variables. Fortunately, Pandas provides an easy way out!\n",
    "X_data_frame = pandas.get_dummies( X_data_frame )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cleanup - clear `cleaned_data_frame`\n",
    "\n",
    "- Back to [Table of Contents](#Table-of-Contents)\n",
    "\n",
    "At this point, we have split `cleaned_data_frame` out into new DataFrames that hold just the X and y variables, and so we now have two separate and identical copies of our data loaded into memory - the original `cleaned_data_frame`, and the data in that DataFrame split into X and y variables.  Since there are two copies and we won't subsequently use the full data frame, we can clear the original `cleaned_data_frame` from memory.\n",
    "\n",
    "To do this, we'll empty the variable that references `cleaned_data_frame`, telling Python that the DataFrame instance is no longer needed, then we'll tell Python to garbage collect memory, removing the DataFrame from memory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# set the variable that references the DataFrame to None.\n",
    "cleaned_data_frame = None\n",
    "\n",
    "# Tell Python to free up memory.\n",
    "gc.collect()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training a model\n",
    "\n",
    "- Back to [Table of Contents](#Table-of-Contents)\n",
    "\n",
    "If we're building a model, we're going to need a way to know whether or not it's working. Convincing others of the quality of results is often the most challenging part of an analysis.  In machine learning, making repeatable, well-documented work with clear success metrics makes all the difference.\n",
    "\n",
    "For our classifier, we're going to use the following build methodology (Ghani, 2014):\n",
    "\n",
    "<img src=\"https://s3.amazonaws.com/demo-datasets/traintest.png\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In brief, this methodology involves:\n",
    "\n",
    "- First [**splitting your data**](#1.-Split-data-into-training-and-testing-data-sets) into a training set (75% of your data) and a test set (25% of your data).\n",
    "- \"**Feature engineering** is the process of transforming raw data into features that better represent the underlying problem/data to the predictive models, resulting in improved model accuracy on unseen data.\" ( from [Discover Feature Engineering](http://machinelearningmastery.com/discover-feature-engineering-how-to-engineer-features-and-how-to-get-good-at-it/) ).  In text, for example, this might involve deriving traits of the text like word counts, verb counts, or topics to feed into a model rather than simply giving it the raw text.\n",
    "- In the [**Model Build**](#2.-Model-Build---Fit-model) phase, you decide on a model then train or fit your model using your training data.\n",
    "- In the [**Evaluate Performance**](#3.-Evaluate-Performance---accuracy) phase, you run your fitted model on your set of testing predictors, then assess the quality of the model by comparing the predicted values to the actual values for each record in your testing data set. \n",
    "\n",
    "Since we have a limited number of relatively basic features, we won't be going into any Feature Engineering examples for this exercise.  However, feature engineering is an essential part of implementing quality machine learning - to learn more, start with the \"Discover Feature Engineering\" tutorial: [http://machinelearningmastery.com/discover-feature-engineering-how-to-engineer-features-and-how-to-get-good-at-it/](http://machinelearningmastery.com/discover-feature-engineering-how-to-engineer-features-and-how-to-get-good-at-it/)\n",
    "\n",
    "Let us now split our dataset into test and training using the `train_test_split()` function from scikit learn's sklearn.cross_validation module ( [http://scikit-learn.org/stable/modules/cross_validation.html](http://scikit-learn.org/stable/modules/cross_validation.html) ):"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Split data into training and testing data sets\n",
    "\n",
    "- Back to [Table of Contents](#Table-of-Contents)\n",
    "\n",
    "Let us now split our dataset into test and training using the `train_test_split()` function from scikit learn's sklearn.cross_validation module ( [http://scikit-learn.org/stable/modules/cross_validation.html](http://scikit-learn.org/stable/modules/cross_validation.html) ):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# use train_test_split() to split our X and Y variables into separate 75% and 25%\n",
    "#    DataFrames of training (X_train and y_train) and testing (X_test and y_test) data.\n",
    "X_train, X_test, y_train, y_test = train_test_split( X_data_frame, y_data_frame, test_size = 0.25, random_state = 0 )\n",
    "\n",
    "# Before we fit the model, we also need to change the datatype of the y_train variable.\n",
    "# y_train currently is a Pandas Series, however, scikit-learn requires it to be a numpy array\n",
    "# So all we need to do is extract the raw values of y_train, and pass them onto scikit-learn\n",
    "y_train_values = y_train[ 'org_dept' ].values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cleanup - clear X and y data frames\n",
    "\n",
    "- Back to [Table of Contents](#Table-of-Contents)\n",
    "\n",
    "At this point, we have split `X_data_frame` and `y_data_frame` out into new DataFrames that hold training and test data for X and y variables, and so we once again have two separate and identical copies of our data loaded into memory - the original `X_data_frame` and `y_data_frame`, and the data from those DataFrames split into training and test data.  Since there are two copies and we won't subsequently use `X_data_frame` and `y_data_frame`, we can clear them from memory.\n",
    "\n",
    "To do this, we'll empty the variables that reference `X_data_frame` and `y_data_frame`, telling Python that the DataFrame instances are no longer needed, then we'll tell Python to garbage collect memory, removing the DataFrames from memory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Free up memory\n",
    "X_data_frame = None\n",
    "y_data_frame = None\n",
    "\n",
    "# Tell Python to free up memory.\n",
    "gc.collect()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Model Build - Fit model\n",
    "\n",
    "- Back to [Table of Contents](#Table-of-Contents)\n",
    "\n",
    "Python's `scikit-learn` is a very well known machine library. It is also well documented and maintained. You can learn all about it here: [http://scikit-learn.org/stable/](http://scikit-learn.org/stable/). We will be using different classifiers from this library for our predictions in this workbook. \n",
    "\n",
    "We will start with the simplest `LogisticRegression` model ( [http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html](http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) ) and see how well that does.\n",
    "\n",
    "You can use any number of metrics to judge your models, but we will be using scikit learn's provided `accuracy_score()` (ratio of correct predictions to total number of predictions, based on comparing a set of predicted to values to a set of actual values) as our measure ( [http://scikit-learn.org/stable/modules/generated/sklearn.metrics.accuracy_score.html](http://scikit-learn.org/stable/modules/generated/sklearn.metrics.accuracy_score.html) )."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Let's fit the model\n",
    "model = LogisticRegression()\n",
    "model.fit( X_train, y_train_values )\n",
    "print(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When we print the model, we see different parameters we can adjust as we refine the model based on running it against test data (values such as `intercept_scaling`, `max_iters`, `penalty`, and `solver`).  Example output:\n",
    "\n",
    "    LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
    "          intercept_scaling=1, max_iter=100, multi_class='ovr',\n",
    "          penalty='l2', random_state=None, solver='liblinear', tol=0.0001,\n",
    "          verbose=0)\n",
    "\n",
    "To adjust these parameters, one would alter the call that creates the `LogisticRegression()` model instance, passing it one or more of these parameters with a value other than the default.  So, to re-fit the model with `max_iter` of 1000, `intercept_scaling` of 2, and `solver` of \"lbfgs\" (pulled from thin air as an example), you'd create your model as follows:\n",
    "\n",
    "    model = LogisticRegression( max_iter = 1000, intercept_scaling = 2, solver = \"lbfgs\" )\n",
    "\n",
    "More details on what each of thee parameters mean is on the `LogisticRegression` documentation page: [http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html](http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html)\n",
    "\n",
    "The basic way one would tune these parameters is to iterate over fitting your model to your training data with different parameters (hopefully chosen based on your knowledge of your data and the model you are fitting), then testing it against training data until the model's accuracy is as high as you can get it.  Unfortunately, this exposes one to the potential that the model is over-fitted to one's test data, and so won't perform as well when it is used to predict other sets of data.\n",
    "\n",
    "Cross-validation is a good way to fine-tune the parameters with less risk of over-fitting.  It involves dividing your training data into 5 or so equal sets called folds, then choosing a different fold to serve as the test data set each time you test a new set of parameters. This sounds complicated, but scikit learn has functions to help, and a good tutorial on cross-validation can be found on the scikit learn site: [http://scikit-learn.org/stable/modules/cross_validation.html](http://scikit-learn.org/stable/modules/cross_validation.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Evaluate Performance - accuracy\n",
    "\n",
    "- Back to [Table of Contents](#Table-of-Contents)\n",
    "\n",
    "Now let's use the model we just fit to make predictions on our test dataset, and see what our accuracy score is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# store our test \"to predict\" variables in \"expected\".\n",
    "expected = y_test\n",
    "\n",
    "# predict values from our \"predictors\" usin the model.\n",
    "predicted = model.predict(X_test)\n",
    "\n",
    "# generate an accuracy score by comparing expected to predicted.\n",
    "accuracy = accuracy_score(expected, predicted)\n",
    "print( \"Accuracy = \" + str( accuracy ) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We get an accuracy score of 0.453... (45.3%). This is not a great score, however, it is much better than random guessing, which would have had a chance of 1/18 of succeeding. The other way to guess would be to take the mode, which in this case is MEDICINE with a frequency of 39392, which would give us an accuracy score of 39392/86721 = 0.4542383 (45.4%). So logistic regression is about as good as just always assigning the mode when department is missing. Let's see if other classifiers can do any better."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercise 3 - Train, fit and evaluate your model\n",
    "\n",
    "- Back to [Table of Contents](#Table-of-Contents)\n",
    "\n",
    "Complete the function below to train different classifiers from the scikit library.\n",
    "\n",
    "The `classifier()` function that you will implement:\n",
    "\n",
    "- Accepts X_train_IN, y_train_IN (should be type numpy.ndarray, not a Series or DataFrame - to convert, call \"`.values`\"), X_test_IN, and y_test_IN variables.\n",
    "- creates a model, fits the model, tests the model, and calculates an accuracy score for the model (like we did above).\n",
    "- returns the accuracy score as a percent (so a number between 1 and 100, not a decimal between 0 and 1 - ... so multiply by 100).\n",
    "\n",
    "Your goal is to come up with a classifier that gives at least 70% accuracy on the test dataset.  To do this, you can:\n",
    "\n",
    "- choose different models from those offered as part of scikit learn.\n",
    "\n",
    "    - To start, here are some resources to help with choosing a model:\n",
    "\n",
    "        - The scikit learn tutorial on choosing a model - [http://scikit-learn.org/stable/tutorial/machine_learning_map/index.html](http://scikit-learn.org/stable/tutorial/machine_learning_map/index.html)\n",
    "        - This video on choosing and tuning a model - [http://blog.kaggle.com/2015/05/14/scikit-learn-video-5-choosing-a-machine-learning-model/](http://blog.kaggle.com/2015/05/14/scikit-learn-video-5-choosing-a-machine-learning-model/)\n",
    "    \n",
    "    - In particular, here are some sets of models you could explore from the \"predicting a category\" with \"labeled data\" branch of the [scikit learn tutorial on choosing a model](http://scikit-learn.org/stable/tutorial/machine_learning_map/index.html) (just make sure your data doesn't violate the assumptions of the model and that the model you choose is appropriate for your data - predicting a categorical variable using a mix of numeric and dummied categorical data):\n",
    "\n",
    "        - other Linear Models like the `LogisticRegression` - [http://scikit-learn.org/stable/modules/linear_model.htmlFeel](http://scikit-learn.org/stable/modules/linear_model.html)\n",
    "        - Decision Tree models - [http://scikit-learn.org/stable/modules/tree.html](http://scikit-learn.org/stable/modules/tree.html)\n",
    "        - Ensemble classifiers - [http://scikit-learn.org/stable/modules/ensemble.html](http://scikit-learn.org/stable/modules/ensemble.html)\n",
    "        - Nearest neighbors classifiers - [http://scikit-learn.org/stable/modules/neighbors.html#nearest-neighbors-classification](http://scikit-learn.org/stable/modules/neighbors.html#nearest-neighbors-classification)\n",
    "        - Stocahstic Gradient Descent - [http://scikit-learn.org/stable/modules/sgd.html#classification](http://scikit-learn.org/stable/modules/sgd.html#classification)\n",
    "        - Kernel Approximation + SGDClassifier - [http://scikit-learn.org/stable/modules/kernel_approximation.html](http://scikit-learn.org/stable/modules/kernel_approximation.html)\n",
    "\n",
    "- play around with different parameters for the models you try.\n",
    "- experiment with different sets of X variables.\n",
    "- _Advanced_ - You can try starting again from the top with an SQL query that uses JOINs to pull in columns from other tables, to add more variables to your pool of available predictors.\n",
    "- _Advanced_ - You could also try to derive additional features from the data present in your query and add those features to your predictors.\n",
    "\n",
    "Again, in general, make sure that the model and parameters you choose are appropriate for both your X and Y variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "nbgrader": {
     "grade": true,
     "grade_id": "ML_Ex3",
     "locked": false,
     "points": 1,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "def classifier(X_train_IN, y_train_IN, X_test_IN, y_test_IN):\n",
    "    \"\"\"\n",
    "    Parameters\n",
    "    ----------\n",
    "    X_train_IN : A pandas DataFrame of features used for training the classifier\n",
    "    y_train_IN : A numpy array of y values used for training the classifier\n",
    "    X_test_IN, y_test_IN : Use these to test the accuracy of your classifier\n",
    "    \n",
    "    Returns\n",
    "    -------\n",
    "    accuracy score : a float giving the percent (0 to 100) of accurate predictions you made\n",
    "    \"\"\"\n",
    "    \n",
    "    ### BEGIN SOLUTION\n",
    "    # declare variables\n",
    "    y_train_type = None\n",
    "    y_train_array = None\n",
    "    my_accuracy_score = -1\n",
    "   \n",
    "    # check to see if y_train_IN is either a Series or a DataFrame\n",
    "    y_train_type = type( y_train_IN )\n",
    "    if ( ( y_train_type == pandas.core.series.Series ) or ( y_train_type == pandas.core.frame.DataFrame ) ):\n",
    "        \n",
    "        # Series or DataFrame - convert to pandas array\n",
    "        y_train_array = y_train_IN.values\n",
    "        \n",
    "    elif ( y_train_type == numpy.ndarray ):\n",
    "        \n",
    "        # this is what it should be.\n",
    "        y_train_array = y_train_IN\n",
    "        \n",
    "    else:\n",
    "        \n",
    "        # not Series or DataFrame numpy ndarray - just use it and see what happens...?\n",
    "        print( \"Unexpected y_train_IN type: \" + str( y_train_type ) )\n",
    "        y_train_array = y_train_IN\n",
    "        \n",
    "    #-- END check to see if y_train_IN is wrong type. --#\n",
    "    \n",
    "    # for fitting model, use y_train_array rather than y_train_IN.\n",
    "\n",
    "    from sklearn.ensemble import RandomForestClassifier\n",
    "    model = RandomForestClassifier()\n",
    "    model.fit(X_train_IN, y_train_array)\n",
    "    expected = y_test_IN\n",
    "    predicted = model.predict(X_test_IN)\n",
    "    \n",
    "    # get accuracy score\n",
    "    my_accuracy_score = accuracy_score( expected, predicted )\n",
    "    \n",
    "    # * 100 to turn into percentage value\n",
    "    my_accuracy_score = my_accuracy_score * 100\n",
    "    \n",
    "    return my_accuracy_score\n",
    "    ### END SOLUTION\n",
    "\n",
    "#-- END function classifier() --#\n",
    "\n",
    "print( \"function classifier() declared at \" + str( datetime.datetime.now() ) )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "nbgrader": {
     "grade": true,
     "grade_id": "ML_Ex3-test",
     "locked": true,
     "points": 1,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "# Remember,  first extract the values of y_train, before calling the classifier function\n",
    "y_train_values = y_train[\"org_dept\"].values\n",
    "\n",
    "# TEST to see if your accuracy is greater than 70%. This might take several minutes to run!!!\n",
    "test_accuracy_score = classifier( X_train, y_train_values, X_test, y_test )\n",
    "print( \"Accuracy Percentage: \" + str( test_accuracy_score ) )\n",
    "\n",
    "# TEST - is it greater than 70%?\n",
    "assert test_accuracy_score >= 70"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## References\n",
    "\n",
    "- Back to [Table of Contents](#Table-of-Contents)\n",
    "\n",
    "Links to documentation:\n",
    "\n",
    "* [Scikit-Learn Documentation](#http://scikit-learn.org/stable/)\n",
    "* [NIH Reporter Documentation](#http://exporter.nih.gov/about.aspx)\n",
    "* Ghani, Rayid (2014), 2- Case-Study.ipynb"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "celltoolbar": "Create Assignment",
  "kernelspec": {
   "display_name": "python2 - conda",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}