P2-Supervised_Learning/P2_Stein/student_intervention_Stein.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Project 2: Supervised Learning\n",
    "### Building a Student Intervention System"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Classification vs Regression\n",
    "\n",
    "Your goal is to identify students who might need early intervention - which type of supervised machine learning problem is this, classification or regression? Why? \n",
    "    \n",
    "    This is a classification problem because it involves determining which of a finite number of categories(pass or fail) a student fits into based on a number of features(sex, address, age, school, etc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Exploring the Data\n",
    "\n",
    "Let's go ahead and read in the student dataset first.\n",
    "\n",
    "_To execute a code cell, click inside it and press **Shift+Enter**._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Import libraries\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "from sklearn import cross_validation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Student data read successfully!\n"
     ]
    }
   ],
   "source": [
    "# Read student data\n",
    "student_data = pd.read_csv(\"student-data.csv\")\n",
    "print \"Student data read successfully!\"\n",
    "# Note: The last column 'passed' is the target/label, all other are feature columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, can you find out the following facts about the dataset?\n",
    "- Total number of students\n",
    "- Number of students who passed\n",
    "- Number of students who failed\n",
    "- Graduation rate of the class (%)\n",
    "- Number of features\n",
    "\n",
    "_Use the code block below to compute these values. Instructions/steps are marked using **TODO**s._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "395\n",
      "265.0\n",
      "Total number of students: 395\n",
      "Number of students who passed: 265.0\n",
      "Number of students who failed: 130.0\n",
      "Number of features: 30\n",
      "Graduation rate of the class: 67.09%\n"
     ]
    }
   ],
   "source": [
    "# TODO: Compute desired values - replace each '?' with an appropriate expression/function call\n",
    "n_students = student_data.shape[0]\n",
    "n_features = student_data.shape[1]-1 #subtract 1 because last column is label, not feature\n",
    "n_passed = 0.0\n",
    "n_failed = 0.0\n",
    "for i in student_data.passed:\n",
    "    if i == \"yes\":\n",
    "        n_passed = n_passed + 1\n",
    "    else:\n",
    "        n_failed = n_failed + 1\n",
    "print n_students\n",
    "print n_passed\n",
    "grad_rate = (n_passed/n_students)*100\n",
    "print \"Total number of students: {}\".format(n_students)\n",
    "print \"Number of students who passed: {}\".format(n_passed)\n",
    "print \"Number of students who failed: {}\".format(n_failed)\n",
    "print \"Number of features: {}\".format(n_features)\n",
    "print \"Graduation rate of the class: {:.2f}%\".format(grad_rate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Preparing the Data\n",
    "In this section, we will prepare the data for modeling, training and testing.\n",
    "\n",
    "### Identify feature and target columns\n",
    "It is often the case that the data you obtain contains non-numeric features. This can be a problem, as most machine learning algorithms expect numeric data to perform computations with.\n",
    "\n",
    "Let's first separate our data into feature and target columns, and see if any features are non-numeric.<br/>\n",
    "**Note**: For this dataset, the last column (`'passed'`) is the target or label we are trying to predict."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Feature column(s):-\n",
      "['school', 'sex', 'age', 'address', 'famsize', 'Pstatus', 'Medu', 'Fedu', 'Mjob', 'Fjob', 'reason', 'guardian', 'traveltime', 'studytime', 'failures', 'schoolsup', 'famsup', 'paid', 'activities', 'nursery', 'higher', 'internet', 'romantic', 'famrel', 'freetime', 'goout', 'Dalc', 'Walc', 'health', 'absences']\n",
      "Target column: passed\n",
      "\n",
      "Feature values:-\n",
      "  school sex  age address famsize Pstatus  Medu  Fedu     Mjob      Fjob  \\\n",
      "0     GP   F   18       U     GT3       A     4     4  at_home   teacher   \n",
      "1     GP   F   17       U     GT3       T     1     1  at_home     other   \n",
      "2     GP   F   15       U     LE3       T     1     1  at_home     other   \n",
      "3     GP   F   15       U     GT3       T     4     2   health  services   \n",
      "4     GP   F   16       U     GT3       T     3     3    other     other   \n",
      "\n",
      "    ...    higher internet  romantic  famrel  freetime goout Dalc Walc health  \\\n",
      "0   ...       yes       no        no       4         3     4    1    1      3   \n",
      "1   ...       yes      yes        no       5         3     3    1    1      3   \n",
      "2   ...       yes      yes        no       4         3     2    2    3      3   \n",
      "3   ...       yes      yes       yes       3         2     2    1    1      5   \n",
      "4   ...       yes       no        no       4         3     2    1    2      5   \n",
      "\n",
      "  absences  \n",
      "0        6  \n",
      "1        4  \n",
      "2       10  \n",
      "3        2  \n",
      "4        4  \n",
      "\n",
      "[5 rows x 30 columns]\n"
     ]
    }
   ],
   "source": [
    "# Extract feature (X) and target (y) columns\n",
    "feature_cols = list(student_data.columns[:-1])  # all columns but last are features\n",
    "target_col = student_data.columns[-1]  # last column is the target/label\n",
    "print \"Feature column(s):-\\n{}\".format(feature_cols)\n",
    "print \"Target column: {}\".format(target_col)\n",
    "\n",
    "X_all = student_data[feature_cols]  # feature values for all students\n",
    "y_all = student_data[target_col]  # corresponding targets/labels\n",
    "print \"\\nFeature values:-\"\n",
    "print X_all.head()  # print the first 5 rows"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preprocess feature columns\n",
    "\n",
    "As you can see, there are several non-numeric columns that need to be converted! Many of them are simply `yes`/`no`, e.g. `internet`. These can be reasonably converted into `1`/`0` (binary) values.\n",
    "\n",
    "Other columns, like `Mjob` and `Fjob`, have more than two values, and are known as _categorical variables_. The recommended way to handle such a column is to create as many columns as possible values (e.g. `Fjob_teacher`, `Fjob_other`, `Fjob_services`, etc.), and assign a `1` to one of them and `0` to all others.\n",
    "\n",
    "These generated columns are sometimes called _dummy variables_, and we will use the [`pandas.get_dummies()`](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.get_dummies.html?highlight=get_dummies#pandas.get_dummies) function to perform this transformation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Processed feature columns (48):-\n",
      "['school_GP', 'school_MS', 'sex_F', 'sex_M', 'age', 'address_R', 'address_U', 'famsize_GT3', 'famsize_LE3', 'Pstatus_A', 'Pstatus_T', 'Medu', 'Fedu', 'Mjob_at_home', 'Mjob_health', 'Mjob_other', 'Mjob_services', 'Mjob_teacher', 'Fjob_at_home', 'Fjob_health', 'Fjob_other', 'Fjob_services', 'Fjob_teacher', 'reason_course', 'reason_home', 'reason_other', 'reason_reputation', 'guardian_father', 'guardian_mother', 'guardian_other', 'traveltime', 'studytime', 'failures', 'schoolsup', 'famsup', 'paid', 'activities', 'nursery', 'higher', 'internet', 'romantic', 'famrel', 'freetime', 'goout', 'Dalc', 'Walc', 'health', 'absences']\n"
     ]
    }
   ],
   "source": [
    "# Preprocess feature columns\n",
    "def preprocess_features(X):\n",
    "    outX = pd.DataFrame(index=X.index)  # output dataframe, initially empty\n",
    "\n",
    "    # Check each column\n",
    "    for col, col_data in X.iteritems():\n",
    "        # If data type is non-numeric, try to replace all yes/no values with 1/0\n",
    "        if col_data.dtype == object:\n",
    "            col_data = col_data.replace(['yes', 'no'], [1, 0])\n",
    "        # Note: This should change the data type for yes/no columns to int\n",
    "\n",
    "        # If still non-numeric, convert to one or more dummy variables\n",
    "        if col_data.dtype == object:\n",
    "            col_data = pd.get_dummies(col_data, prefix=col)  # e.g. 'school' => 'school_GP', 'school_MS'\n",
    "\n",
    "        outX = outX.join(col_data)  # collect column(s) in output dataframe\n",
    "\n",
    "    return outX\n",
    "\n",
    "X_all = preprocess_features(X_all)\n",
    "print \"Processed feature columns ({}):-\\n{}\".format(len(X_all.columns), list(X_all.columns))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Split data into training and test sets\n",
    "\n",
    "So far, we have converted all _categorical_ features into numeric values. In this next step, we split the data (both features and corresponding labels) into training and test sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training set: 300 samples\n",
      "Test set: 95 samples\n"
     ]
    }
   ],
   "source": [
    "# First, decide how many training vs test samples you want\n",
    "num_all = student_data.shape[0]  # same as len(student_data)\n",
    "num_train = 300  # about 75% of the data\n",
    "num_test = num_all - num_train\n",
    "\n",
    "# TODO: Then, select features (X) and corresponding labels (y) for the training and test sets\n",
    "# Note: Shuffle the data or randomly select samples to avoid any bias due to ordering in the dataset\n",
    "#Implement train-test split to utilize cross validation, test on subset of training data\n",
    "X_train, X_test, y_train, y_test = cross_validation.train_test_split(X_all, y_all, train_size= num_train, random_state=42)\n",
    "\n",
    "print \"Training set: {} samples\".format(X_train.shape[0])\n",
    "print \"Test set: {} samples\".format(X_test.shape[0])\n",
    "# Note: If you need a validation set, extract it from within training data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Training and Evaluating Models\n",
    "Choose 3 supervised learning models that are available in scikit-learn, and appropriate for this problem. For each model:\n",
    "\n",
    "- What are the general applications of this model? What are its strengths and weaknesses?\n",
    "- Given what you know about the data so far, why did you choose this model to apply?\n",
    "- Fit this model to the training data, try to predict labels (for both training and test sets), and measure the F<sub>1</sub> score. Repeat this process with different training set sizes (100, 200, 300), keeping test set constant.\n",
    "\n",
    "Produce a table showing training time, prediction time, F<sub>1</sub> score on training set and F<sub>1</sub> score on test set, for each training set size.\n",
    "\n",
    "Note: You need to produce 3 such tables - one for each model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##4.1 Decision Trees\n",
    "    Decision trees are used for both classification and regression problems, offering a simple and easy to understand method to model data and perform predictions. This simplicity is a primary strength, along with it ease to visualize the model. Also, it doen't require extensive data preparation although its cost grows logarithmically with the number of data points.  The decision tree is a white box model, which allows easier explation of the results obtained. The model can also be validated using statistical tests, which makes it easier to determine the models reliability. However, decision trees can still create over complex, over fitting models. Also, common algorithms used in its implementation are not guaranteed to find the optimum global solution. Decision trees are used in data mining and price prediction.(1) \n",
    "    Since decision trees are able to perform the necessary classification problem and are easy to implement and interpret it seems to be a good model to test further. It handles the data categories and the relatively low number of samples means the cost to use it should be reasonable. \n",
    "    \n",
    "    (1)1.10. Decision Trees — scikit-learn 0.17.1 documentation \n",
    "    http://scikit-learn.org/stable/modules/tree.html\n",
    "    \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##4.2 Naive Bayes\n",
    "    Naive Bayes' have the ability to utilize a small amount of training data and perform very quick compared to more complicated models. The assumed independence of features allows it to scale well with increasing dimensions and perform well as long as the correct class is the more probable one. However, the assumption that each feature contributes independently to the variable being predicted and Naive Bayes' poor performance at estimation are negatives that need to be taken into account.  Naive Bayes' speed and proven record of performace in topics of text classification and spam filtering  make it another model to explore. (2)\n",
    "    \n",
    "    (2)1.9. Naive Bayes — scikit-learn 0.17.1 documentation\n",
    "http://scikit-learn.org/stable/modules/naive_bayes.html\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##4.3 Support Vector Machines(SVM)\n",
    "    SVMs are capable of modeling high dimensional spaces, even if they out number the samples. The versatility of SVMs through the use of differing kernel functions also make them attracitve. The implementation of using support vectors to create the decision function reduces the cost of creating the model as well. These attributes lead to SVMs use in image and text classification. However, SVMs cannot provide direct probability estimates.\n",
    "    SVMs are capable of performing classification problems and the data preparation performed which created many more dimensions to account for categorical variables make SVMs a good candidate.(3)\n",
    "    \n",
    "    (3) 1.4. Support Vector Machines — scikit-learn 0.17.1 documentation\n",
    "http://scikit-learn.org/stable/modules/svm.html#scores-probabilities"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training DecisionTreeClassifier...\n",
      "Done!\n",
      "Training time (secs): 0.002\n"
     ]
    }
   ],
   "source": [
    "# Train a model --Decision Tree\n",
    "import time\n",
    "\n",
    "def train_classifier(clf, X_train, y_train):\n",
    "    print \"Training {}...\".format(clf.__class__.__name__)\n",
    "    start = time.time()\n",
    "    clf.fit(X_train, y_train)\n",
    "    end = time.time()\n",
    "    print \"Done!\\nTraining time (secs): {:.3f}\".format(end - start)\n",
    "\n",
    "# TODO: Choose a model, import it and instantiate an object\n",
    "from sklearn import tree\n",
    "clf = tree.DecisionTreeClassifier()\n",
    "\n",
    "# Fit model to training data\n",
    "train_classifier(clf, X_train, y_train)  # note: using entire training set here\n",
    "#print clf  # you can inspect the learned model by printing it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Predicting labels using DecisionTreeClassifier...\n",
      "Done!\n",
      "Prediction time (secs): 0.000\n",
      "F1 score for training set: 1.0\n"
     ]
    }
   ],
   "source": [
    "# Predict on training set and compute F1 score\n",
    "from sklearn.metrics import f1_score\n",
    "\n",
    "def predict_labels(clf, features, target):\n",
    "    print \"Predicting labels using {}...\".format(clf.__class__.__name__)\n",
    "    start = time.time()\n",
    "    y_pred = clf.predict(features)\n",
    "    end = time.time()\n",
    "    print \"Done!\\nPrediction time (secs): {:.3f}\".format(end - start)\n",
    "    return f1_score(target.values, y_pred, pos_label='yes')\n",
    "\n",
    "train_f1_score = predict_labels(clf, X_train, y_train)\n",
    "print \"F1 score for training set: {}\".format(train_f1_score)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Predicting labels using DecisionTreeClassifier...\n",
      "Done!\n",
      "Prediction time (secs): 0.001\n",
      "F1 score for test set: 0.69918699187\n"
     ]
    }
   ],
   "source": [
    "# Predict on test data\n",
    "print \"F1 score for test set: {}\".format(predict_labels(clf, X_test, y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "------------------------------------------\n",
      "Training set size: 300\n",
      "Training DecisionTreeClassifier...\n",
      "Done!\n",
      "Training time (secs): 0.003\n",
      "Predicting labels using DecisionTreeClassifier...\n",
      "Done!\n",
      "Prediction time (secs): 0.001\n",
      "F1 score for training set: 1.0\n",
      "Predicting labels using DecisionTreeClassifier...\n",
      "Done!\n",
      "Prediction time (secs): 0.000\n",
      "F1 score for test set: 0.616666666667\n",
      "------------------------------------------\n",
      "Training set size: 200\n",
      "Training DecisionTreeClassifier...\n",
      "Done!\n",
      "Training time (secs): 0.002\n",
      "Predicting labels using DecisionTreeClassifier...\n",
      "Done!\n",
      "Prediction time (secs): 0.000\n",
      "F1 score for training set: 1.0\n",
      "Predicting labels using DecisionTreeClassifier...\n",
      "Done!\n",
      "Prediction time (secs): 0.000\n",
      "F1 score for test set: 0.748091603053\n",
      "------------------------------------------\n",
      "Training set size: 100\n",
      "Training DecisionTreeClassifier...\n",
      "Done!\n",
      "Training time (secs): 0.001\n",
      "Predicting labels using DecisionTreeClassifier...\n",
      "Done!\n",
      "Prediction time (secs): 0.001\n",
      "F1 score for training set: 1.0\n",
      "Predicting labels using DecisionTreeClassifier...\n",
      "Done!\n",
      "Prediction time (secs): 0.001\n",
      "F1 score for test set: 0.64347826087\n"
     ]
    }
   ],
   "source": [
    "# Train and predict using different training set sizes\n",
    "def train_predict(clf, X_train, y_train, X_test, y_test):\n",
    "    print \"------------------------------------------\"\n",
    "    print \"Training set size: {}\".format(len(X_train))\n",
    "    train_classifier(clf, X_train, y_train)\n",
    "    print \"F1 score for training set: {}\".format(predict_labels(clf, X_train, y_train))\n",
    "    print \"F1 score for test set: {}\".format(predict_labels(clf, X_test, y_test))\n",
    "\n",
    "# TODO: Run the helper function above for desired subsets of training data\n",
    "#Training size 300\n",
    "train_predict(clf, X_train[0:300], y_train[0:300], X_test, y_test)\n",
    "\n",
    "#Training size 200\n",
    "train_predict(clf, X_train[0:200], y_train[0:200], X_test, y_test)\n",
    "\n",
    "#Training size 100\n",
    "train_predict(clf, X_train[0:100], y_train[0:100], X_test, y_test)\n",
    "\n",
    "\n",
    "# Note: Keep the test set constant"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# TODO: Train and predict using two other models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training GaussianNB...\n",
      "Done!\n",
      "Training time (secs): 0.001\n",
      "Predicting labels using GaussianNB...\n",
      "Done!\n",
      "Prediction time (secs): 0.001\n",
      "F1 score for training set: 0.80378250591\n",
      "Predicting labels using GaussianNB...\n",
      "Done!\n",
      "Prediction time (secs): 0.000\n",
      "F1 score for test set: 0.763358778626\n",
      "------------------------------------------\n",
      "Training set size: 300\n",
      "Training GaussianNB...\n",
      "Done!\n",
      "Training time (secs): 0.001\n",
      "Predicting labels using GaussianNB...\n",
      "Done!\n",
      "Prediction time (secs): 0.000\n",
      "F1 score for training set: 0.80378250591\n",
      "Predicting labels using GaussianNB...\n",
      "Done!\n",
      "Prediction time (secs): 0.000\n",
      "F1 score for test set: 0.763358778626\n",
      "------------------------------------------\n",
      "Training set size: 200\n",
      "Training GaussianNB...\n",
      "Done!\n",
      "Training time (secs): 0.001\n",
      "Predicting labels using GaussianNB...\n",
      "Done!\n",
      "Prediction time (secs): 0.001\n",
      "F1 score for training set: 0.840579710145\n",
      "Predicting labels using GaussianNB...\n",
      "Done!\n",
      "Prediction time (secs): 0.000\n",
      "F1 score for test set: 0.724409448819\n",
      "------------------------------------------\n",
      "Training set size: 100\n",
      "Training GaussianNB...\n",
      "Done!\n",
      "Training time (secs): 0.001\n",
      "Predicting labels using GaussianNB...\n",
      "Done!\n",
      "Prediction time (secs): 0.000\n",
      "F1 score for training set: 0.846715328467\n",
      "Predicting labels using GaussianNB...\n",
      "Done!\n",
      "Prediction time (secs): 0.000\n",
      "F1 score for test set: 0.802919708029\n"
     ]
    }
   ],
   "source": [
    "# Train a model --Naive Bayes\n",
    "import time\n",
    "\n",
    "def train_classifier(clf, X_train, y_train):\n",
    "    print \"Training {}...\".format(clf.__class__.__name__)\n",
    "    start = time.time()\n",
    "    clf.fit(X_train, y_train)\n",
    "    end = time.time()\n",
    "    print \"Done!\\nTraining time (secs): {:.3f}\".format(end - start)\n",
    "\n",
    "# TODO: Choose a model, import it and instantiate an object\n",
    "from sklearn.naive_bayes import GaussianNB\n",
    "clf = GaussianNB()\n",
    "\n",
    "# Fit model to training data\n",
    "train_classifier(clf, X_train, y_train)  # note: using entire training set here\n",
    "#print clf  # you can inspect the learned model by printing it\n",
    "\n",
    "# Predict on training set and compute F1 score\n",
    "from sklearn.metrics import f1_score\n",
    "\n",
    "def predict_labels(clf, features, target):\n",
    "    print \"Predicting labels using {}...\".format(clf.__class__.__name__)\n",
    "    start = time.time()\n",
    "    y_pred = clf.predict(features)\n",
    "    end = time.time()\n",
    "    print \"Done!\\nPrediction time (secs): {:.3f}\".format(end - start)\n",
    "    return f1_score(target.values, y_pred, pos_label='yes')\n",
    "\n",
    "train_f1_score = predict_labels(clf, X_train, y_train)\n",
    "print \"F1 score for training set: {}\".format(train_f1_score)\n",
    "\n",
    "# Predict on test data\n",
    "print \"F1 score for test set: {}\".format(predict_labels(clf, X_test, y_test))\n",
    "\n",
    "# Train and predict using different training set sizes\n",
    "def train_predict(clf, X_train, y_train, X_test, y_test):\n",
    "    print \"------------------------------------------\"\n",
    "    print \"Training set size: {}\".format(len(X_train))\n",
    "    train_classifier(clf, X_train, y_train)\n",
    "    print \"F1 score for training set: {}\".format(predict_labels(clf, X_train, y_train))\n",
    "    print \"F1 score for test set: {}\".format(predict_labels(clf, X_test, y_test))\n",
    "\n",
    "# TODO: Run the helper function above for desired subsets of training data\n",
    "# TODO: Run the helper function above for desired subsets of training data\n",
    "#Training size 300\n",
    "train_predict(clf, X_train[0:300], y_train[0:300], X_test, y_test)\n",
    "\n",
    "#Training size 200\n",
    "train_predict(clf, X_train[0:200], y_train[0:200], X_test, y_test)\n",
    "\n",
    "#Training size 100\n",
    "train_predict(clf, X_train[0:100], y_train[0:100], X_test, y_test)\n",
    "\n",
    "# Note: Keep the test set constant\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training SVC...\n",
      "Done!\n",
      "Training time (secs): 0.008\n",
      "Predicting labels using SVC...\n",
      "Done!\n",
      "Prediction time (secs): 0.006\n",
      "F1 score for training set: 0.876068376068\n",
      "Predicting labels using SVC...\n",
      "Done!\n",
      "Prediction time (secs): 0.002\n",
      "F1 score for test set: 0.783783783784\n",
      "------------------------------------------\n",
      "Training set size: 300\n",
      "Training SVC...\n",
      "Done!\n",
      "Training time (secs): 0.009\n",
      "Predicting labels using SVC...\n",
      "Done!\n",
      "Prediction time (secs): 0.006\n",
      "F1 score for training set: 0.876068376068\n",
      "Predicting labels using SVC...\n",
      "Done!\n",
      "Prediction time (secs): 0.001\n",
      "F1 score for test set: 0.783783783784\n",
      "------------------------------------------\n",
      "Training set size: 200\n",
      "Training SVC...\n",
      "Done!\n",
      "Training time (secs): 0.003\n",
      "Predicting labels using SVC...\n",
      "Done!\n",
      "Prediction time (secs): 0.003\n",
      "F1 score for training set: 0.867924528302\n",
      "Predicting labels using SVC...\n",
      "Done!\n",
      "Prediction time (secs): 0.001\n",
      "F1 score for test set: 0.781456953642\n",
      "------------------------------------------\n",
      "Training set size: 100\n",
      "Training SVC...\n",
      "Done!\n",
      "Training time (secs): 0.002\n",
      "Predicting labels using SVC...\n",
      "Done!\n",
      "Prediction time (secs): 0.000\n",
      "F1 score for training set: 0.877697841727\n",
      "Predicting labels using SVC...\n",
      "Done!\n",
      "Prediction time (secs): 0.001\n",
      "F1 score for test set: 0.774647887324\n"
     ]
    }
   ],
   "source": [
    "# Train a model --Support Vector Machine\n",
    "import time\n",
    "\n",
    "def train_classifier(clf, X_train, y_train):\n",
    "    print \"Training {}...\".format(clf.__class__.__name__)\n",
    "    start = time.time()\n",
    "    clf.fit(X_train, y_train)\n",
    "    end = time.time()\n",
    "    print \"Done!\\nTraining time (secs): {:.3f}\".format(end - start)\n",
    "\n",
    "# TODO: Choose a model, import it and instantiate an object\n",
    "from sklearn import svm\n",
    "clf = svm.SVC()\n",
    "\n",
    "# Fit model to training data\n",
    "train_classifier(clf, X_train, y_train)  # note: using entire training set here\n",
    "#print clf  # you can inspect the learned model by printing it\n",
    "\n",
    "# Predict on training set and compute F1 score\n",
    "from sklearn.metrics import f1_score\n",
    "\n",
    "def predict_labels(clf, features, target):\n",
    "    print \"Predicting labels using {}...\".format(clf.__class__.__name__)\n",
    "    start = time.time()\n",
    "    y_pred = clf.predict(features)\n",
    "    end = time.time()\n",
    "    print \"Done!\\nPrediction time (secs): {:.3f}\".format(end - start)\n",
    "    return f1_score(target.values, y_pred, pos_label='yes')\n",
    "\n",
    "train_f1_score = predict_labels(clf, X_train, y_train)\n",
    "print \"F1 score for training set: {}\".format(train_f1_score)\n",
    "\n",
    "# Predict on test data\n",
    "print \"F1 score for test set: {}\".format(predict_labels(clf, X_test, y_test))\n",
    "\n",
    "# Train and predict using different training set sizes\n",
    "def train_predict(clf, X_train, y_train, X_test, y_test):\n",
    "    print \"------------------------------------------\"\n",
    "    print \"Training set size: {}\".format(len(X_train))\n",
    "    train_classifier(clf, X_train, y_train)\n",
    "    print \"F1 score for training set: {}\".format(predict_labels(clf, X_train, y_train))\n",
    "    print \"F1 score for test set: {}\".format(predict_labels(clf, X_test, y_test))\n",
    "\n",
    "# TODO: Run the helper function above for desired subsets of training data\n",
    "# TODO: Run the helper function above for desired subsets of training data\n",
    "#Training size 300\n",
    "train_predict(clf, X_train[0:300], y_train[0:300], X_test, y_test)\n",
    "\n",
    "#Training size 200\n",
    "train_predict(clf, X_train[0:200], y_train[0:200], X_test, y_test)\n",
    "\n",
    "#Training size 100\n",
    "train_predict(clf, X_train[0:100], y_train[0:100], X_test, y_test)\n",
    "\n",
    "# Note: Keep the test set constant\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Decision Tree</th>\n",
       "      <th>Unnamed: 1</th>\n",
       "      <th>Unnamed: 2</th>\n",
       "      <th>Unnamed: 3</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Training Size</td>\n",
       "      <td>300.000</td>\n",
       "      <td>200.000</td>\n",
       "      <td>100.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Training Time</td>\n",
       "      <td>0.003</td>\n",
       "      <td>0.002</td>\n",
       "      <td>0.001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Prediction Time</td>\n",
       "      <td>0.001</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>F1 Score Training</td>\n",
       "      <td>1.000</td>\n",
       "      <td>1.000</td>\n",
       "      <td>1.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>F1 Score Test</td>\n",
       "      <td>0.617</td>\n",
       "      <td>0.748</td>\n",
       "      <td>0.643</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       Decision Tree  Unnamed: 1  Unnamed: 2  Unnamed: 3\n",
       "0      Training Size     300.000     200.000     100.000\n",
       "1      Training Time       0.003       0.002       0.001\n",
       "2    Prediction Time       0.001       0.000       0.000\n",
       "3  F1 Score Training       1.000       1.000       1.000\n",
       "4      F1 Score Test       0.617       0.748       0.643"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import display\n",
    "import pandas\n",
    "df = pandas.read_csv('DecisionTree_table.csv')\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Na�ve Bayes-Gaussian</th>\n",
       "      <th>Unnamed: 1</th>\n",
       "      <th>Unnamed: 2</th>\n",
       "      <th>Unnamed: 3</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Training Size</td>\n",
       "      <td>300.000</td>\n",
       "      <td>200.000</td>\n",
       "      <td>100.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Training Time</td>\n",
       "      <td>0.001</td>\n",
       "      <td>0.001</td>\n",
       "      <td>0.001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Prediction Time</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.001</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>F1 Score Training</td>\n",
       "      <td>0.804</td>\n",
       "      <td>0.841</td>\n",
       "      <td>0.847</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>F1 Score Test</td>\n",
       "      <td>0.763</td>\n",
       "      <td>0.724</td>\n",
       "      <td>0.803</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Na�ve Bayes-Gaussian  Unnamed: 1  Unnamed: 2  Unnamed: 3\n",
       "0        Training Size     300.000     200.000     100.000\n",
       "1        Training Time       0.001       0.001       0.001\n",
       "2      Prediction Time       0.000       0.001       0.000\n",
       "3    F1 Score Training       0.804       0.841       0.847\n",
       "4        F1 Score Test       0.763       0.724       0.803"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pandas.read_csv('GaussianNB_table.csv')\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>SVM</th>\n",
       "      <th>Unnamed: 1</th>\n",
       "      <th>Unnamed: 2</th>\n",
       "      <th>Unnamed: 3</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Training Size</td>\n",
       "      <td>300.000</td>\n",
       "      <td>200.000</td>\n",
       "      <td>100.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Training Time</td>\n",
       "      <td>0.008</td>\n",
       "      <td>0.003</td>\n",
       "      <td>0.001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Prediction Time</td>\n",
       "      <td>0.005</td>\n",
       "      <td>0.003</td>\n",
       "      <td>0.001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>F1 Score Training</td>\n",
       "      <td>0.876</td>\n",
       "      <td>0.868</td>\n",
       "      <td>878.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>F1 Score Test</td>\n",
       "      <td>0.784</td>\n",
       "      <td>0.781</td>\n",
       "      <td>0.775</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 SVM  Unnamed: 1  Unnamed: 2  Unnamed: 3\n",
       "0      Training Size     300.000     200.000     100.000\n",
       "1      Training Time       0.008       0.003       0.001\n",
       "2    Prediction Time       0.005       0.003       0.001\n",
       "3  F1 Score Training       0.876       0.868     878.000\n",
       "4      F1 Score Test       0.784       0.781       0.775"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pandas.read_csv('SVM_table.csv')\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Choosing the Best Model\n",
    "\n",
    "\n",
    "- Based on the experiments you performed earlier, in 1-2 paragraphs explain to the board of supervisors what single model you chose as the best model. Which model is generally the most appropriate based on the available data, limited resources, cost, and performance?\n",
    "    \n",
    "    I have chosen to implement a Naive Bayes model given it has acheived the highest F1(.80) score on the test set. This demonstrates it is able to correctly classify whether a student will pass. This model also fits well with the generally low number of datapoints available to train on and is able to adapt if a feature value is missing. Naive Bayes also scales well if other factors are added in to the model as these additions will result in lower increases in time to train the model and the time it takes to perform predictions. This model is also very fast due to its simplicity so it should be able to perform well even with limited resources able to be dedicated to it in terms of computation power and time. \n",
    " \n",
    " \n",
    "- In 1-2 paragraphs explain to the board of supervisors in layman's terms how the final model chosen is supposed to work (for example if you chose a Decision Tree or Support Vector Machine, how does it make a prediction).\n",
    "    \n",
    "    A Naive Bayes model assumes that each attribute examined, (e.g. gender, family size, location) all contribute independently to the overall probability of the students success. Using the training data, the Naive Bayes learner creates assigns a probability to each feature based on how likely it is associated with a student passing label. When given a new sample set, it combines these probabilities based on each feature value to provide the overall probability between 0-1 which is the probability of our student getting a passing label.\n",
    "\n",
    "\n",
    "- Fine-tune the model. Use Gridsearch with at least one important parameter tuned and with at least 3 settings. Use the entire training set for this.\n",
    " \n",
    " Given that Gaussian Naive Bayes does not have hyper parameters, there are no parameters to tune using grid search. \n",
    "\n",
    "\n",
    "- What is the model's final F<sub>1</sub> score?\n",
    "\n",
    ".803"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.803"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# TODO: Fine-tune your model and report the best F1 score\n",
    ".803"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}