logistic_regression_tutorial_answers.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Logistic Regression Tutorial"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn import linear_model, metrics, model_selection\n",
    "import random"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Classification\n",
    "\n",
    "In regression we were trying to predict a continuous value - e.g. exam score.\n",
    "\n",
    "In classification we predict a class label for a given data point - e.g. pass/fail the exam. It might be a two-class problem like this (binary classification) or a many-class problem.\n",
    "\n",
    "Not only do we predict a class label, we also predict a probability for each class.\n",
    "\n",
    "Our example of pass fail can be pictured like this\n",
    "\n",
    "![logistic_regression](https://upload.wikimedia.org/wikipedia/commons/6/6d/Exam_pass_logistic_curve.jpeg)\n",
    "\n",
    "To perform classification we need to make a **decision**. This requires defining a **decision boundary**. This is going to be an affine set - in 1D (i.e. with one feature) this will be a point; in 2D (with two features) this will be a line; with three features a plane; etc...\n",
    "\n",
    "In our case above we could say if `num_hours_studying` $\\gt 2.7$ then we will predict `pass`.\n",
    "\n",
    "In a bivariate case our decision boundary might look like this\n",
    "\n",
    "![bivariate_logistic_regression](https://i0.wp.com/ucanalytics.com/blogs/wp-content/uploads/2017/09/Scatter-Plot-with-Boundary-Logistic-Regression.jpg?resize=768%2C578)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hypothesis\n",
    "\n",
    "Our hypothesis for logistic regression can be linear or non-linear. What is different with linear regression is we want to output class probabilities. We need a way to express our hypothesis as a probability value.\n",
    "\n",
    "To do this we use the sigmoid function\n",
    "\n",
    "$$\\sigma(h) = \\frac{e^h}{e^h + 1} = \\frac{1}{1 + e^{-h}}$$\n",
    "\n",
    "![sigmoid](https://upload.wikimedia.org/wikipedia/commons/8/88/Logistic-curve.svg)\n",
    "\n",
    "We can see that this function is going to squash its input into the range $(0, 1)$ giving us a valid probability value.\n",
    "\n",
    "We will pass our linear or non-linear function of the data through the sigmoid function to get our final hypothesis\n",
    "\n",
    "$$h_\\theta(\\mathbf{x}) = P(y=1|\\mathbf{x}) = \\sigma(\\mathbf{a}^\\top\\mathbf{x})$$\n",
    "\n",
    "We can arbitrarily decide what $y=1$ means - in the example let's say passes the exam. Then the probability of failing the exam is given by\n",
    "\n",
    "$$P(y=0|\\mathbf{x}) = 1 - P(y=1|\\mathbf{x}) = 1 - \\sigma(\\mathbf{a}^\\top\\mathbf{x})$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loss Function\n",
    "\n",
    "MSE is not an appropriate loss function here. We are dealing with probabilities so the natural choice is negative log loss\n",
    "\n",
    "$$\\text{NLL} = -y \\log(h_\\theta) - (1 - y)\\log(1 - h_\\theta))$$\n",
    "\n",
    "$y$ is going to be either $1$ or $0$ so only one of these two terms will apply."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Optimization\n",
    "\n",
    "We cannot use a closed form here so we will have to use **gradient descent**.\n",
    "\n",
    "When we learned about linear regression we noted that the loss surface was convex - from any point you know how to reach the global minimum.\n",
    "\n",
    "Our loss here is highly non-convex. Instead we need to use an iterative algorithm to take clever steps from an initial starting point to try and find a good local minimum. The picture looks like this\n",
    "\n",
    "![gradient_descent](https://cdn-images-1.medium.com/max/1600/1*f9a162GhpMbiTVTAua_lLQ.png)\n",
    "\n",
    "The key idea is that if you take the gradient of the parameters with respect to the loss at a particular point, it will tell you how to change those parameters to reduce the loss\n",
    "\n",
    "![gradient_descent_2](https://cdn-images-1.medium.com/max/800/0*rBQI7uBhBKE8KT-X.png)\n",
    "\n",
    "Clearly in this diagram if we start on the right, we need to **decrease** $w$ in order to lower our loss. That's really the main idea.\n",
    "\n",
    "With this algorithm we make a number of \"steps\" to update an improve the parameters. Each step the update is\n",
    "\n",
    "$$\\theta' = \\theta - \\beta \\frac{\\partial \\text{NLL}(h_\\theta, y)}{\\partial \\theta}$$\n",
    "\n",
    "Here the hyperparameter $\\beta$ is very important - it controls how big our steps are. In the above diagram if the learning rate made us step half the width of the \"U\" we would just bounce from side to side and never settle in the optimum.\n",
    "\n",
    "We also need to pay attention to how many steps we take. In the diagram if we just took two we wouldn't give the algorithm time to settle. In practice we will take many more steps than two. Be aware that you will have to tune this hyperparameter."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset\n",
    "\n",
    "A retrospective sample of males in a heart-disease high-risk region of the Western Cape, South Africa. There are roughly two controls per case of coronary heart disease (CHD). Many of the CHD positive men have undergone blood pressure reduction treatment and other programs to reduce their risk factors after their CHD event. In some cases the measurements were made after these treatments. These data are taken from a larger dataset, described in  Rousseauw et al, 1983, South African Medical Journal.  Downloaded from https://web.stanford.edu/~hastie/ElemStatLearn/.\n",
    "\n",
    "Features:\n",
    "- sbp: systolic blood pressure\n",
    "- tobacco:\tcumulative tobacco (kg)\n",
    "- ldl: low densiity lipoprotein cholesterol\n",
    "- adiposity\n",
    "- famhist: family history of heart disease (Present, Absent)\n",
    "- typea: type-A behavior\n",
    "- obesity\n",
    "- alcohol: current alcohol consumption\n",
    "- age: age at onset\n",
    "- chd: response, coronary heart disease"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preprocess the Data\n",
    "\n",
    "### Load the Data\n",
    "\n",
    "As usual, load the data with `pd.read_csv`. We have an index column in position zero and the separator is a comma. Load the data into a variable named `data`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "Task: load the data into a variable named data\n",
    "\"\"\"\n",
    "file_name = 'SAheart.data'\n",
    "data = pd.read_csv(file_name, sep=',', index_col=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sbp</th>\n",
       "      <th>tobacco</th>\n",
       "      <th>ldl</th>\n",
       "      <th>adiposity</th>\n",
       "      <th>famhist</th>\n",
       "      <th>typea</th>\n",
       "      <th>obesity</th>\n",
       "      <th>alcohol</th>\n",
       "      <th>age</th>\n",
       "      <th>chd</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>row.names</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>160</td>\n",
       "      <td>12.00</td>\n",
       "      <td>5.73</td>\n",
       "      <td>23.11</td>\n",
       "      <td>Present</td>\n",
       "      <td>49</td>\n",
       "      <td>25.30</td>\n",
       "      <td>97.20</td>\n",
       "      <td>52</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>144</td>\n",
       "      <td>0.01</td>\n",
       "      <td>4.41</td>\n",
       "      <td>28.61</td>\n",
       "      <td>Absent</td>\n",
       "      <td>55</td>\n",
       "      <td>28.87</td>\n",
       "      <td>2.06</td>\n",
       "      <td>63</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>118</td>\n",
       "      <td>0.08</td>\n",
       "      <td>3.48</td>\n",
       "      <td>32.28</td>\n",
       "      <td>Present</td>\n",
       "      <td>52</td>\n",
       "      <td>29.14</td>\n",
       "      <td>3.81</td>\n",
       "      <td>46</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>170</td>\n",
       "      <td>7.50</td>\n",
       "      <td>6.41</td>\n",
       "      <td>38.03</td>\n",
       "      <td>Present</td>\n",
       "      <td>51</td>\n",
       "      <td>31.99</td>\n",
       "      <td>24.26</td>\n",
       "      <td>58</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>134</td>\n",
       "      <td>13.60</td>\n",
       "      <td>3.50</td>\n",
       "      <td>27.78</td>\n",
       "      <td>Present</td>\n",
       "      <td>60</td>\n",
       "      <td>25.99</td>\n",
       "      <td>57.34</td>\n",
       "      <td>49</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           sbp  tobacco   ldl  adiposity  famhist  typea  obesity  alcohol  \\\n",
       "row.names                                                                    \n",
       "1          160    12.00  5.73      23.11  Present     49    25.30    97.20   \n",
       "2          144     0.01  4.41      28.61   Absent     55    28.87     2.06   \n",
       "3          118     0.08  3.48      32.28  Present     52    29.14     3.81   \n",
       "4          170     7.50  6.41      38.03  Present     51    31.99    24.26   \n",
       "5          134    13.60  3.50      27.78  Present     60    25.99    57.34   \n",
       "\n",
       "           age  chd  \n",
       "row.names            \n",
       "1           52    1  \n",
       "2           63    1  \n",
       "3           46    0  \n",
       "4           58    1  \n",
       "5           49    1  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Manage Categorial Variable\n",
    "\n",
    "We once again have a categorical variable, `famhist`, which we need to make into a one-hot encoding. A one-hot encoding assigns a class label to a position in list of binary variables (`0` or `1`), and puts a `1` in the position that corresponds to the desired class.\n",
    "\n",
    "So for `famhist` we will make a one hot encoding of two binary variables\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "\\text{famhist: Present} \\rightarrow \\begin{bmatrix} 1 & 0 \\end{bmatrix} \\\\\n",
    "\\text{famhist: Absent} \\rightarrow \\begin{bmatrix} 0 & 1 \\end{bmatrix} \\\\\n",
    "\\end{align}\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "data['famhist_true'] = data['famhist'] == 'Present'\n",
    "data['famhist_false'] = data['famhist'] == 'Absent'\n",
    "data = data.drop(['famhist'], axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because we will perform feature selection later, this time let's take the time to make a convenient function to separate a dataset into `train` and `test` and also split $x$ and $y$ in boths sets for us. We do this for you."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def split(data):\n",
    "    # control randomization for reproducibility\n",
    "    np.random.seed(42)\n",
    "    random.seed(42)\n",
    "    train, test = model_selection.train_test_split(data)\n",
    "    x_train = train.loc[:, train.columns != 'chd']\n",
    "    y_train = train['chd']\n",
    "    x_test = test.loc[:, test.columns != 'chd']\n",
    "    y_test = test['chd']\n",
    "    return x_train, y_train, x_test, y_test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Visualization\n",
    "\n",
    "Let's now take a look at the relationships in the data. We can still use scatterplots for classification, but they look a bit different. Again let's make a convenient function for later use. As you will see we stretch the plot size to try and make these plots a bit clearer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_feature(data, feature_name):\n",
    "    plt.figure(figsize=(10, 3))\n",
    "    plt.scatter(data[feature_name], data['chd'])\n",
    "    plt.xlabel(feature_name)\n",
    "    plt.ylabel('chd')\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll take a look at one feature together then you should explore the rest of the dataset on your own and form your own opinions about which features are going to be useful."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmQAAADTCAYAAAAvQQ9YAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAGhJJREFUeJzt3XuQHfV14PHv8QiBTMCyjKCMJCyCFRyVIcae5RF5HfxQEOCAQoyNytQalxdqKyZrLw5bsFCwsFCQsPGG2pBssOOAYwLBGCuqRbHMYrzZYo3MYBxkBAriKQmCJryDMQ/t2T9uX3E1mum+mjutvjPz/VRR3P79un997rn9OOrbfScyE0mSJDXnbU0HIEmSNN1ZkEmSJDXMgkySJKlhFmSSJEkNsyCTJElqmAWZJElSwyzIJEmSGmZBJkmS1DALMkmSpIbNaDqAXbXffvvlwoULmw5DkiSp0r333vvPmTm3ar5JV5AtXLiQoaGhpsOQJEmqFBFPdDOfX1lKkiQ1zIJMkiSpYRZkkiRJDbMgkyRJapgFmSRJUsNqe8oyIr4BfBLYmpnvH6U/gKuBE4CfA2dk5k/qime6W3nfFq5as4GnXniVA2fP4tzjDmX5EfOaDmu36Of33ktsF65cx41rN7Etk4EIVhy1gMuWH9Z1f9W6P/u1H3HXI89tn15yyBxuOPOYnuPuJralX/0hD299Zfv0ov335vZzju0qtqplj7r8dp55+fXt0wfsM5O1FyydsP6y2A6/+Hu89Nq27X377jnA/Zcs2z79vgtW84ttuX16r4HgoctP6Dov7z3/Nt58a3FmBGy84sSuYq+KrWrsqtirxi+LrSrnvYwN1dtj2Xsr+7y7Gbtq+ar+sn2xKi+9HkN6ia3X41Ov+vG8EJlZPdd4Bo74CPAvwDfHKMhOAH6PVkF2FHB1Zh5VNe7g4GD6sxe7ZuV9Wzj/1nW8+sZbO+asPQa44pTDGt8A69bP772X2C5cuY5v3f3kTu2nH30Qly0/rLK/at0jD7RtSw6Zw6mDB/WU06rYRhYdbe3ioyy2rS+/VrrsyBNzW/sE3Wt/WWzrNr+4w8mxrX2SHHnSb2uf/KvyMrJgamsXTmWxv/r6ttLYqsauin1kYTBy/LLYgNKc9zL22guWVm6PZe/tQwvfOebnfcOZx1SOXba93HDmMZX9ZfvxRSvXleal12NIL7ENPfFcT8enXu3u80JE3JuZg1Xz1faVZWb+PbDzp/WWk2kVa5mZdwOzI+LddcUznV21ZsMOGx7Aq29s46o1GxqKaPfp5/feS2w3rt1U2l7VX7Xu0Q607fZec1oV22hFR2d7WWxVy452Yu5s77W/LLbRTo7A9vbRTvqd7VXvbbSCqbO9LPaq2KrGroq9avyy2Kpy3svYUL09lr23ss+7m7Grlq/qL9sXq/LS6zGkl9h6PT71ql/PC03eQzYP6PxUNhdtO4mIsyJiKCKGhoeHd0twU8lTL7y6S+1TST+/915i2zbGle12e1V/L+vuNadVsUm7U53bY93bepPHkF5iq/P41GtsTZoUN/Vn5rWZOZiZg3PnVv71AY1w4OxZu9Q+lfTze+8ltoGI0vaq/l7W3WtOq2KTdqc6t8e6t/UmjyG9xFbn8anX2JrUZEG2BVjQMT2/aNMEO/e4Q5m1x8AObbP2GODc4w5tKKLdp5/fey+xrThqQWl7VX/VupccMmfU5ZccMqfnnFbFtmj/vUftb7eXxVa1bPuepJHa7b32l8W2754Do/a12/caGP0k1W6vem8zxjhPttvLYq+KrWrsqtirxi+LrSrnvYwN1dtj2Xsr+7y7Gbtq+ar+sn2xKi+9HkN6ia3X41Ov+vW80GRBtgr4N9FyNPBiZj7dYDxT1vIj5nHFKYcxb/YsApg3e1Zf3NS+O/Tze+8ltsuWH8bpRx+0w79m2zfEdtNfte4bzjxmpwNu+2bdXnNaFdvt5xy7U/HR+TRhWWxVy669YOlOJ+jOJ+567S+L7f5Llu10kux86u2hy0/Y6eTf+TRf1XvbeMWJOxVOnU9ClsVeFVvV2FWxV41fFltVznsZG6q3x7L3VvZ5dzN21fJV/WX7YlVeej2G9BJbr8enXvXreaHOpyxvBI4F9gOeAS4G9gDIzP9R/OzFnwDLaP3sxeczs/LxSZ+ylCRJk0W3T1nW9jtkmbmioj+BL9a1fkmSpMliUtzUL0mSNJVZkEmSJDXMgkySJKlhFmSSJEkNsyCTJElqmAWZJElSwyzIJEmSGmZBJkmS1DALMkmSpIZZkEmSJDXMgkySJKlhFmSSJEkNsyCTJElqmAWZJElSwyzIJEmSGmZBJkmS1DALMkmSpIZZkEmSJDXMgkySJKlhtRZkEbEsIjZExMaIOG+U/oMi4s6IuC8i7o+IE+qMR5IkqR/VVpBFxABwDXA8sBhYERGLR8x2IXBzZh4BnAb8aV3xSJIk9as6r5AdCWzMzEcz83XgJuDkEfMksG/x+h3AUzXGI0mS1JfqLMjmAZs6pjcXbZ3+M3B6RGwGVgO/N9pAEXFWRAxFxNDw8HAdsUqSJDWm6Zv6VwDXZeZ84ATgryJip5gy89rMHMzMwblz5+72ICVJkupUZ0G2BVjQMT2/aOv0BeBmgMz8EbAXsF+NMUmSJPWdOguye4BFEXFwRMykddP+qhHzPAl8HCAifpVWQeZ3kpIkaVqprSDLzDeBs4E1wIO0nqZ8ICIujYiTitm+ApwZEf8A3AickZlZV0ySJEn9aEadg2fmalo363e2XdTxej2wpM4YJEmS+l3TN/VLkiRNexZkkiRJDbMgkyRJapgFmSRJUsMsyCRJkhpmQSZJktQwCzJJkqSGWZBJkiQ1zIJMkiSpYRZkkiRJDbMgkyRJapgFmSRJUsMsyCRJkhpmQSZJktQwCzJJkqSGWZBJkiQ1zIJMkiSpYRZkkiRJDbMgkyRJapgFmSRJUsNqLcgiYllEbIiIjRFx3hjzfDoi1kfEAxHx13XGI0mS1I9m1DVwRAwA1wBLgc3APRGxKjPXd8yzCDgfWJKZz0fE/nXFI0mS1K/qvEJ2JLAxMx/NzNeBm4CTR8xzJnBNZj4PkJlba4xHkiSpL9VZkM0DNnVMby7aOv0K8CsRcVdE3B0Ry0YbKCLOioihiBgaHh6uKVxJkqRmNH1T/wxgEXAssAL4WkTMHjlTZl6bmYOZOTh37tzdHKIkSVK96izItgALOqbnF22dNgOrMvONzHwM+EdaBZokSdK0UWdBdg+wKCIOjoiZwGnAqhHzrKR1dYyI2I/WV5iP1hiTJElS36mtIMvMN4GzgTXAg8DNmflARFwaEScVs60Bno2I9cCdwLmZ+WxdMUmSJPWjyMymY9glg4ODOTQ01HQYkiRJlSLi3swcrJqv6Zv6JUmSpj0LMkmSpIaV/lJ/RJxS1p+Zt05sOJIkSdNP1Z9O+q3i//sDvw78oJj+KPB/AQsySZKkHpUWZJn5eYCI+D6wODOfLqbfDVxXe3SSJEnTQLf3kC1oF2OFZ4CDaohHkiRp2qn6yrLtjohYA9xYTH8G+F/1hCRJkjS9dFWQZebZxQ3+/7poujYzv1tfWJIkSdNHt1fI2k9UehO/JEnSBOvqHrKIOCUiHo6IFyPipYh4OSJeqjs4SZKk6aDbK2R/CPxWZj5YZzCSJEnTUbdPWT5jMSZJklSPbn+pfygi/gZYCbzW7veX+iVJknrX7S/1J/Bz4Dc7+hJv8pckSepZt7/Ufz3wpcx8oZh+J/BH9YcnSZI09XV7D9nh7WIMIDOfB46oJyRJkqTppduC7G3FVTEAImIOu/AbZpIkSRpbt0XVHwE/iohvF9OnApfXE5IkSdL00u2fTvpmRAwBHyuaTsnM9fWFJUmSNH3syp9OWg9YhEmSJE2wbu8hG5eIWBYRGyJiY0ScVzLf70RERsRgnfFIkiT1o9oKsogYAK4BjgcWAysiYvEo8+0DfAlYW1cskiRJ/azOK2RHAhsz89HMfB24CTh5lPn+C/AHwC9qjEWSJKlv1VmQzQM2dUxvLtq2i4gPAgsy87aygSLirIgYioih4eHhiY9UkiSpQbXeQ1YmIt4GfBX4StW8mXltZg5m5uDcuXPrD06SJGk3qrMg2wIs6JieX7S17QO8H/hhRDwOHA2s8sZ+SZI03dRZkN0DLIqIgyNiJnAasKrdmZkvZuZ+mbkwMxcCdwMnZeZQjTFJkiT1ndoKssx8EzgbWAM8CNycmQ9ExKURcVJd65UkSZpsav17lJm5Glg9ou2iMeY9ts5YJEmS+lVjN/VLkiSpxYJMkiSpYRZkkiRJDbMgkyRJapgFmSRJUsMsyCRJkhpmQSZJktQwCzJJkqSGWZBJkiQ1zIJMkiSpYRZkkiRJDbMgkyRJapgFmSRJUsMsyCRJkhpmQSZJktQwCzJJkqSGWZBJkiQ1zIJMkiSpYRZkkiRJDau1IIuIZRGxISI2RsR5o/SfExHrI+L+iLgjIt5TZzySJEn9qLaCLCIGgGuA44HFwIqIWDxitvuAwcw8HLgF+MO64pEkSepXdV4hOxLYmJmPZubrwE3AyZ0zZOadmfnzYvJuYH6N8UiSJPWlOguyecCmjunNRdtYvgD83WgdEXFWRAxFxNDw8PAEhihJktS8vripPyJOBwaBq0brz8xrM3MwMwfnzp27e4OTJEmq2Ywax94CLOiYnl+07SAiPgFcAPxGZr5WYzySJEl9qc4rZPcAiyLi4IiYCZwGrOqcISKOAP4cOCkzt9YYiyRJUt+qrSDLzDeBs4E1wIPAzZn5QERcGhEnFbNdBfwS8O2I+GlErBpjOEmSpCmrzq8syczVwOoRbRd1vP5EneuXJEmaDPripn5JkqTpzIJMkiSpYRZkkiRJDbMgkyRJapgFmSRJUsMsyCRJkhpmQSZJktQwCzJJkqSGWZBJkiQ1zIJMkiSpYRZkkiRJDbMgkyRJapgFmSRJUsMsyCRJkhpmQSZJktQwCzJJkqSGWZBJkiQ1zIJMkiSpYRZkkiRJDbMgkyRJatiMOgePiGXA1cAA8PXMvHJE/57AN4EPAc8Cn8nMx+uMqczK+7Zw1ZoNPPXCqxw4exbnHncoy4+Y11Q400ZV3uv8XKrGvnDlOm5cu4ltmQxEsOKoBVy2/LCu+6vG/+zXfsRdjzy3fXrJIXO44cxjulp26Vd/yMNbX9k+vWj/vbn9nGO7GrvX2KvWXTX2+y5YzS+25fbpvQaChy4/Yfv0wefdxlu9EMBjV57Y1XuvytvC825jpMc7xq6zv8l1Q3lee1m27tjee/5tvNnROSNg4xUTt+6q7bFse6uKrSpvh1/8PV56bdv26X33HOD+S5Ztn67aj4+6/Haeefn17dMH7DOTtRcsBXo7/nSjyWN3r7H1o8jM6rnGM3DEAPCPwFJgM3APsCIz13fM87vA4Zn57yLiNOC3M/MzZeMODg7m0NDQhMe78r4tnH/rOl59460dY9YeA1xxymF9/yFOZlV5r/NzqRr7wpXr+NbdT+603OlHH8Rlyw+r7K8af+TBsG3JIXM4dfCg0mVHniDa2ieKsrFvOPOYnmK/5s6HS9ddNfbIk19b+yQ48gTW1j6Rlb33L350UWneRjsxtz1+5Ym19pepe92PX3liaV7LzgJVyz5Wc2wDwQ4FT1u78Ol13VXbY9n29tjwK6WxVeVtZDHW1i7KqvbjkcVY2wH7zOT8ExaP+/jTTVHW5LG719h2t4i4NzMHq+ar8yvLI4GNmfloZr4O3AScPGKek4Hri9e3AB+PiKgxpjFdtWbDDh8ewKtvbOOqNRuaCGfaqMp7nZ9L1dg3rt006nLt9qr+qvFHOxi226uWHe0E0dleNnavsVetu2rs0U5+ne1jFQft9rL1ux+PrSqvdS3bjbLxRyt4KGnfVVXbY9n2VhVbVd5GK8Y626v249GKsXZ7L8efbjR57O41tn5VZ0E2D+g8Mm8u2kadJzPfBF4E3jVyoIg4KyKGImJoeHi4lmCfeuHVXWrXxKjKe52fS9XY28a4etxur+rvJfa6t8c6Y68au07ux1JL3ftCk8fuKpP1ODApburPzGszczAzB+fOnVvLOg6cPWuX2jUxqvJe5+dSNfbAGBdr2+1V/b3EXvf2WGfsVWPXyf1Yaql7X2jy2F1lsh4H6izItgALOqbnF22jzhMRM4B30Lq5f7c797hDmbXHwA5ts/YY4NzjDm0inGmjKu91fi5VY684asFoi21vr+qvGn/JIXNGXX7JIXMql120/96jLttuLxu719ir1l019l4Doxdm7faxyrZ2e9n63Y/HVpXXupbtRtn4M8boHKt9V1Vtj2XbW1VsVXnbd8+BUfvb7VX78QH7zBy1/4B9ZvZ0/OlGk8fuXmPrV3UWZPcAiyLi4IiYCZwGrBoxzyrgc8XrTwE/yLqeMqiw/Ih5XHHKYcybPYsA5s2e5Q39u0FV3uv8XKrGvmz5YZx+9EE7XBFr35jeTX/V+DececxOB7/2DbVVy95+zrE7nSg6n/wqG7vX2KvWXTX2Q5efsNNJsPOptseuPHGnE1nnk2ll66/KW+eTdZ3a7XX2N7luKM9rL8vWHdvGK07cqfDpfJKx13VXbY9l21tVbFV5u/+SZTsVZZ1PWVbtx2svWLpTUdZ+yrKX4083mjx29xpbv6rtKUuAiDgB+GNaP3vxjcy8PCIuBYYyc1VE7AX8FXAE8BxwWmY+WjZmXU9ZSpIkTbRun7Ks9XfIMnM1sHpE20Udr38BnFpnDJIkSf1uUtzUL0mSNJVZkEmSJDXMgkySJKlhFmSSJEkNq/UpyzpExDDwBLAf8M8NhzMZmbfxMW/jY97Gx7yNj3kbH/M2Pt3m7T2ZWfmr9pOuIGuLiKFuHiPVjszb+Ji38TFv42Pexse8jY95G5+JzptfWUqSJDXMgkySJKlhk7kgu7bpACYp8zY+5m18zNv4mLfxMW/jY97GZ0LzNmnvIZMkSZoqJvMVMkmSpCnBgkySJKlhk6Igi4hvRMTWiPhZR9uciLg9Ih4u/v/OJmPsRxGxICLujIj1EfFARHypaDd3JSJir4j4cUT8Q5G3S4r2gyNibURsjIi/iYiZTcfabyJiICLui4j/WUybsy5ExOMRsS4ifhoRQ0Wb+2mFiJgdEbdExEMR8WBEHGPeykXEocV21v7vpYj4snmrFhH/oTgn/CwibizOFRN2jJsUBRlwHbBsRNt5wB2ZuQi4o5jWjt4EvpKZi4GjgS9GxGLMXZXXgI9l5q8BHwCWRcTRwB8A/y0z3ws8D3yhwRj71ZeABzumzVn3PpqZH+j4XSP302pXA9/LzPcBv0Zr2zNvJTJzQ7GdfQD4EPBz4LuYt1IRMQ/498BgZr4fGABOYwKPcZOiIMvMvweeG9F8MnB98fp6YPluDWoSyMynM/MnxeuXaR2s5mHuSmXLvxSTexT/JfAx4Jai3byNEBHzgROBrxfTgTnrhftpiYh4B/AR4C8AMvP1zHwB87YrPg48kplPYN66MQOYFREzgLcDTzOBx7hJUZCN4YDMfLp4/U/AAU0G0+8iYiFwBLAWc1ep+Ortp8BW4HbgEeCFzHyzmGUzreJWb/lj4D8C/6+YfhfmrFsJfD8i7o2Is4o299NyBwPDwF8WX5N/PSL2xrztitOAG4vX5q1EZm4B/ivwJK1C7EXgXibwGDeZC7LtsvXbHf5+xxgi4peA7wBfzsyXOvvM3egyc1txSX8+cCTwvoZD6msR8Ulga2be23Qsk9SHM/ODwPG0bi34SGen++moZgAfBP4sM48AXmHE12zmbWzFvU4nAd8e2WfedlbcU3cyrX8IHAjszc63UvVkMhdkz0TEuwGK/29tOJ6+FBF70CrGbsjMW4tmc9el4iuQO4FjgNnFpWpoFWpbGgus/ywBToqIx4GbaF3Gvxpz1pXiX99k5lZa9/Mciftplc3A5sxcW0zfQqtAM2/dOR74SWY+U0ybt3KfAB7LzOHMfAO4ldZxb8KOcZO5IFsFfK54/TngbxuMpS8V9/D8BfBgZn61o8vclYiIuRExu3g9C1hK6/67O4FPFbOZtw6ZeX5mzs/MhbS+BvlBZn4Wc1YpIvaOiH3ar4HfBH6G+2mpzPwnYFNEHFo0fRxYj3nr1gre+roSzFuVJ4GjI+Ltxbm1vb1N2DFuUvxSf0TcCBwL7Ac8A1wMrARuBg4CngA+nZkjb/yf1iLiw8D/Adbx1n09/4nWfWTmbgwRcTitmzMHaP2j5ebMvDQifpnW1Z85wH3A6Zn5WnOR9qeIOBb4/cz8pDmrVuTou8XkDOCvM/PyiHgX7qelIuIDtB4imQk8CnyeYp/FvI2pKPyfBH45M18s2tzeKhQ/gfQZWr9gcB/wb2ndMzYhx7hJUZBJkiRNZZP5K0tJkqQpwYJMkiSpYRZkkiRJDbMgkyRJapgFmSRJUsMsyCRNKRExOyJ+t+k4JGlXWJBJmmpmAxZkkiYVCzJJU82VwCER8dOI+HZELG93RMQNEXFyRJwREX8bET+MiIcj4uKOeU6PiB8Xy/95RAwU7X8WEUMR8UDxA5GSNGEsyCRNNecBjxR/HP5PgDMAIuIdwK8DtxXzHQn8DnA4cGpEDEbEr9L6Je4lxfLbgM8W81+QmYPF/L9R/EUHSZoQM6pnkaTJKTP/d0T8aUTMpVV8fScz32z9KTpuz8xnASLiVuDDtP4kyoeAe4p5ZvHWH1n+dEScReu4+W5gMXD/7nw/kqYuCzJJU903gdNp/dHzz3e0j/y7cQkEcH1mnt/ZEREHA78P/KvMfD4irgP2qi1iSdOOX1lKmmpeBvbpmL4O+DJAZq7vaF8aEXMiYhawHLgLuAP4VETsD1D0vwfYF3gFeDEiDgCOr/1dSJpWvEImaUrJzGcj4q6I+Bnwd5l5bkQ8CKwcMeuPge8A84FvZeYQQERcCHw/It4GvAF8MTPvjoj7gIeATbSKN0maMJE58qq9JE0dEfF2YB3wwcx8sWg7AxjMzLObjE2S2vzKUtKUFRGfAB4E/nu7GJOkfuQVMkmSpIZ5hUySJKlhFmSSJEkNsyCTJElqmAWZJElSwyzIJEmSGvb/ATvuFm8Ho6y/AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 720x216 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "\"\"\"\n",
    "Feature list:\n",
    "sbp tobacco ldl adiposity famhist_true famhist_false typea obesity alcohol age\n",
    "\"\"\"\n",
    "plot_feature(data, 'typea')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fit Baseline Model\n",
    "\n",
    "### Evaluation\n",
    "\n",
    "Since we are predicting the label of our classes (heart disease or none), we have a much more intuitive measure of performance: prediction accuracy. To calculate this we will use `sklearn.metrics.accuracy_score` - reference here http://scikit-learn.org/stable/modules/generated/sklearn.metrics.accuracy_score.html."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "def evaluate(model, x_train, y_train, x_test, y_test):\n",
    "    train_preds = model.predict(x_train)\n",
    "    test_preds = model.predict(x_test)\n",
    "    train_acc = metrics.accuracy_score(y_train, train_preds)\n",
    "    test_acc = metrics.accuracy_score(y_test, test_preds)\n",
    "    print('Train accuracy: %s' % train_acc)\n",
    "    print('Test accuracy: %s' % test_acc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Gradient Descent Model\n",
    "\n",
    "Here I will implement a baseline gradient descent model. I will not perform feature selection or tune regularization. You will then need to beat this baseline.\n",
    "\n",
    "The baseline use the `sklearn.linear_model.SGDClassifier` class - reference here http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.SGDClassifier.html#sklearn.linear_model.SGDClassifier. By passing the argument `loss='log'` we get a logistic regression model.\n",
    "\n",
    "I append `bl` to the variables here to mark them as the baseline. Let's also add one more convenient helper function that will split the data, train the model, and return train and test accuracies."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train accuracy: 0.6994219653179191\n",
      "Test accuracy: 0.6982758620689655\n"
     ]
    }
   ],
   "source": [
    "def split_train_evaluate(model, data):\n",
    "    x_train, y_train, x_test, y_test = split(data)\n",
    "    model.fit(x_train, y_train)\n",
    "    evaluate(model, x_train, y_train, x_test, y_test)\n",
    "\n",
    "model_bl = linear_model.SGDClassifier(loss='log', max_iter=10000)\n",
    "split_train_evaluate(model_bl, data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Grid Search\n",
    "\n",
    "What may be difficult about this task is that for the first time during our tutorials we have a number of hyperparameters. This is the common situation in machine learning practice. One approach to dealing with this issue is to define a reasonable set of values for each hyperparameter and then search over all combinations of them using cross validation on the training set. This technique is called **grid search**.\n",
    "\n",
    "We can perform gid search with `sklearn.model_selection.GridSearchCV`, reference here http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html.\n",
    "\n",
    "Let's try it out with a very small search space to show you how it works."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/hanshan/anaconda3/envs/ai/lib/python3.7/site-packages/sklearn/model_selection/_search.py:841: DeprecationWarning: The default of the `iid` parameter will change from True to False in version 0.22 and will be removed in 0.24. This will change numeric results when test-set sizes are unequal.\n",
      "  DeprecationWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=10, error_score='raise-deprecating',\n",
       "       estimator=SGDClassifier(alpha=0.0001, average=False, class_weight=None,\n",
       "       early_stopping=False, epsilon=0.1, eta0=0.0, fit_intercept=True,\n",
       "       l1_ratio=0.15, learning_rate='optimal', loss='log', max_iter=None,\n",
       "       n_iter=None, n_iter_no_change=5, n_jobs=None, penalty='l2',\n",
       "       power_t=0.5, random_state=None, shuffle=True, tol=None,\n",
       "       validation_fraction=0.1, verbose=0, warm_start=False),\n",
       "       fit_params=None, iid='warn', n_jobs=None,\n",
       "       param_grid={'alpha': [0.01, 0.1, 1.0], 'max_iter': [1000, 10000]},\n",
       "       pre_dispatch='2*n_jobs', refit=True, return_train_score=True,\n",
       "       scoring=None, verbose=0)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# this may take a few seconds to run\n",
    "x_train, y_train, x_test, y_test = split(data)\n",
    "grid_search = model_selection.GridSearchCV(\n",
    "    estimator=linear_model.SGDClassifier(loss='log'),\n",
    "    param_grid={'alpha': [0.01, 0.1, 1.],\n",
    "                'max_iter': [1000, 10000]},\n",
    "    cv=10,\n",
    "    return_train_score=True)\n",
    "grid_search.fit(x_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The grid search results come in a dictionary that can be loaded directly into a pandas `DataFrame` for viewing so you can see what the parameter choices did."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>params</th>\n",
       "      <th>mean_train_score</th>\n",
       "      <th>mean_test_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>{'alpha': 0.01, 'max_iter': 1000}</td>\n",
       "      <td>0.669310</td>\n",
       "      <td>0.664740</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>{'alpha': 0.01, 'max_iter': 10000}</td>\n",
       "      <td>0.720621</td>\n",
       "      <td>0.699422</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>{'alpha': 0.1, 'max_iter': 1000}</td>\n",
       "      <td>0.711024</td>\n",
       "      <td>0.708092</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>{'alpha': 0.1, 'max_iter': 10000}</td>\n",
       "      <td>0.727994</td>\n",
       "      <td>0.719653</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>{'alpha': 1.0, 'max_iter': 1000}</td>\n",
       "      <td>0.704560</td>\n",
       "      <td>0.684971</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                               params  mean_train_score  mean_test_score\n",
       "0   {'alpha': 0.01, 'max_iter': 1000}          0.669310         0.664740\n",
       "1  {'alpha': 0.01, 'max_iter': 10000}          0.720621         0.699422\n",
       "2    {'alpha': 0.1, 'max_iter': 1000}          0.711024         0.708092\n",
       "3   {'alpha': 0.1, 'max_iter': 10000}          0.727994         0.719653\n",
       "4    {'alpha': 1.0, 'max_iter': 1000}          0.704560         0.684971"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r = pd.DataFrame(grid_search.cv_results_)\n",
    "# we only want a subset of the columns for a precise summary\n",
    "r[['params', 'mean_train_score', 'mean_test_score']].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The grid seach also picks the best model for you automatically which you can then use with the optimal parameters from the grid search"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'alpha': 0.1, 'max_iter': 10000}\n",
      "Train accuracy: 0.7369942196531792\n",
      "Test accuracy: 0.7586206896551724\n"
     ]
    }
   ],
   "source": [
    "best_model = grid_search.best_estimator_\n",
    "print(grid_search.best_params_)\n",
    "evaluate(best_model, x_train, y_train, x_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So we have already improved our baseline."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Your Turn\n",
    "\n",
    "You can use the grid search code above to find your best model and beat our baseline test accuracy of $72.4$ percent.\n",
    "\n",
    "You will want to examine the parameters in the `SGDClassifier` documentation (http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.SGDClassifier.html#sklearn.linear_model.SGDClassifier) and make your own decisions about what a reasonable search space should look like.\n",
    "\n",
    "Also be aware that if your best value for a parameter is on the edge of your search space, you want to expand the space further in that direction to see if you can keep climbing. For example if my space for $\\alpha$ was `[0.1, 0.5, 1.]`, and the best result came with `1.`, then I should definitely try `2.` and `5.` and so on.\n",
    "\n",
    "Also don't forget feature selection. Return to the section where we were visualizing. Try and apply some $L2$ penalty through the parameters to perform ridge regression if you like."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "Task: Beat the baseline 75.86% test accuracy\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is just a helper function for below (don't edit it)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_feature_set(data, wanted_features):\n",
    "    return data.loc[:, [col in wanted_features for col in data.columns]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Edit this cell to define your own grid search space."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "param_grid = {\n",
    "    'alpha': [0.01, 0.06, 0.1, 0.3,  1.],\n",
    "    'max_iter': [1000, 10000, 100000]\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Replace `data.columns` with a list of your own chose features to perform feature selection - e.g. `['feature1', 'feature2']`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "wanted_features = ['sbp', 'tobacco', 'ldl', 'adiposity', 'famhist_true', 'typea',\n",
    "                   'famhist_false', 'obesity', 'alcohol', 'age', 'chd']  # must have chd!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After defining your feature set and grid space, run the cell below to get your best result. Be patient it may take a while, especially if your grid space is big!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/hanshan/anaconda3/envs/ai/lib/python3.7/site-packages/sklearn/model_selection/_search.py:841: DeprecationWarning: The default of the `iid` parameter will change from True to False in version 0.22 and will be removed in 0.24. This will change numeric results when test-set sizes are unequal.\n",
      "  DeprecationWarning)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'alpha': 0.1, 'max_iter': 10000}\n",
      "Train accuracy: 0.7369942196531792\n",
      "Test accuracy: 0.7586206896551724\n"
     ]
    }
   ],
   "source": [
    "# this may take a few seconds to load\n",
    "my_data = get_feature_set(data, wanted_features)  # feature selection\n",
    "x_train, y_train, x_test, y_test = split(my_data) # splits\n",
    "grid_search = model_selection.GridSearchCV(       # perform grid search\n",
    "    estimator=linear_model.SGDClassifier(loss='log'),\n",
    "    param_grid=param_grid,\n",
    "    cv=10,\n",
    "    return_train_score=True)\n",
    "grid_search.fit(x_train, y_train)\n",
    "best_model = grid_search.best_estimator_\n",
    "print(grid_search.best_params_)\n",
    "evaluate(best_model, x_train, y_train, x_test, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "py37",
   "language": "python",
   "name": "py37"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}