notebooks/Lab5_Logistic_Regression/Supervised Learning - Logistic Regression.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Machine Learning\n",
    "\n",
    "Jay Urbain, PhD\n",
    "\n",
    "References:\n",
    "\n",
    "- Games, G., Witten, D., Hastie, T., and Tibshirani, R. (2013) An Introduction to Statistical Learning\n",
    "with applications in R, www.StatLearning.com, Springer-Verlag, New York. Chapter 4.\n",
    "\n",
    "- [scikit-learn](http://scikit-learn.org/stable/)\n",
    "\n",
    "In this notebook we will be applying logistic regression classification using *scikit-learn* to classify instances as likely or not likely to default using the *Credit Card Default* dataset.\n",
    "\n",
    "Answer **questions** in the empty cell(s) below each question in the notebook. Here are the steps we will follow. \n",
    "\n",
    "1 - Importing the data\n",
    "\n",
    "2 - The Logistic Function\n",
    "\n",
    "3 - Getting, preparing, and visualizing the data\n",
    "\n",
    "4 - Logistic Regression classification\n",
    "\n",
    "5 - Evaluating the model on the training data\n",
    "\n",
    "6 - Summary for applying Logistic Regression with scikit-learn\n",
    "\n",
    "7 - Questions\n",
    "\n",
    "Dataset: **Credit Card Default Data**\n",
    "\n",
    "Description: A simulated data set containing information on ten thousand customers. The aim here is to predict which customers will default on their credit card debt. A data frame with 10000 observations on the following 4 variables:\n",
    "\n",
    "- **default** A factor (binary categorical attribute) with levels (enumerated values) No and Yes indicating whether the customer defaulted on their debt.\n",
    "\n",
    "- **student** A factor with levels No and Yes indicating whether the customer is a student.\n",
    "\n",
    "- **balance** The average balance that the customer has remaining on their credit card after making\n",
    "their monthly payment.\n",
    "\n",
    "- **income** Income of customer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Install the following libraries if you have not already done so. \n",
    "\n",
    "[Searborn statistical](https://stanford.edu/~mwaskom/software/seaborn/) visualization library installation.\n",
    "\n",
    "Use the conda package installing in a terminal window as follows:\n",
    "\n",
    "conda install seaborn\n",
    "    \n",
    "[Statsmodel](http://statsmodels.sourceforge.net/) allows you to explore data, estimate statistical models, and perform statistical tests.\n",
    "\n",
    "Use the conda package installing in a terminal window as follows:\n",
    "\n",
    "conda install statsmodels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Dataset Import\n",
    "import statsmodels.api as sm\n",
    "\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# plots within notebook versus launching a separate window\n",
    "%matplotlib inline\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. The Logistic Function\n",
    "\n",
    "The [Logistic Function](http://en.wikipedia.org/wiki/Logistic_function) takes an input from negative to positive infinity and outputs a value between 0 and 1. \n",
    "\n",
    "The logistic function is defined as:\n",
    "\n",
    "$ g(z)= \\frac{1}{1+e^{-z}}$\n",
    "\n",
    "Where $z=\\beta^TX = \\beta _0 x_{i_0} + ... + \\beta_p x_{ip}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plotting the [Logisitc Function](http://matplotlib.org/users/mathtext.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Figure 1. Logistic Function\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZEAAAESCAYAAAA8BeghAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAH75JREFUeJzt3XmUVeWZ7/HvQyGDgoAyiAyCETUYuWBXwIgDCWiD2pBBW+mY5Y0aOx29t+2YQWO3sc3N6pjRZMXEGLVNvCJyjSZISKNhVBMSioCMEkrUWEILggIyFzz3j/ec1PF4Cqp2nTrvPuf8Pmvttd89VNXDXqfqx3738Jq7IyIikkSH2AWIiEj5UoiIiEhiChEREUlMISIiIokpREREJDGFiIiIJKYQERGRxBQiIiKSmEJEREQS6xi7gPbWu3dvHzJkSOwyRETKytKlS9909z5H2q/iQ2TIkCHU1dXFLkNEpKyY2ast2U/dWSIikphCREREElOIiIhIYgoRERFJTCEiIiKJpSZEzOxBM9tsZqua2W5m9gMzqzezFWZ2VqlrFBGRd0tNiAAPARMPs30SMCwzXQ/8uAQ1iYjIYaTmORF3X2RmQw6zyxTg5x7G811sZj3NrL+7bypJgSJFdNNNNwFw9913R66kzO3fD7t2vXfavx/27Ss8z50OHmzd1NgIhw6Be+EJmrbnt93DcqF2/vfIrs91uKHMm9vWqxc88UTrjmkrpSZEWmAA8FrOckNm3XtCxMyuJ5ytMHjw4JIUJ9Iay5cvj11Ceu3fD//932HasgXeeitM27bB22+H9vbt8M474Y+6RFVOIWIF1hWMX3e/D7gPoLa29jDxLSJRuMObb8LLL8OGDWH++ushOLZubZ+fWVMDnTpB585hftRR0LFjWN+ayQw6ZK4EdOgQluHd6/PbrZlyHW45f1uh/bp0SXasWqGcQqQBGJSzPBDYGKkWEWmN3bth7VpYvRpWroR162DnzpZ//dFHw3HHQc+eoYumZ0/o1g2OOebdU7duYd8uXUJQ5IZGTU37/fuqWDmFyEzgRjObDowBtut6iEhKuYczjMWLw7RmTdM1gEJ69oTBg+HEE+GEE6B//zDv1y9s69y5dLVLq6QmRMzsUWAc0NvMGoCvAkcBuPu9wGzgYqAe2A18Ok6lItKsDRvgmWdg3jzYvLnwPoMGwRlnwCmnwMknw9ChISikLKUmRNx96hG2O3BDicoRkZbaswfmzIGnngohku+kk2D0aBg5MoRHjx6lr1HaTWpCRETKzNat8PjjMGtWuFMq14gRMG4cnH126JqSiqUQEZHW2bEDpk2DJ58Mt+Nm9e8PkybBhAkKjiqiEBGRljl4MDy49tBD4W6rrBEj4LLLYOzYpttapWooRETkyNauhe98B156qWndmWfCddeFEJGqpRARkeY1NoYzj2nTml6tMXAg3HhjuFh+uAfepCooRESksE2b4GtfC2chEJ7uvuoqmDo1PLwngkJERApZtgy++tWmp8pPPhluvz3criuSQyEiIu/21FPw/e+HC+kAH/84/OM/6uxDClKIiEjgDj//ebgGAqH76otfhIsuilqWpJtCRERCgNx7L8yYEZZ79oSvfx2GD49bl6SeQkSk2rnDPffAL34Rlvv2DbfzDhwYty4pCwoRkWr38MNNAXLiifDd74a354q0gEJEpJr98pfwn/8Z2n37wve+F+YiLaR3FIhUqz/8AX7wg9Du0QO+/W0FiLSaQkSkGjU0hAcJ3cOAT3fdFcb5EGklhYhItdm1C267LcwBbrkFTjstbk1SthQiItXEPXRb/eUvYfmqq8K4HyIJKUREqsmcObBgQWiPGQPXXBO1HCl/ChGRatHQEF5nAnDccaEbS2/hlTZSiIhUg4MHwxPoe/eG5VtvDU+li7SRQkSkGjzxBLz4YmhffjnU1satRyqGQkSk0m3aBA88ENoDB4bRCEWKRCEiUsncw2tM9u0LyzffrFe6S1EpREQq2fz5UFcX2hdfDCNHxq1HKo5CRKRS7dsHP/lJaPfsCZ/9bNx6pCIpREQq1YwZsHlzaF93HXTvHrceqUgKEZFKtHUrPPJIaL/vfTBpUtx6pGIpREQq0f33N11Mv+EG6KBfdWkf+mSJVJpXXw2vNwE491wYNSpuPVLRFCIileahh8KtvWZ6JkTaXapCxMwmmtk6M6s3s1sKbB9sZvPNbJmZrTCzi2PUKZJaL73U9ILFCRPgpJOiliOVLzUhYmY1wD3AJGA4MNXMhuft9q/ADHcfBVwJ/Ki0VYqkXHao2w4d4Oqr49YiVSE1IQKMBurdfYO77wemA1Py9nHg2Ey7B7CxhPWJpNv69fD886E9cSIMGBC3HqkKHWMXkGMA8FrOcgMwJm+fO4Cnzex/AccAE0pTmkgZmDYtzGtq4FOfiluLVI00nYkUGtjA85anAg+5+0DgYuBhM3vPv8HMrjezOjOr27JlSzuUKpIyDQ2wcGFojx8PJ5wQtx6pGmkKkQZgUM7yQN7bXXUtMAPA3X8PdAF6538jd7/P3WvdvbZPnz7tVK5Iijz2WLgjC2Dq1Li1SFVJU4gsAYaZ2VAz60S4cD4zb5+/AOMBzOz9hBDRqYZUt23bmp4L+dCHYMiQqOVIdUlNiLh7I3AjMAdYS7gLa7WZ3WlmkzO73Qx8xsxeAB4F/qe753d5iVSXX/wCDhwIbZ2FSIml6cI67j4bmJ237vac9hpgbKnrEkmtffvgqadC+4wz4Mwz49YjVSc1ZyIiksDcubBzZ2hfdlncWqQqKUREypU7PPlkaPfuHd6TJVJiChGRcrVqFdTXh/bkydAxVb3TUiUUIiLlKnsW0rEjXHpp3FqkailERMrR1q2waFFof/jD0KtX3HqkailERMrRr38NBw+G9sc/HrcWqWoKEZFyc+hQCBGAYcPg9NPj1iNVTSEiUm7+9CfYvDm0L7kkbi1S9RQiIuUmexbSqVN42aJIRAoRkXKyfTs891xoX3ABdOsWtx6pegoRkXLyzDPQ2Bja6sqSFFCIiJQLd5idebXcgAEwYkTcekRQiIiUj3Xr4OWXQ/vii8EKjeMmUloKEZFy8cwzYW4GF10UtxaRDIWISDk4eBDmzQvts84KL1wUSQGFiEg5WLoU3n47tC+8MG4tIjkUIiLlINuV1akTnHde3FpEcihERNJuzx549tnQHjsWjj46bj0iORQiImn33HNhGFxQV5akjkJEJO2yXVnHHgsf/GDcWkTyKERE0mzbNqirC+2PfESjF0rqKERE0mzhwvCkOsCECXFrESlAISKSZgsWhHnfvjB8eNRSRApRiIik1datsHJlaF9wgV5zIqmkEBFJq0WLmrqyPvzhuLWINEMhIpJWuV1ZGgJXUkohIpJGuV1Z48apK0tSSyEikka5d2WNGxe1FJHDUYiIpNHChWGurixJOYWISNqoK0vKiEJEJG1yu7J0V5akXKpCxMwmmtk6M6s3s1ua2efvzWyNma02s2mlrlGk3WXvyurXD047LWopIkeSmhfxmFkNcA9wIdAALDGzme6+JmefYcCtwFh3f8vM+sapVqSdvPUWrFoV2nrAUMpAms5ERgP17r7B3fcD04Epeft8BrjH3d8CcPfNJa5RpH397ndNXVnnnx+3FpEWSFOIDABey1luyKzLdSpwqpk9b2aLzWxioW9kZtebWZ2Z1W3ZsqWdyhVpB889F+a9esH73x+3FpEWSFOIFDpv97zljsAwYBwwFbjfzHq+54vc73P3Wnev7dOnT9ELFWkXu3eHsdQhjGDYIU2/niKFpelT2gAMylkeCGwssM+v3P2Au78MrCOEikj5W7IEDhwI7XPPjVuLSAulKUSWAMPMbKiZdQKuBGbm7fNL4MMAZtab0L21oaRVirSXbFdW164walTcWkRaKDUh4u6NwI3AHGAtMMPdV5vZnWY2ObPbHGCrma0B5gNfdPetcSoWKaLGRvj970N7zBjo1CluPSItlOgW38ztuEOA3plVbwKvuPvBthTj7rOB2Xnrbs9pO/D5zCRSOV54AXbtCm11ZUkZaXGImNmxwCeBTwBnA13zdtljZn8AHgemufv2olUpUumefTbMa2rCmYhImThiiJhZL+A24LOE4FhDeIZjA7CV0CXWC3gfMAb4IfBtM/sx8PXsMx0i0gx3eP750B41Crp1i1uPSCu05ExkA7AF+Cow3d1fP9zOZjaQcFH8euBaQsCISHPWrYM33wxtdWVJmWlJiNxACI9DLfmG7t5AOBP5LnBFW4oTqQrZu7IAzjknXh0iCRzx7ix3n5YbIGZ2YaEH/Ap83SF3f7StBYpUvGyInH466OFYKTNJbvGdA7xoZsOLXYxI1WlogFdfDW11ZUkZSvqcSFdgvpmdkb/BzC4xs1+1rSyRKpHblaUQkTKUNES+RHjlyLwCQdITuLRNVYlUi2yIDBwIgwfHrUUkgaQhshOYCLxIOCP5QPFKEqkSjY2wJjNczrnnauwQKUuJX3vi7ruBSYTnRuaZ2ZlFq0qkGmzf3jR2iLqypEy16d1ZOUGyihAkI4pSlUg12J55qYPGDpEy1uYXMLr7HuASYAUwFxjZ1u8pUvEOHYKdO0NbY4dIGUvyAsZ+QGPuCnffY2aXALOAm3nvYFIikmvHDnVlSUVodYi4e8HxZt19r5ldCvwc+Nu2FhbbTTfdxPLly2OXIRVq+YsvQmMj41asgC99SRfVpV2MHDmSu+++u11/RqJXwTfH3fcCf1/M7ylScdzhYGbUhO7dFSBS1lryFt+T3P3VJN+8LV8bW3unt1SxujrGTZgAwIIHHoDx4yMXJJJcS67mrTez+82sxeN1mtlZZvYg8OfkpYlUqOwDhmYaO0TKXku6sy4CvgHUmdl64GnCeOgbgOxYIdnxRD6Y2f8UYCkVcG1EpKhyxw7p1k1jh0jZO2KIuPsC4GwzGw98BriGMBZ6/h1YBuwm3KH1OXefW9xSRSpA7tghPXrErUWkCFp8YT0TCnPNrCNQC5wOZN9bvQVYCyx198ZmvoWI5L5wUSEiFSDJ3VkjgGXuvrjYxYhUvGyIHH00HHVU3FpEiiBJiNQBjZnrIyuAFzLzFZlRDUWkkNyxQ3QWIhUiSYj8A/DPwBjg/TQNgetm9jYhUBYDs9z9+aJUKVIJ1JUlFSjJC3t6EsJjKtCfcCfWJOAJoAdwBiFkFpnZ783s1CLVKlLecscO6dIlbi0iRZIkRG4BvuXuj7n7G+6+wd3nuPvlhFt6DwKjCSEzGHjWzAYWr2SRMrRt27vHDhGpEElCpC+wq9CGzB1cjwG3uvsMQph0Av41cYUileB3v9MLF6UiJQmR5cAnzJp94c8qMg8ZuvvrwAOE7i6R6pXtyurVC4YPj1uLSBElCZH/A5wDPGxmPQtsP593P4i4Djghwc8RqQy7d8PSpaE9dqxeuCgVJcmr4Geb2WeAHwKTzewpYBnhWsh5wBTg/+Z8SX9gexFqFSlPf/xjGE8d1JUlFSfRq+Dd/UEzexb4CiE0pmY3AY8D/xsg0+V1BbCy7aWKlKlsV1bXrnDWWXFrESmyxGNyuvt6d/800Bs4GfgAcKy7X+Hu2TOPjsBNwK0t+Z5mNtHM1plZvZndcpj9LjMzN7PapPWLlERjIyzOvNxhzBg9pS4Vp82DUrn7IeCVZrYdAJ5pyfcxsxrgHuBCoAFYYmYz3X1N3n7dCWc6f2hD2SKlsXw57MrczKiuLKlAic9E2sFooD7z3Ml+YDqhqyzf14BvAntLWZxIItmurI4d4eyz49Yi0g7SFCIDgNdylhsy6/4qMzDWIHefVcrCRBJxh2efDe2RI+GYY+LWI9IO0hQihe57/OutwmbWAfgecPMRv5HZ9WZWZ2Z1W7ZsKWKJIq2walV4Uh3gggvi1iLSTtIUIg3AoJzlgcDGnOXuhIv3C8zsFeBsYGahi+vufp+717p7bZ8+ffI3i5TGwoVhbqbrIVKx0hQiS4BhZjbUzDoBVwIzsxvdfbu793b3Ie4+hPCm4MnuXhenXJHDcIdFi0J75EjoWei5XJHyl5oQyYyIeCMwhzBK4gx3X21md5rZ5LjVibTS2rWQ7UpVV5ZUsDbf4ltM7j4bmJ237vZm9h1XippEElmwIMzN4LzzopYi0p5ScyYiUjHcm66HnHkmHHdc3HpE2pFCRKTY1q2DzZtDW11ZUuEUIiLFlj0LATj//Hh1iJSAQkSkmHK7ss44A3r3jluPSDtTiIgUU309bNoU2uPGRS1FpBQUIiLFNG9eU1tdWVIFFCIixeLeFCJnngl9+8atR6QEFCIixbJyZdNdWePHx61FpEQUIiLFMndumNfU6HqIVA2FiEgxNDbC/PmhXVsLPXrErUekRBQiIsVQVwc7d4b2hAlxaxEpIYWISDH89rdh3rkzjB0btxaRElKIiLTV3r1Nw+COHQtdu8atR6SEFCIibfX887BvX2jrriypMgoRkbb6r/8K8+7dYfTouLWIlJhCRKQtNm+GpUtD+8ILoWOqhugRaXcKEZG2ePrp8KQ6wMSJcWsRiUAhIpKUe1NX1sknwymnxK1HJAKFiEhSK1fC66+H9qRJYShckSqjEBFJKnsWUlOjBwylailERJLYswcWLAjtD30IevaMWo5ILAoRkSTmzg1BAqErS6RKKUREWssdfvWr0O7TB8aMiVuPSEQKEZHWWrs2DIMLcOml4ZqISJVSiIi0VvYspKYGLrkkbi0ikSlERFpjx46mcUPOOw+OPz5uPSKRKUREWuM3v4EDB0J78uS4tYikgEJEpKUOHWrqyho8GEaOjFuPSAooRERa6vnnYdOm0J4yRU+oi6AQEWm5xx4L8+7d9WyISIZCRKQl1qyB1atDe/JkjV4okpGqEDGziWa2zszqzeyWAts/b2ZrzGyFmc01s5Ni1ClVaMaMMO/YET72sbi1iKRIakLEzGqAe4BJwHBgqpkNz9ttGVDr7iOAx4FvlrZKqUqbNsGiRaE9YYJu6xXJkZoQAUYD9e6+wd33A9OBKbk7uPt8d9+dWVwMDCxxjVKNpk1rGnjq8svj1iKSMmkKkQHAaznLDZl1zbkW+E2hDWZ2vZnVmVndli1biliiVJ033gjPhgCcc04YfEpE/ipNIVLofkkvuKPZVUAt8K1C2939PnevdffaPn36FLFEqTqPPAIHD4b21VfHrUUkhTrGLiBHAzAoZ3kgsDF/JzObANwGXODu+0pUm1SjzZubzkLOPhtOPTVuPSIplKYzkSXAMDMbamadgCuBmbk7mNko4CfAZHffHKFGqSbTpkFjY2jrLESkoNSEiLs3AjcCc4C1wAx3X21md5pZ9iVF3wK6Af/PzJab2cxmvp1I2zQ0wKxZoT16NJx+etx6RFIqTd1ZuPtsYHbeuttz2hrIWkrj/vvDtRAzuO662NWIpFZqzkREUmP1ali4MLQvvBCGDYtbj0iKKUREcrnDvfeG9lFHwTXXxK1HJOUUIiK55s+HVatC+xOfgH794tYjknIKEZGsXbvgnntCu2dP+OQn49YjUgYUIiJZDz4I27aF9j/9E3TrFrcekTKgEBEBWL8ennwytEeMCBfUReSIFCIijY1w113honpNDfzLv2jUQpEWUoiI/Oxn8NJLoT11KgwZErUckXKiEJHqtmZNeMkiwPvep9ebiLSSQkSq1+7d8B//EbqxOnaEr3wlzEWkxRQiUp3c4ZvfDO/IArj2Wo0VIpKAQkSq0+OPN73aZPRouOKKuPWIlCmFiFSf5cubXm3Srx/cdpvuxhJJSCEi1eXVV+Hf/g0OHQrXP/793+HYY2NXJVK2FCJSPbZuhS9/Gd55Jyx/8Ytw2mlxaxIpcwoRqQ47doQAeeONsHzNNXDRRXFrEqkAChGpfDt2wM03Nz1QeMklcNVVcWsSqRAKEalsb78Nn/881NeH5fHj9VoTkSJSiEjleu01+Nznms5Axo+HW28N78cSkaJQiEhleuEFuOEG2LQpLE+cGJ5IV4CIFJXe8SCVxR2mT4f77w+38QJ8+tPwqU+pC0ukHShEpHJs3Qrf/jYsXhyWjzoKvvAF3YUl0o4UIlL+3GHOnDC0bfYZkBNPhDvugGHDopYmUukUIlLe6uvhRz+CZcua1n3kI+GOrGOOiVeXSJVQiEh5euMNeOihcAbiHtYdf3y4fXfs2KiliVQThYiUlw0b4NFHYd68pgvnHTrA5MnhKfTu3ePWJ1JlFCKSfvv2wbPPwuzZ7+62AjjnHPjsZ2HQoDi1iVQ5hYik0/79sHRpCI9Fi2DXrqZtZjBuXBgPXRfORaJSiEg6uMMrr4SxPpYtg7o62LPn3fv06BFu1/3oR8PdVyISnUJESs8dtm2DP/8Z1q8P06pV4T1X+Tp1gtraEB5jx2oMdJGUSdVvpJlNBL4P1AD3u/s38rZ3Bn4O/A2wFbjC3V8pdZ3SAocOwVtvhQcAN26E118P77JqaAjT9u3Nf+1xx8FZZ8F558EHPwhdu5aubhFpldSEiJnVAPcAFwINwBIzm+nua3J2uxZ4y91PMbMrgbsADY7dng4eDBe29+8P3Us7dzZN77wTpuzytm0hNN58M7Szd08dSf/+YXCokSNh1KhwkVyvKBEpC6kJEWA0UO/uGwDMbDowBcgNkSnAHZn248APzczcsw8KFNGCBeEPKITul+yPyJ3nrm/LuvzvmeRnHjoU6m1sDPPcqdC63OnAgRAU+/bB3r0hMPbuDcuNjW0+lH/VqxcMHBimIUPg1FPhlFOgW7fi/QwRKak0hcgA4LWc5QZgTHP7uHujmW0HjgfeLHo13/hG+CMqR1ZTE57P6NUrPPDXu/e75/36heA4+ujYlYpIkaUpRAr1X+SfYbRkH8zseuB6gMGDB7e9snJhFv6g19SEC9DZ9uHWZdd36QKdO4epS5dwQTt/XefOISy6dw9nD9l2587qfmqlkSNHxi5BpCjSFCINQO4TYwOBjc3s02BmHYEewLb8b+Tu9wH3AdTW1ibr6vrpT0M3kVnTH8jceXPtI63L/WPbknUt/Tk1NfpDXkbuvvvu2CWIFEWaQmQJMMzMhgKvA1cC/5C3z0zgauD3wGXAvHa5HgJ6AlpEpAVSEyKZaxw3AnMIt/g+6O6rzexOoM7dZwIPAA+bWT3hDOTKeBWLiEhqQgTA3WcDs/PW3Z7T3gtcXuq6RESkMI2xLiIiiSlEREQkMYWIiIgkphAREZHEFCIiIpKYtddjFmlhZluAVxN+eW/a45Uqbae6Wkd1tV5aa1NdrdOWuk5y9z5H2qniQ6QtzKzO3Wtj15FPdbWO6mq9tNamulqnFHWpO0tERBJTiIiISGIKkcO7L3YBzVBdraO6Wi+ttamu1mn3unRNREREEtOZiIiIJKYQKcDMvmVmL5rZCjN70sx65my71czqzWydmf1tieu63MxWm9khM6vNWT/EzPaY2fLMdG8a6spsi3a88uq4w8xezzlGF8eqJVPPxMwxqTezW2LWksvMXjGzlZljVBexjgfNbLOZrcpZd5yZPWNm6zPzXimpK/pny8wGmdl8M1ub+V3858z69j9m7q4pbwIuAjpm2ncBd2Xaw4EXgM7AUOAloKaEdb0fOA1YANTmrB8CrIp4vJqrK+rxyqvxDuALsT9bmVpqMsfiZKBT5hgNj11XprZXgN4pqON84KzczzXwTeCWTPuW7O9lCuqK/tkC+gNnZdrdgT9nfv/a/ZjpTKQAd3/a3Rszi4sJoywCTAGmu/s+d38ZqAdGl7Cute6+rlQ/r6UOU1fU45Vio4F6d9/g7vuB6YRjJRnuvoj3jlo6BfhZpv0z4KMlLYpm64rO3Te5+58y7Z3AWmAAJThmCpEjuwb4TaY9AHgtZ1tDZl0aDDWzZWa20MzOi11MRtqO142ZLsoHY3SF5EjbccnlwNNmttTMro9dTJ5+7r4Jwh9NoG/kenKl5bOFmQ0BRgF/oATHLFWDUpWSmf0WOKHAptvc/VeZfW4DGoFHsl9WYP+i3t7WkroK2AQMdvetZvY3wC/N7Ax33xG5rnY/Xu/6YYepEfgx8LXMz/8a8B3CfxBiKOlxaaWx7r7RzPoCz5jZi5n/fUvzUvPZMrNuwC+Am9x9h1mhj1pxVW2IuPuEw203s6uBS4HxnulQJPyPMXfw9YHAxlLW1czX7AP2ZdpLzewl4FSgaBdGk9RFCY5XrpbWaGY/BWa1Vx0tUNLj0hruvjEz32xmTxK63tISIm+YWX9332Rm/YHNsQsCcPc3su2Yny0zO4oQII+4+xOZ1e1+zNSdVYCZTQS+DEx29905m2YCV5pZZzMbCgwD/hijxlxm1sfMajLtkwl1bYhbFZCi45X5Bcr6GLCquX1LYAkwzMyGmlkn4ErCsYrKzI4xs+7ZNuEGk5jHKd9M4OpM+2qguTPgkkrDZ8vCKccDwFp3/27OpvY/ZjHvKEjrRLgA/BqwPDPdm7PtNsKdNeuASSWu62OE/8XuA94A5mTWfwJYTbjL50/A36WhrtjHK6/Gh4GVwIrML1b/yJ+xiwl30LxE6BKMVktOTSdnPkMvZD5P0eoCHiV00x7IfLauBY4H5gLrM/PjUlJX9M8WcC6hO21Fzt+ti0txzPTEuoiIJKbuLBERSUwhIiIiiSlEREQkMYWIiIgkphAREZHEFCIiIpKYQkRERBJTiIiISGIKERERSUwhIlJCZtbXzPwwU7TRH0WSqNq3+IpE0gh8Km9dN8IIdPsIL2cUKRt6d5ZIRJm35f6GMMTwR9x9ZeSSRFpF3VkikWQCZDYhQMYrQKQcqTtLJAIzOxr4NXAG4QxkReSSRBLRmYhIiWUCZBbwAcIZiAJEypbORERKyMy6Ak8B/4MQIC9ELkmkTXQmIlIiOQEyEpjg7ssjlyTSZro7S6REzGwGcDnwXWBZ3ua33X1W6asSaRuFiEgJmJkB24Huzewyy93/roQliRSFQkRERBLTNREREUlMISIiIokpREREJDGFiIiIJKYQERGRxBQiIiKSmEJEREQSU4iIiEhiChEREUlMISIiIon9fxrue/vfvtwqAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# generate 1000 samples between -20 and 20\n",
    "x = np.linspace(-20, 20, 1000)\n",
    "beta = [0, 0.5]\n",
    "y = np.exp(beta[0] + beta[1]*x) / (1 + np.exp(beta[0] + beta[1]*x))\n",
    "\n",
    "# Plotting\n",
    "plt.plot(x, y, 'r', alpha=0.75, linewidth=2.5)\n",
    "plt.plot([0,0], [0, 1], 'k')\n",
    "plt.plot([-20,20], [0.5, 0.5], 'k')\n",
    "plt.xlabel(r'$z$', fontsize='xx-large')\n",
    "plt.ylabel(r'$g(z)$', fontsize='xx-large')\n",
    "\n",
    "print ('Figure 1. Logistic Function')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can interpret $g(z)$  as the probability of positive classification. Values $>= 0.5$ are classified as positive (1: likely to default), and values $< 0.5$ are classified as negative (0: not likely to default)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Getting, preparing, and visualizing the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>default</th>\n",
       "      <th>student</th>\n",
       "      <th>balance</th>\n",
       "      <th>income</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>No</td>\n",
       "      <td>729.526495</td>\n",
       "      <td>44361.62507</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>Yes</td>\n",
       "      <td>817.180407</td>\n",
       "      <td>12106.13470</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>No</td>\n",
       "      <td>1073.549164</td>\n",
       "      <td>31767.13895</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>No</td>\n",
       "      <td>529.250605</td>\n",
       "      <td>35704.49394</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>No</td>\n",
       "      <td>785.655883</td>\n",
       "      <td>38463.49588</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   default student      balance       income\n",
       "0        0      No   729.526495  44361.62507\n",
       "1        0     Yes   817.180407  12106.13470\n",
       "2        0      No  1073.549164  31767.13895\n",
       "3        0      No   529.250605  35704.49394\n",
       "4        0      No   785.655883  38463.49588"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Read Default.csv into a DataFrame\n",
    "d = pd.read_csv('data/Default.csv')\n",
    "d.head()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "            default       balance        income\n",
      "count  10000.000000  10000.000000  10000.000000\n",
      "mean       0.033300    835.374886  33516.981876\n",
      "std        0.179428    483.714985  13336.639563\n",
      "min        0.000000      0.000000    771.967729\n",
      "25%        0.000000    481.731105  21340.462905\n",
      "50%        0.000000    823.636973  34552.644800\n",
      "75%        0.000000   1166.308387  43807.729275\n",
      "max        1.000000   2654.322576  73554.233500\n"
     ]
    }
   ],
   "source": [
    "print (d.describe())\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Prepare the data for classification\n",
    "\n",
    "sklearn.model_selection.train_test_split(*arrays, **options):  \n",
    "http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.series.Series'>\n"
     ]
    }
   ],
   "source": [
    "# Convert classification attribute to numeric binary value for binary classification\n",
    "\n",
    "d.loc[d.student == 'Yes', 'student'] = 1\n",
    "d.loc[d.student == 'No', 'student'] = 0\n",
    "\n",
    "# Split the data into training and test sets\n",
    "X = d[['balance','student','income']]\n",
    "y = d.default\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=1)\n",
    "print( type(y_train))\n",
    "# Convert the split data lists back into a DataFrame\n",
    "train = pd.DataFrame(data=X_train, columns=['balance','student','income'])\n",
    "train['default'] = y_train\n",
    "test = pd.DataFrame(data=X_test, columns=['balance','student','income'])\n",
    "test['default'] = y_test\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Visualize the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Figure 2. Histogram of all variables\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAEICAYAAABWJCMKAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvDW2N/gAAIABJREFUeJzt3X+YXFWd5/H3R34ECIEkRrIhiTRoxMFFJUSCDy4TjBsCuIRR1LisJAwuOoLCPsxI0JkBQcYwzyMCMguiRAJCAkZYIqLQAr2MOyZAEAgQMQ0GExIJkBCSoGjwu3/c03Slu6q7uqrrV9/P63nq6Vvnnrp1zu1T9a17zrn3KiIwM7P8eUujC2BmZo3hAGBmllMOAGZmOeUAYGaWUw4AZmY55QBgZpZTDgB1IGmNpI9U8LoOSZ+tRZnMGknS9ZK+Xka+gyX9StJWSV+q8j0vlPSDarYx1Oza6AKYmfXhy0BHRBw2mBuV1Ab8FtgtInYM5rZbiY8AzKyZHQA82ehCDFUOAPXzAUlPSdos6fuS9pA0StKdkl5M6XdKmlDsxZLeIek+SS9LeknSTZJGFqxfI+nvJT0uaYukWyTtUbB+lqRHJb0q6RlJM1P6vpKuk7RB0vOSvi5pl9rvDssTSYdJeiR15dwCFLbNj6a2+Yqk/5D03pR+H3AMcJWkbZLeJemE1CX0qqS1ki4s2M40Set6vG+p7tcH0t9X0rY/ONh1bgUOAPVzCnAs8A7gXcA/ku3/75P9ynk78AfgqhKvF/ANYH/gr4CJwIU98nwSmAkcCLwXmAsg6QjgBuAfgJHA0cCa9JqFwA7gncBhwAzA4w42aCTtDvwf4EZgNPBD4ONp3WRgAfA54K3Ad4ClkoZFxIeBfwfOioi9I+I3wHbgVLJ2fALwd5JOqqBYR6e/I9O2f1lxBVuYA0D9XBURayNiE3AJ8OmIeDkifhQRr0XE1pT+18VeHBGdEdEeEa9HxIvAZUXyXhkR69N7/Bh4f0o/HViQXv+XiHg+In4taSxwHHBORGyPiI3At4DZg157y7Mjgd2AyyPizxGxBHgorfufwHciYnlEvBERC4HX02t6iYiOiFiZ2vHjwCJKfGasfx4Erp+1BcvPAftL2ovsC3cmMCqtGyFpl4h4o/DFkvYDrgT+CzCCLHhv7vEevy9Yfo3saAGyo4W7ipTpALIP5gZJXWlv6VFWs2rtDzwfO1958rn09wBgjqQvFqzbne62uxNJU4H5wH9O+YaRHVFYBXwEUD8TC5bfDqwHzgUOBqZGxD50H5aK3r4BBPDelPd/lMhXzFqyrqdi6a8DYyJiZHrsExHvKXO7ZuXYAIxXwa8Mss8AZG3wkoL2NzIi9oqIRSW2dTOwFJgYEfsC19D9OdgO7NWVMY1lva3EdnwZZBwA6ulMSRMkjQa+AtxC9kv+D2QDUaOBC/p4/QhgW8o7nqw/v1zXAadJmi7pLZLGS3p3RGwA7gG+KWmftO4dknxIbYPpl2TjTF+StKukjwFHpHXfBT4vaaoyw9NA74gS2xoBbIqIP6axrf9esO43wB7p9buRjbMNK7GdF4G/AAdVWbeW5gBQPzeTfdk+mx5fBy4H9gReApYBP+vj9V8DJgNbgJ8At5X7xhHxIHAaWXfTFuD/kh16QzagtjvwFFmX0hJgXLnbNutPRPwJ+BjZpITNwKdI7TciHiYbB7gqretM+Ur5AnCRpK3APwO3FrzPlrT+e8DzZEcE64ptJCJeIxtz+39p9lHRMYehTr4hjJlZPvkIwMwspxwAzMxyygHAzCynHADMzHKqqU8EGzNmTLS1tRVdt337doYPH17fAtXIUKoLNF99VqxY8VJElJoP3nRKtftm26+N4v3QrdS+KLfNN3UAaGtr4+GHHy66rqOjg2nTptW3QDUylOoCzVcfSc/1n6t5lGr3zbZfG8X7oVupfVFum3cXkJlZTjkAmJnllAOAmVlONfUYQCtrm/eTsvOee+gO5s77CWvmn1DDElmrW/n8FuYOoF0BblPWJx8BmJnllAOAmVlOOQCYmeWUA4CZWU55ELgfAxnMNTNrJT4CMDPLKQcAsyIkjZS0RNKvJa2S9EFJoyW1S1qd/o5KeSXpSkmdkh6XNLlgO3NS/tWS5jSuRma9OQCYFXcF8LOIeDfwPmAVMA+4NyImAfem5wDHAZPS4wzgaoCC+zxPJbsH7gVdQcOsGTgAmPUgaR/gaOA6yO5pGxGvALOAhSnbQuCktDwLuCEyy4CRksYBxwLtEbEpIjYD7cDMOlbFrE8eBDbr7SDgReD7kt4HrADOBsZGxAaAiNggab+UfzywtuD161JaqfReJJ1BdvTA2LFj6ejo6JVn7J7ZWeMDUWw7rW7btm1Dsl6VqHZfOACY9bYrMBn4YkQsl3QF3d09xahIWvSR3jsx4lrgWoApU6ZEsUv8fvumO/jmyoF9ZNec0ns7rc6Xg+5W7b5wF5BZb+uAdRGxPD1fQhYQXkhdO6S/GwvyTyx4/QRgfR/pZk3BAcCsh4j4PbBW0sEpaTrwFLAU6JrJMwe4Iy0vBU5Ns4GOBLakrqK7gRmSRqXB3xkpzawpuAvIrLgvAjdJ2h14FjiN7AfTrZJOB34HfCLlvQs4HugEXkt5iYhNki4GHkr5LoqITfWrglnfHADMioiIR4EpRVZNL5I3gDNLbGcBsGBwS2c2OBwAmkgll53w9d7NrFIeAzAzyykHADOznHIAMDPLKQcAM7OccgAwM8upfgOApAWSNkp6oiDNl8U1M2tx5RwBXE/vKxj6srhmZi2u3/MAIuIBSW09kmcB09LyQqADOI+Cy+ICy9JNNcalvO1dZ0FK6ros7qKqazAAvr2jmVm3Sk8Ea+hlcaGyy6AO9FK69VLJZX67NONlcX25XrPWMNhnAtflsrhQ2WVQ5zbpEcC5h+4Y8GV+uzTj5X59uV6z1lDpLCBfFtfMrMVVGgB8WVwzsxbXb7+DpEVkg7hjJK0jm80zH18W18yspZUzC+jTJVb5srhmZi2sZS8HvfL5LU07qGtm1gp8KQgzs5xyADArQdIukn4l6c70/EBJy9PlTG5Jt4tE0rD0vDOtbyvYxvkp/WlJxzamJmbFOQCYlXY2sKrg+aXAt9IlUDYDp6f004HNEfFO4FspH5IOAWYD7yE78/1/S9qlTmU365cDgFkRkiYAJwDfS88FfBhYkrIsBE5Ky7PSc9L66Sn/LGBxRLweEb8lmx13RH1qYNY/BwCz4i4Hvgz8JT1/K/BKRHRds6PwciZvXuokrd+S8pd9CRSzRmjZWUBmtSLpo8DGiFghaVpXcpGs0c+6si+BUs41sCq5ZtRQvCaTrzXVrdp94QBg1ttRwImSjgf2APYhOyIYKWnX9Cu/8HImXZc6WSdpV2BfYBMDuARKOdfA+vZNdwz4mlHNeK2oavlaU92q3RfuAjLrISLOj4gJEdFGNoh7X0ScAtwPnJyy9bwEStelUU5O+SOlz06zhA4ku0/Gg3Wqhlm/fARgVr7zgMWSvg78CrgupV8H3Cipk+yX/2yAiHhS0q3AU8AO4MyIeKP+xTYrzgHArA8R0UF2wyMi4lmKzOKJiD/SfT2snusuAS6pXQnNKucA0OIqucvZmvkn1KAkZtZqPAZgZpZTDgBmZjnlAGBmllMOAGZmOeUAYGaWUw4AZmY55QBgZpZTDgBmZjnlAGBmllMOAGZmOeUAYGaWU1UFAElrJK2U9Kikh1PaaEnt6cbZ7ZJGpXRJujLdIPtxSZMHowJmZlaZwTgCOCYi3h8RU9LzecC96cbZ96bnAMeRXQ99Etmdj64ehPc2M7MK1aILqPAG2T1vnH1DZJaR3V1pXA3e38zMylDt5aADuEdSAN9Jt7UbGxEbACJig6T9Ut5SN8jeULjBcu6NCpXdH7VZ1bsutb6fqu/ZatYaqg0AR0XE+vQl3y7p133kLesG2eXcGxUquz9qszr30B11rUut7xPre7aatYaquoAiYn36uxG4nexuSS90de2kvxtT9rJvkG1mZrVXcQCQNFzSiK5lYAbwBDvfILvnjbNPTbOBjgS2dHUVmTUTSRMl3S9plaQnJZ2d0gc8w03SnJR/taQ5pd7TrBGq6XcYC9wuqWs7N0fEzyQ9BNwq6XTgd3TfK/Uu4HigE3gNOK2K9zarpR3AuRHxSPqRs0JSOzCXbIbbfEnzyGa4ncfOM9ymks1wmyppNHABMIWsu3OFpKURsbnuNTIrouIAkG6Q/b4i6S8D04ukB3Bmpe9nVi/pyLRrIsNWSavIJizMAqalbAvJbhZ/HgUz3IBlkrpmuE0D2iNiE0AKIjOBRXWrjFkfhsYoqlmNSGoDDgOWM/AZbqXSi71Pv7PfKpktNhRnY3mWWbdq94UDgFkJkvYGfgScExGvpu7OolmLpEUf6b0Ty5j9VsnMt1rP+GoEzzLrVu2+8LWAzIqQtBvZl/9NEXFbSh7oDDfPfLOm5gBg1oOyn/rXAasi4rKCVQOd4XY3MEPSqDRjaEZKM2sK7gIy6+0o4DPASkmPprSvAPMZwAy3iNgk6WLgoZTvoq4BYbNm4ACQQ23zflLR69bMP2GQS9KcIuIXFO+/hwHOcIuIBcCCwSud2eBxF5CZWU45AJiZ5ZQDgJlZTjkAmJnllAOAmVlOOQCYmeWUA4CZWU45AJiZ5ZQDgJlZTjkAmJnllAOAmVlOOQCYmeWULwZnZSv3InLnHrqDuSlvXi4gZ9aKfARgZpZTDgBmZjnlAGBmllMOAGZmOeUAYGaWU3WfBSRpJnAFsAvwvYiYX+8yWP1UcvvJoTZzyG3e+lPpbVqvnzm8qvet6xGApF2AfwOOAw4BPi3pkHqWwaye3OatmdX7COAIoDMingWQtBiYBTxV53JYExtiRw1u89a06h0AxgNrC56vA6YWZpB0BnBGerpN0tMltjUGeGnQS9gAXxpCdYHG1EeX9rn6gDoVo5h+2zyU3e4HvF/72S+takh9XqpxzKUl90VZbb7eAUBF0mKnJxHXAtf2uyHp4YiYMlgFa6ShVBcYevWpUr9tHspr996vGe+HbtXui3rPAloHTCx4PgFYX+cymNWT27w1rXoHgIeASZIOlLQ7MBtYWucymNWT27w1rbp2AUXEDklnAXeTTYlbEBFPVri5fruJWshQqgsMvfpUzG2+JrwfulW1LxTRqzvSzMxywGcCm5nllAOAmVlOtVwAkDRT0tOSOiXNa3R5SpG0QNJGSU8UpI2W1C5pdfo7KqVL0pWpTo9Lmlzwmjkp/2pJcxpUl4mS7pe0StKTks5u5fo0u/7auKRhkm5J65dLaqt/KWuvjP0wV9KLkh5Nj882opy1Vuy7pMf6kp+3fkVEyzzIBtGeAQ4CdgceAw5pdLlKlPVoYDLwREHavwLz0vI84NK0fDzwU7I540cCy1P6aODZ9HdUWh7VgLqMAyan5RHAb8gua9CS9WnmRzltHPgCcE1ang3c0uhyN2g/zAWuanRZ67Aven2X9Fhf9PNWzqPVjgDePK0+Iv4EdJ1W33Qi4gFgU4/kWcDCtLwQOKkg/YbILANGShoHHAu0R8SmiNgMtAMza1/6nUXEhoh4JC1vBVaRneHakvVpcuW08cL9vgSYLqnYCWetrGU+67VW4rukUKnPW79aLQAUO61+fIPKUomxEbEBsi9VYL+UXqpeTVff1N1wGLCcIVCfJlTOPnozT0TsALYAb61L6eqn3Lby8dTtsUTSxCLr86Diz1WrBYCyTqtvQaXq1VT1lbQ38CPgnIh4ta+sRdKarj5Nqpx9lIf9WE4dfwy0RcR7gZ/TfVSUNxW3h1YLAK1+Wv2ekj4GkA7RNqb0UvVqmvpK2o3sy/+miLgtJb/QdajZavVpYuXsozfzSNoV2Je+uwhaUZ/7QdIa4LCIeD0lfRc4vG6lay4Vf65aLQC0+mn1VwLvSstzgDvS8lLg1DSafySwJXWp3A3MkDQqzbCZkdLqKvUvXwesiojLClYtJasHtFB9mlw5bbxwv58M3BdpNLAVSLpQ0g/6yVbOfhhdsHwi2dhULcrS7Ep93vpV9zuCVSMG97T6mpK0CJgGjJG0DrgAmA/cKul04HfAJ1L2u8hG8juB14DTACJik6SLyT4MABdFRCN+6R0FfAZYKenRlPYVWrc+TatUG5d0EfBwRCwlC8Y3Suok++U/u3Elro3+9kPK9jeSLgB2kO2HuQ0pbI2V+C7ZDSAirqHE560sjZ7ilKcHsAb4CHAhcCtwA7AVeBKYUpBvInAb8CLwMmmqG9kR2z8Cz5F1t9wA7JvWtZH1+51GNiC0Gfg88AHgceAVekyZA/6W7FfTZrIP2gGN3kd+DK0HcB7wfGrnTwMnAH8C/gxsAx5L+dYAHyl43YXADwqefya1+5eBrxbmT5+LeWTTRl9On63RaV3X52IO2Y+Ul4CvpnUzi5UlT49W6wIaSk4km9o2kuwQ7ip48xaCd5I19jay0fzF6TVz0+MYsvnRe3e9rsBUYBLwKeBysg/LR4D3AJ+U9NfpfU4i+xX/MeBtwL8Diwa5jpZjkg4GzgI+EBEjyKYB/xr4F7JzF/aOiPeVsZ1DgKvJgsD+ZDOeJhRk+RLZFOS/Tus3k92Gs9CHgIOB6cA/S/qriPjZQMsy1DgANM4vIuKuiHgDuBHoanxHkDXif4iI7RHxx4j4RVp3CnBZZHOjtwHnA7PTQGCXi9Nr7gG2A4siYmNEPE/2JX9Yyvc54BsRsSqyqYT/ArxfUiPvnmVDyxvAMOAQSbtFxJqIeKaC7ZwM3BkRD0Q26PtPwF8K1n+O7Ff9urT+QuDkHp+Lr0XEHyLiMbKTynL3ZV+MA0Dj/L5g+TVgj9RgJwLPpS/lnvYnOzLo8hzZOM7YgrQXCpb/UOT53mn5AOAKSa9IeoWsD1V4Xr4NkojoBM4h+0LeKGmxpP0r2NT+FMxzj4jtZF09XQ4Abi9oy6vIgk/h56Ln521vzAGgCa0F3t7j10uX9ex8r8+3kw2AvVAkbznv87mIGFnw2DMi/qOCbZkVFRE3R8SHyNptAJdSfI76dmCvguf/qWB5AwXTHCXtxc4nvq0FjuvRlvdIR739FrHMqgxJDgDN50GyBj9f0nBJe0g6Kq1bBPyvNDVub7r7L4sdLfTnGuB8Se8BkLSvpE/08xqzskk6WNKHJQ0D/kh2BPoG2Q+WNkmF3z+PknVn7iZpClm3T5clwEclfShNCb2Inb+7rgEu6eq+lPQ2SeVeNqJYWXIjl5VuZmlM4L8B7ySbtbCObEAXYAHZeMEDwG/JPlRfrPB9bif7NbZY0qvAE8BxVRXebGfDyKYKv0TWBbMf2cSDH6b1L0t6JC3/E/AOsgHcrwE3d20ksqneZ6a0DSnPuoL3uYJsIsU9krYCy8gmQ5SjWFlyw3cEMzPLKR8BmJnllAOAmVlOOQCYmeWUA4CZWU419cXgxowZE21tbb3St2/fzvDhw+tfoCq4zPXTs9wrVqx4KSLe1sAiDchQave14P3QrdS+KLvNN/piRH09Dj/88Cjm/vvvL5rezFzm+ulZbrKraDa8PZf7GErtvha8H7qV2hfltnl3AZmZ5ZQDgJlZTjkAmJnlVFMPAlv/2ub9ZMCvWTP/hBqUxGpt5fNbmDvA/7f/19YXHwGYmeWUA4CZWU45AJiZ5ZQDgJlZTjkAmJnllAOAmVlOOQCYmeWUA4CZWU75RLB+VHKiFfgEHDNrfj4CMDPLKQcAM7OccgAwM8spBwAzs5xyADArQdIukn4l6c70/EBJyyWtlnSLpN1T+rD0vDOtbyvYxvkp/WlJxzamJmbFeRZQE6l0xpHVzNnAKmCf9PxS4FsRsVjSNcDpwNXp7+aIeKek2SnfpyQdAswG3gPsD/xc0rsi4o16V8SsGB8BmBUhaQJwAvC99FzAh4ElKctC4KS0PCs9J62fnvLPAhZHxOsR8VugEziiPjUw65+PAMyKuxz4MjAiPX8r8EpE7EjP1wHj0/J4YC1AROyQtCXlHw8sK9hm4Wt2IukM4AyAsWPH0tHR0SvP2D3h3EN39ErvS7HttLpt27YNyXpVotp94QBg1oOkjwIbI2KFpGldyUWyRj/r+nrNzokR1wLXAkyZMiWmTZvWK8+3b7qDb64c2Ed2zSm9t9PqOjo6KLZ/8qjafeEAYNbbUcCJko4H9iAbA7gcGClp13QUMAFYn/KvAyYC6yTtCuwLbCpI71L4GrOG8xiAWQ8RcX5ETIiINrJB3Psi4hTgfuDklG0OcEdaXpqek9bfFxGR0menWUIHApOAB+tUDbN++QjArHznAYslfR34FXBdSr8OuFFSJ9kv/9kAEfGkpFuBp4AdwJmeAWTNxAHArA8R0QF0pOVnKTKLJyL+CHyixOsvAS6pXQnNKtdvF5CkBZI2SnqiIG20pPZ0Qky7pFEpXZKuTCe+PC5pcsFr5qT8qyXNKfZeZmZWP+WMAVwPzOyRNg+4NyImAfem5wDHkfVzTiKb0nY1ZAEDuACYSvYL6oKuoGFmZo3RbxdQRDxQeGp7MguYlpYXkh0in5fSb0gDYMskjZQ0LuVtj4hNAJLayYLKoqprYANW7hnH5x66g7kFeX2PA7OhpdIxgLERsQEgIjZI2i+lv3lCTNJ14kup9F7KOSGmnieCDPTEmy49y1dOmSt9r1rpeeJRq5x84xOFzMoz2IPAdTkhpp4ngsyt9I5gPU7AKafMlb5XrZx76I6dTjxqlZOKfKKQWXkqPQ/ghdS1Q/q7MaWXOvHFJ8SYmTWZSgNA4YkvPU+IOTXNBjoS2JK6iu4GZkgalQZ/Z6Q0MzNrkH67gCQtIhvEHSNpHdlsnvnArZJOB35H9xzou4Djya56+BpwGkBEbJJ0MfBQyndR14CwmZk1RjmzgD5dYtX0InkDOLPEdhYACwZUOjMzqxlfC8jMLKccAMzMcsoBwMwsp3wxuBrpebZtz7NqzcwazUcAZmY55QBgZpZTDgBmZjnlAGBmllMOAGZmOeUAYGaWUw4AZmY55QBgZpZTDgBmZjnlAGBmllMOAGZmOeUAYGaWUw4AZmY55QBgZpZTDgBmPUiaKOl+SaskPSnp7JQ+WlK7pNXp76iULklXSuqU9LikyQXbmpPyr5Y0p1F1MismV/cD6HmNfrMSdgDnRsQjkkYAKyS1A3OBeyNivqR5wDzgPOA4YFJ6TAWuBqZKGg1cAEwBIm1naURsrnuNzIrwEYBZDxGxISIeSctbgVXAeGAWsDBlWwiclJZnATdEZhkwUtI44FigPSI2pS/9dmBmHati1qdcHQGYDZSkNuAwYDkwNiI2QBYkJO2Xso0H1ha8bF1KK5VutpNKeyeunzm8qvd1ADArQdLewI+AcyLiVUklsxZJiz7Si73XGcAZAGPHjqWjo6NXnrF7ZrcWHYhi22l127ZtG3L1Guj/tUu1+6KqACBpDbAVeAPYERFTUr/nLUAbsAb4ZERsVvbpuQI4HngNmNt1mG3WbCTtRvblf1NE3JaSX5A0Lv36HwdsTOnrgIkFL58ArE/p03qkdxR7v4i4FrgWYMqUKTFt2rReeb590x18c+XAPrJrTum9nVbX0dFBsf3Tyiq9X/j1M4dXtS8GYwzgmIh4f0RMSc/nkQ2UTQLuTc9h54GyM8gGysyaTvqxch2wKiIuK1i1FOiayTMHuKMg/dQ0G+hIYEvqKrobmCFpVJoxNCOlmTWFWnQBzaL7V89Csl8851EwUAYskzSy69dUDcpgVo2jgM8AKyU9mtK+AswHbpV0OvA74BNp3V1kR7adZEe3pwFExCZJFwMPpXwXRcSm+lTBrH/VBoAA7pEUwHfSYexAB8p2CgDl9IVW2u9VaT/bYKik/7bRepa5Vfpdq+0XjYhfULz/HmB6kfwBnFliWwuABRUXxqyGqg0AR0XE+vQl3y7p133kLWtArJy+0Er7ACvtZxsM5x66Y8D9t43Ws8yt0p88FPuIzWqhqjGAiFif/m4EbgeOIA2UAZQ5UGZmZg1Q8U9SScOBt0TE1rQ8A7iI7oGy+fQeKDtL0mKysyW3uP+/tVQyV3nN/BNqUBIzGwzV9EmMBW5Pc6N3BW6OiJ9JeogBDJSZmVljVBwAIuJZ4H1F0l9mgANlZmZWf74WkJlZTjkAmJnllAOAmVlOOQCYmeWUA4CZWU45AJiZ5ZQDgJlZTjkAmJnllAOAmVlOOQCYmeWUA4CZWU45AJiZ5ZQDgJlZTjkAmJnllAOAmVlOOQCYmeWUA4CZWU45AJiZ5VQ19wQ265dvJG/WvHwEYGaWUw4AZmY55QBgZpZTDgBmZjlV9wAgaaakpyV1SppX7/c3qze3eWtWdZ0FJGkX4N+A/wqsAx6StDQinhrotiqZXWKtYSjNHBrMNm822Op9BHAE0BkRz0bEn4DFwKw6l8GsntzmrWnV+zyA8cDagufrgKmFGSSdAZyRnm6T9HSR7YwBXqpJCWvkSy5zTenSnZ72LPcBdS3Mzvpt81C7dt9jvwwVLdMua+2YS0vui7LafL0DgIqkxU5PIq4Fru1zI9LDETFlMAtWay5z/TRZuftt8zB0230teD90q3Zf1LsLaB0wseD5BGB9nctgVk9u89a06h0AHgImSTpQ0u7AbGBpnctgVk9u89a06toFFBE7JJ0F3A3sAiyIiCcr2FSfh8pNymWun6Yp9yC2eWiiejWY90O3qvaFInp1R5qZWQ74TGAzs5xyADAzy6mWCgCNPqVe0kRJ90taJelJSWen9AslPS/p0fQ4vuA156fyPi3p2P7qkgYLl0taLemWNHA4GGVfI2llKt/DKW20pPb0Xu2SRqV0Sboyle1xSZMLtjMn5V8taU5B+uFp+53ptcWmPw6kvAcX7M9HJb0q6ZxW2NfV6q+dSxqWytuZyt9W/1LWXhn7Ya6kFwvawmcbUc5ak7RA0kZJT5RYX/Lz2q+IaIkH2QDaM8BBwO7AY8AhdS7DOGByWh4B/AY4BLgQ+Psi+Q9J5RwGHJjKv0tfdQFuBWan5WuAvxuksq8BxvRI+1dgXlqeB1yalo8Hfko2h/1IYHlKHw08m/6OSsuj0roHgQ+m1/wUOG6Q//e/Jzu5pen3da3bOfAF4Jq0PBu4pdHlbtB+mAtc1eiy1mFfHA0XczsJAAADR0lEQVRMBp4osb7o57WcRysdATT8lPqI2BARj6TlrcAqsjM9S5kFLI6I1yPit0AnWT2K1iX9av4wsCS9fiFwUm1q82b5FhZ5r1nADZFZBoyUNA44FmiPiE0RsRloB2amdftExC8ja5E3DHK5pwPPRMRz/dSlmfd1ucpp54X/tyXA9GqPuJpQwz/vzSIiHgA29ZGl1Oe1X60UAIqdUt/Xl29NpcPuw4DlKemsdPi1oKsrhdJlLpX+VuCViNjRI30wBHCPpBXKLjsAMDYiNkAW3ID9Kiz3+LTcM32wzAYWFTxv9n1djXLa+Zt5Uvm3kNVnKCn38/7x1BaWSJpYZH0eVPzd2EoBoKxT6utB0t7Aj4BzIuJV4GrgHcD7gQ3AN7uyFnl5VJA+GI6KiMnAccCZko7uI2/TlDv1y58I/DAltcK+rkY55WrWsg+mcur4Y6AtIt4L/Jzuo6K8qbg9tFIAaIpT6iXtRvblf1NE3AYQES9ExBsR8Rfgu2SHr1C6zKXSXyI7fNu1R3rVImJ9+rsRuD2V8YWuQ8X0d2OF5V6XlnumD4bjgEci4oVU/qbf11Uqp52/mSeVf1/67iJoRf3uh4h4OSJeT0+/Cxxep7I1m4q/G1spADT8lPrUz3odsCoiLitIL+xv+xuga7R+KTA7zdo4EJhENlhatC6p//x+4OT0+jnAHYNQ7uGSRnQtAzNSGZem9+j5XkuBU9PsgiOBLamL6G5ghqRRqetlBnB3WrdV0pFpH506GOVOPk1B90+z7+tBUE47L/y/nQzcl+ozlPS7H3q0hRPJxuTyqNTntX+NHuEe4Gj48WQzb54BvtqA9/8Q2aHV48Cj6XE8cCOwMqUvBcYVvOarqbxPUzAzplRdyGY9PEg2iPlDYNgglPsgslkUjwFPdr0fWb/xvcDq9Hd0ShfZTUyeSfWaUrCtv01l6wROK0ifQvZl/AxwFeks8yrLvRfwMrBvQVpT7+tatXPgIuDEtLxHKm9nKv9BjS5zg/bDN1J7fowsmL+70WWu0X5YRNbd+WeyX/unA58HPp/Wl/y89vfwpSDMzHKqlbqAzMxsEDkAmJnllAOAmVlOOQCYmeWUA4CZWU45AJiZ5ZQDgJlZTv1/HD8qkjRMCCIAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 4 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Create a histogram of all variables\n",
    "# Histograms show the distribution of each predictor variable\n",
    "train.hist()\n",
    "print('Figure 2. Histogram of all variables')\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Q0) Create a scatter plot of the income vs. balance** Answer in the cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a scatter plot of the income vs. balance\n",
    "# Scatterplots are useful for visualizing the association between two variables\n",
    "\n",
    "# your work here\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We can mark defaults (cassification attribute) with a different color and symbol\n",
    "train_nd = d[d.default == 0]\n",
    "train_d = d[d.default == 1]\n",
    "\n",
    "plt.figure()\n",
    "plt.scatter(train_nd.balance, train_nd.income, alpha = .5, marker='+', c= 'b')\n",
    "\n",
    "plt.scatter(train_d.balance, train_d.income, marker='o', \n",
    "            edgecolors = 'r', facecolors = 'none')\n",
    "plt.legend( ('no default', 'default'), loc='upper right')\n",
    "plt.xlabel(\"Balance $\")\n",
    "plt.ylabel(\"Income $\")\n",
    "plt.title ('Figure 4. Scatterplot of incomve vs. balance - default, non-default')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Q1) What precentage of individuals in the dataset default?** Answer in the cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Q2) From 'Figure 4. Scatterplot of income vs. balance - default, non-default':**\n",
    "\n",
    "Can you identfy an income threshold for default vs. non-default? If so, what is it?\n",
    "\n",
    "Can you identfy a balance threshold for default vs. non-default? If so, what is it?\n",
    "\n",
    "Which individual attribute, i.e., income or balance is likely to be the more accurate predictor? \n",
    "\n",
    "Provide your answers in the cell below.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Logistic Regression classification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Fit (learn) a logistic regression model on the balance variable\n",
    "balance = LogisticRegression()\n",
    "balance.fit(train[['balance']], y_train)\n",
    "\n",
    "# Is the beta_1 value associated with balance significant?\n",
    "B1 = balance.coef_[0][0]\n",
    "B0 = balance.intercept_[0]\n",
    "print ('B1: ', B1, ' B0: ', B0)\n",
    "print ('e^B1: ', np.exp(B1))\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\beta_1$ is significant.\n",
    "\n",
    "Logistic function: $\\dfrac{g(z)}{1-g(z)} = \\dfrac{e^{\\beta_0+\\beta_1x}}{1-e^{\\beta_0+\\beta_1x}}$\n",
    "\n",
    "The log-odds can be defined as the log odds of positive classification:\n",
    "\n",
    "log odds = $log\\big(\\dfrac{e^{\\beta_0+\\beta_1x}}{1-e^{\\beta_0+\\beta_1x}}\\big) = \\beta_0+\\beta_1x$\n",
    "\n",
    "Increasing $X$ by one unit changes the log odds by $\\beta_1$, or equivalently \n",
    "it multiplies the odds by $e^{\\beta_1}$. \n",
    "\n",
    "Because the relationship between $f(X)$ and $X$ is not a straight line, $\\beta_1$ does not correspond to the change \n",
    "in $f(X)$ associated with a one-unit increase in $X$. The amount that \n",
    "$f(X)$ changes due to a one-unit change in $X$ will depend on the current \n",
    "value of $X$.\n",
    "\n",
    "In our example, ${\\beta_1} ~= 0.0043$\n",
    "\n",
    "Log-odds: \n",
    "If you increase $x$ by 1, you increase the log-odds by $0.0043*1$. \n",
    "If you increase $x$ by 800, you increase the log-odds by 0.0043*800 = 3.44\n",
    "\n",
    "To get the change in $f(X)$, if you increase $x$ by 1, you multiply the odds by $e^{0.0043}$. \n",
    "If you increase $x$ by 800, you mutliply the odds by $e^{0.0043*800} = 31.187$, not $800 * e^{0.0043}$\n",
    "\n",
    "[Log-odds](https://en.wikipedia.org/wiki/Logit)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Prediction using the *balance* variable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Predict the probability of default for someone with a balance of $1.2k and $2.5k\n",
    "prob = balance.predict(pd.DataFrame({'balance': [1200, 2500]}))\n",
    "\n",
    "# What does beta mean? Every unit increase in balance corresponds to a beta increase in y. \n",
    "x = np.linspace(test.balance.min(), test.balance.max(),500)\n",
    "beta = [B0,B1]\n",
    "\n",
    "y = np.exp(beta[0] + beta[1]*x) / (1 + np.exp(beta[0] + beta[1]*x))\n",
    "odds = np.exp(beta[0] + beta[1]*x)\n",
    "log_odds = beta[0] + beta[1]*x\n",
    "\n",
    "plt.figure(figsize=(7, 8))\n",
    "plt.subplot(311)\n",
    "plt.plot(x, y, 'r', linewidth=2)\n",
    "plt.ylabel('Probability')\n",
    "plt.text(500, 0.7, r'$\\frac{e^{\\beta_o + \\beta_1x}}{1+e^{\\beta_o + \\beta_1x}}$', fontsize=25)\n",
    "\n",
    "plt.subplot(312)\n",
    "plt.plot(x, odds, 'k', linewidth=2)\n",
    "plt.ylabel('Odds')\n",
    "plt.text(500, 10, r'$e^{\\beta_o + \\beta_1x}$', fontsize=20)\n",
    "\n",
    "plt.subplot(313)\n",
    "plt.plot(x, log_odds, 'c', linewidth=2)\n",
    "plt.ylabel('Log(Odds)')\n",
    "plt.xlabel('x')\n",
    "plt.text(500, 1, r'$\\beta_o + \\beta_1x$', fontsize=15)\n",
    "print ('Figure 5. Probability, Odds, and Log(odds) of balance - default, non-default')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot the fitted logistic function overtop of the data points\n",
    "plt.figure()\n",
    "plt.scatter(test.balance, test.default, alpha=0.1)\n",
    "plt.plot(x, y, 'r', linewidth=2)\n",
    "plt.xlabel(\"Balance\")\n",
    "plt.ylabel(\"Probability of Default\")\n",
    "plt.ylim([-0.05,1.05]); plt.xlim([0, 2800])\n",
    "plt.plot([1200, 2500], prob, 'ko')\n",
    "print ('Figure 6. The fitted logistic model for predicting default from balance')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. Evaluating the model on the training data\n",
    "\n",
    "Accuracy measures the overall correctness of classification. \n",
    "\n",
    "Accuracy = $\\dfrac{TP+TN}{TP+TN+FP+FN}$\n",
    "\n",
    "Sensitivity (also called the true positive rate, or recall) measures the proportion of positives that are correctly identified as such. E.g., people who have cancer.\n",
    "\n",
    "Sensitivity = $\\dfrac{TP}{TP+FN}$\n",
    "\n",
    "Specificity (also called the true negative rate) measures the proportion of negatives that are correctly identified as such. E.g., people who do not have cancer.\n",
    "\n",
    "Specificity = $\\dfrac{TN}{TN+FP}$\n",
    "\n",
    "[Sensititivy and specificity](https://en.wikipedia.org/wiki/Sensitivity_and_specificity)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create predictions using the balance model on the test set\n",
    "test['pred_class'] = balance.predict(test[['balance']])\n",
    "\n",
    "# Accuracy\n",
    "accuracy = sum(test.pred_class == test.default) / float(len(test.default))\n",
    "print ('Accuracy: ', accuracy)\n",
    "\n",
    "# Sensitivity\n",
    "# For those who did default, how many did it predict correctly? \n",
    "test_d = test[test.default == 1]\n",
    "sensitivity = sum(test_d.pred_class == test_d.default) / float(len(test_d.default))\n",
    "print ('Sensitivity: ', sensitivity)\n",
    "\n",
    "# Specificity\n",
    "# For those who didn't default, how many did it predict correctly?\n",
    "test_nd = test[test.default == 0]\n",
    "specificity = sum(test_nd.pred_class == test_nd.default) / float(len(test_nd.default))\n",
    "print ('Specificity: ', specificity)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Class imbalance can result in accuracy measures that are missleading. How does our overall classification accuracy compare to the non-default rate?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "null = 1 - sum(d.default) / float(len(d.default))\n",
    "print ('Classification accuracy vs. non-default rate: ', null)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What this means is that you could have just guessed not going to default, and would be correct 96.67% of the time."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6. Multivariate Logistic Regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Fit (learn) a logistic regression model on the balance variable\n",
    "balance_student = LogisticRegression()\n",
    "balance_student.fit(train[['balance','student']], y_train)\n",
    "\n",
    "B2 = balance_student.coef_[0][1]\n",
    "B1 = balance_student.coef_[0][0]\n",
    "B0 = balance_student.intercept_[0]\n",
    "print ('B0: ', B0, 'B1: ', B1, 'B2: ', B2)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Q3) For a given income level, who are more likely to default, students or non-students?**\n",
    "\n",
    "*Hint: look at the value of the coefficients for the balance_student model created in Step 6. $B0$ is the intercept, $B1$ is the coefficent for income, and $B2$ is the coefficient for student (1=student). Think about the meaning of the coefficients.*\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Q4) Which combination of predictors: student, balance, or income has the highest predictive accuracy?**\n",
    "\n",
    "To answer this question, we will use a feature selection technique called forward selection.\n",
    "\n",
    "- Build 3 separate models for each predictor using the 70/30 split training/test set created above. See *step 6, Multivariate Logisitc Regression* for an example. Of the three, select the most predicitve univariate model. For example, income.\n",
    "\n",
    "- Next build 2 models: income+balance, and income+student. If neither of these models perform better than the previous income model, stop.\n",
    "\n",
    "- Else build 1 model with all three predictors. \n",
    "\n",
    "Show your work, and summarize your results.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7. Summary for applying Logistic Regression with scikit-learn\n",
    "\n",
    "Create LogisticRegression model:\n",
    "\n",
    "*model = LogisticRegression()*\n",
    "\n",
    "Fit a model to our data:\n",
    "\n",
    "*model.fit(X,Y)*\n",
    "\n",
    "Check our accuracy:\n",
    "\n",
    "*model.score(X,Y)*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:tensorflow]",
   "language": "python",
   "name": "conda-env-tensorflow-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}