Predict_card_approvals.ipynb

{"nbformat_minor":2,"cells":[{"source":"## 1. Credit card applications\n<p>Commercial banks receive <em>a lot</em> of applications for credit cards. Many of them get rejected for many reasons, like high loan balances, low income levels, or too many inquiries on an individual's credit report, for example. Manually analyzing these applications is mundane, error-prone, and time-consuming (and time is money!). Luckily, this task can be automated with the power of machine learning and pretty much every commercial bank does so nowadays. In this notebook, we will build an automatic credit card approval predictor using machine learning techniques, just like the real banks do.</p>\n<p><img src=\"https://assets.datacamp.com/production/project_558/img/credit_card.jpg\" alt=\"Credit card being held in hand\"></p>\n<p>We'll use the <a href=\"http://archive.ics.uci.edu/ml/datasets/credit+approval\">Credit Card Approval dataset</a> from the UCI Machine Learning Repository. The structure of this notebook is as follows:</p>\n<ul>\n<li>First, we will start off by loading and viewing the dataset.</li>\n<li>We will see that the dataset has a mixture of both numerical and non-numerical features, that it contains values from different ranges, plus that it contains a number of missing entries.</li>\n<li>We will have to preprocess the dataset to ensure the machine learning model we choose can make good predictions.</li>\n<li>After our data is in good shape, we will do some exploratory data analysis to build our intuitions.</li>\n<li>Finally, we will build a machine learning model that can predict if an individual's application for a credit card will be accepted.</li>\n</ul>\n<p>First, loading and viewing the dataset. We find that since this data is confidential, the contributor of the dataset has anonymized the feature names.</p>","cell_type":"markdown","metadata":{"run_control":{"frozen":true},"editable":false,"tags":["context"],"deletable":false,"dc":{"key":"3"}}},{"source":"# Import pandas\n# ... YOUR CODE FOR TASK 1 ...\nimport pandas as pd\n# Load dataset\ncc_apps = pd.read_csv('datasets/cc_approvals.data',header = None)\n\n# Inspect data\n# ... YOUR CODE FOR TASK 1 ...\ncc_apps.head()","cell_type":"code","execution_count":186,"metadata":{"tags":["sample_code"],"dc":{"key":"3"},"trusted":true},"outputs":[{"data":{"text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>0</th>\n      <th>1</th>\n      <th>2</th>\n      <th>3</th>\n      <th>4</th>\n      <th>5</th>\n      <th>6</th>\n      <th>7</th>\n      <th>8</th>\n      <th>9</th>\n      <th>10</th>\n      <th>11</th>\n      <th>12</th>\n      <th>13</th>\n      <th>14</th>\n      <th>15</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>b</td>\n      <td>30.83</td>\n      <td>0.000</td>\n      <td>u</td>\n      <td>g</td>\n      <td>w</td>\n      <td>v</td>\n      <td>1.25</td>\n      <td>t</td>\n      <td>t</td>\n      <td>1</td>\n      <td>f</td>\n      <td>g</td>\n      <td>00202</td>\n      <td>0</td>\n      <td>+</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>a</td>\n      <td>58.67</td>\n      <td>4.460</td>\n      <td>u</td>\n      <td>g</td>\n      <td>q</td>\n      <td>h</td>\n      <td>3.04</td>\n      <td>t</td>\n      <td>t</td>\n      <td>6</td>\n      <td>f</td>\n      <td>g</td>\n      <td>00043</td>\n      <td>560</td>\n      <td>+</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>a</td>\n      <td>24.50</td>\n      <td>0.500</td>\n      <td>u</td>\n      <td>g</td>\n      <td>q</td>\n      <td>h</td>\n      <td>1.50</td>\n      <td>t</td>\n      <td>f</td>\n      <td>0</td>\n      <td>f</td>\n      <td>g</td>\n      <td>00280</td>\n      <td>824</td>\n      <td>+</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>b</td>\n      <td>27.83</td>\n      <td>1.540</td>\n      <td>u</td>\n      <td>g</td>\n      <td>w</td>\n      <td>v</td>\n      <td>3.75</td>\n      <td>t</td>\n      <td>t</td>\n      <td>5</td>\n      <td>t</td>\n      <td>g</td>\n      <td>00100</td>\n      <td>3</td>\n      <td>+</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>b</td>\n      <td>20.17</td>\n      <td>5.625</td>\n      <td>u</td>\n      <td>g</td>\n      <td>w</td>\n      <td>v</td>\n      <td>1.71</td>\n      <td>t</td>\n      <td>f</td>\n      <td>0</td>\n      <td>f</td>\n      <td>s</td>\n      <td>00120</td>\n      <td>0</td>\n      <td>+</td>\n    </tr>\n  </tbody>\n</table>\n</div>","text/plain":"  0      1      2  3  4  5  6     7  8  9   10 11 12     13   14 15\n0  b  30.83  0.000  u  g  w  v  1.25  t  t   1  f  g  00202    0  +\n1  a  58.67  4.460  u  g  q  h  3.04  t  t   6  f  g  00043  560  +\n2  a  24.50  0.500  u  g  q  h  1.50  t  f   0  f  g  00280  824  +\n3  b  27.83  1.540  u  g  w  v  3.75  t  t   5  t  g  00100    3  +\n4  b  20.17  5.625  u  g  w  v  1.71  t  f   0  f  s  00120    0  +"},"output_type":"execute_result","metadata":{},"execution_count":186}]},{"source":"## 2. Inspecting the applications\n<p>The output may appear a bit confusing at its first sight, but let's try to figure out the most important features of a credit card application. The features of this dataset have been anonymized to protect the privacy, but <a href=\"http://rstudio-pubs-static.s3.amazonaws.com/73039_9946de135c0a49daa7a0a9eda4a67a72.html\">this blog</a> gives us a pretty good overview of the probable features. The probable features in a typical credit card application are <code>Gender</code>, <code>Age</code>, <code>Debt</code>, <code>Married</code>, <code>BankCustomer</code>, <code>EducationLevel</code>, <code>Ethnicity</code>, <code>YearsEmployed</code>, <code>PriorDefault</code>, <code>Employed</code>, <code>CreditScore</code>, <code>DriversLicense</code>, <code>Citizen</code>, <code>ZipCode</code>, <code>Income</code> and finally the <code>ApprovalStatus</code>. This gives us a pretty good starting point, and we can map these features with respect to the columns in the output.   </p>\n<p>As we can see from our first glance at the data, the dataset has a mixture of numerical and non-numerical features. This can be fixed with some preprocessing, but before we do that, let's learn about the dataset a bit more to see if there are other dataset issues that need to be fixed.</p>","cell_type":"markdown","metadata":{"run_control":{"frozen":true},"editable":false,"tags":["context"],"deletable":false,"dc":{"key":"10"}}},{"source":"# Print summary statistics\ncc_apps_description = cc_apps.describe()\nprint(cc_apps_description)\n\nprint(\"\\n\")\n\n# Print DataFrame information\ncc_apps_info = cc_apps.info()\nprint(cc_apps_info)\n\nprint(\"\\n\")\n\n# Inspect missing values in the dataset\n# ... YOUR CODE FOR TASK 2 ...\ncc_apps.tail()","cell_type":"code","execution_count":188,"metadata":{"tags":["sample_code"],"dc":{"key":"10"},"trusted":true},"outputs":[{"text":"               2           7          10             14\ncount  690.000000  690.000000  690.00000     690.000000\nmean     4.758725    2.223406    2.40000    1017.385507\nstd      4.978163    3.346513    4.86294    5210.102598\nmin      0.000000    0.000000    0.00000       0.000000\n25%      1.000000    0.165000    0.00000       0.000000\n50%      2.750000    1.000000    0.00000       5.000000\n75%      7.207500    2.625000    3.00000     395.500000\nmax     28.000000   28.500000   67.00000  100000.000000\n\n\n<class 'pandas.core.frame.DataFrame'>\nRangeIndex: 690 entries, 0 to 689\nData columns (total 16 columns):\n0     690 non-null object\n1     690 non-null object\n2     690 non-null float64\n3     690 non-null object\n4     690 non-null object\n5     690 non-null object\n6     690 non-null object\n7     690 non-null float64\n8     690 non-null object\n9     690 non-null object\n10    690 non-null int64\n11    690 non-null object\n12    690 non-null object\n13    690 non-null object\n14    690 non-null int64\n15    690 non-null object\ndtypes: float64(2), int64(2), object(12)\nmemory usage: 86.3+ KB\nNone\n\n\n","output_type":"stream","name":"stdout"},{"data":{"text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>0</th>\n      <th>1</th>\n      <th>2</th>\n      <th>3</th>\n      <th>4</th>\n      <th>5</th>\n      <th>6</th>\n      <th>7</th>\n      <th>8</th>\n      <th>9</th>\n      <th>10</th>\n      <th>11</th>\n      <th>12</th>\n      <th>13</th>\n      <th>14</th>\n      <th>15</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>685</th>\n      <td>b</td>\n      <td>21.08</td>\n      <td>10.085</td>\n      <td>y</td>\n      <td>p</td>\n      <td>e</td>\n      <td>h</td>\n      <td>1.25</td>\n      <td>f</td>\n      <td>f</td>\n      <td>0</td>\n      <td>f</td>\n      <td>g</td>\n      <td>00260</td>\n      <td>0</td>\n      <td>-</td>\n    </tr>\n    <tr>\n      <th>686</th>\n      <td>a</td>\n      <td>22.67</td>\n      <td>0.750</td>\n      <td>u</td>\n      <td>g</td>\n      <td>c</td>\n      <td>v</td>\n      <td>2.00</td>\n      <td>f</td>\n      <td>t</td>\n      <td>2</td>\n      <td>t</td>\n      <td>g</td>\n      <td>00200</td>\n      <td>394</td>\n      <td>-</td>\n    </tr>\n    <tr>\n      <th>687</th>\n      <td>a</td>\n      <td>25.25</td>\n      <td>13.500</td>\n      <td>y</td>\n      <td>p</td>\n      <td>ff</td>\n      <td>ff</td>\n      <td>2.00</td>\n      <td>f</td>\n      <td>t</td>\n      <td>1</td>\n      <td>t</td>\n      <td>g</td>\n      <td>00200</td>\n      <td>1</td>\n      <td>-</td>\n    </tr>\n    <tr>\n      <th>688</th>\n      <td>b</td>\n      <td>17.92</td>\n      <td>0.205</td>\n      <td>u</td>\n      <td>g</td>\n      <td>aa</td>\n      <td>v</td>\n      <td>0.04</td>\n      <td>f</td>\n      <td>f</td>\n      <td>0</td>\n      <td>f</td>\n      <td>g</td>\n      <td>00280</td>\n      <td>750</td>\n      <td>-</td>\n    </tr>\n    <tr>\n      <th>689</th>\n      <td>b</td>\n      <td>35.00</td>\n      <td>3.375</td>\n      <td>u</td>\n      <td>g</td>\n      <td>c</td>\n      <td>h</td>\n      <td>8.29</td>\n      <td>f</td>\n      <td>f</td>\n      <td>0</td>\n      <td>t</td>\n      <td>g</td>\n      <td>00000</td>\n      <td>0</td>\n      <td>-</td>\n    </tr>\n  </tbody>\n</table>\n</div>","text/plain":"    0      1       2  3  4   5   6     7  8  9   10 11 12     13   14 15\n685  b  21.08  10.085  y  p   e   h  1.25  f  f   0  f  g  00260    0  -\n686  a  22.67   0.750  u  g   c   v  2.00  f  t   2  t  g  00200  394  -\n687  a  25.25  13.500  y  p  ff  ff  2.00  f  t   1  t  g  00200    1  -\n688  b  17.92   0.205  u  g  aa   v  0.04  f  f   0  f  g  00280  750  -\n689  b  35.00   3.375  u  g   c   h  8.29  f  f   0  t  g  00000    0  -"},"output_type":"execute_result","metadata":{},"execution_count":188}]},{"source":"## 3. Handling the missing values (part i)\n<p>We've uncovered some issues that will affect the performance of our machine learning model(s) if they go unchanged:</p>\n<ul>\n<li>Our dataset contains both numeric and non-numeric data (specifically data that are of <code>float64</code>, <code>int64</code> and <code>object</code> types). Specifically, the features 2, 7, 10 and 14 contain numeric values (of types float64, float64, int64 and int64 respectively) and all the other features contain non-numeric values.</li>\n<li>The dataset also contains values from several ranges. Some features have a value range of 0 - 28, some have a range of 2 - 67, and some have a range of 1017 - 100000. Apart from these, we can get useful statistical information (like <code>mean</code>, <code>max</code>, and <code>min</code>) about the features that have numerical values. </li>\n<li>Finally, the dataset has missing values, which we'll take care of in this task. The missing values in the dataset are labeled with '?', which can be seen in the last cell's output.</li>\n</ul>\n<p>Now, let's temporarily replace these missing value question marks with NaN.</p>","cell_type":"markdown","metadata":{"run_control":{"frozen":true},"editable":false,"tags":["context"],"deletable":false,"dc":{"key":"17"}}},{"source":"# Import numpy\n# ... YOUR CODE FOR TASK 3 ...\nimport numpy as np\n# Inspect missing values in the dataset\nprint(cc_apps.tail(17))\n\n# Replace the '?'s with NaN\ncc_apps = cc_apps.replace('?',np.nan)\n\n# Inspect the missing values again\n# ... YOUR CODE FOR TASK 3 ...\nprint(cc_apps.tail(17))","cell_type":"code","execution_count":190,"metadata":{"tags":["sample_code"],"dc":{"key":"17"},"trusted":true},"outputs":[{"text":"    0      1       2  3  4   5   6      7  8  9   10 11 12     13   14 15\n673  ?  29.50   2.000  y  p   e   h  2.000  f  f   0  f  g  00256   17  -\n674  a  37.33   2.500  u  g   i   h  0.210  f  f   0  f  g  00260  246  -\n675  a  41.58   1.040  u  g  aa   v  0.665  f  f   0  f  g  00240  237  -\n676  a  30.58  10.665  u  g   q   h  0.085  f  t  12  t  g  00129    3  -\n677  b  19.42   7.250  u  g   m   v  0.040  f  t   1  f  g  00100    1  -\n678  a  17.92  10.210  u  g  ff  ff  0.000  f  f   0  f  g  00000   50  -\n679  a  20.08   1.250  u  g   c   v  0.000  f  f   0  f  g  00000    0  -\n680  b  19.50   0.290  u  g   k   v  0.290  f  f   0  f  g  00280  364  -\n681  b  27.83   1.000  y  p   d   h  3.000  f  f   0  f  g  00176  537  -\n682  b  17.08   3.290  u  g   i   v  0.335  f  f   0  t  g  00140    2  -\n683  b  36.42   0.750  y  p   d   v  0.585  f  f   0  f  g  00240    3  -\n684  b  40.58   3.290  u  g   m   v  3.500  f  f   0  t  s  00400    0  -\n685  b  21.08  10.085  y  p   e   h  1.250  f  f   0  f  g  00260    0  -\n686  a  22.67   0.750  u  g   c   v  2.000  f  t   2  t  g  00200  394  -\n687  a  25.25  13.500  y  p  ff  ff  2.000  f  t   1  t  g  00200    1  -\n688  b  17.92   0.205  u  g  aa   v  0.040  f  f   0  f  g  00280  750  -\n689  b  35.00   3.375  u  g   c   h  8.290  f  f   0  t  g  00000    0  -\n      0      1       2  3  4   5   6      7  8  9   10 11 12     13   14 15\n673  NaN  29.50   2.000  y  p   e   h  2.000  f  f   0  f  g  00256   17  -\n674    a  37.33   2.500  u  g   i   h  0.210  f  f   0  f  g  00260  246  -\n675    a  41.58   1.040  u  g  aa   v  0.665  f  f   0  f  g  00240  237  -\n676    a  30.58  10.665  u  g   q   h  0.085  f  t  12  t  g  00129    3  -\n677    b  19.42   7.250  u  g   m   v  0.040  f  t   1  f  g  00100    1  -\n678    a  17.92  10.210  u  g  ff  ff  0.000  f  f   0  f  g  00000   50  -\n679    a  20.08   1.250  u  g   c   v  0.000  f  f   0  f  g  00000    0  -\n680    b  19.50   0.290  u  g   k   v  0.290  f  f   0  f  g  00280  364  -\n681    b  27.83   1.000  y  p   d   h  3.000  f  f   0  f  g  00176  537  -\n682    b  17.08   3.290  u  g   i   v  0.335  f  f   0  t  g  00140    2  -\n683    b  36.42   0.750  y  p   d   v  0.585  f  f   0  f  g  00240    3  -\n684    b  40.58   3.290  u  g   m   v  3.500  f  f   0  t  s  00400    0  -\n685    b  21.08  10.085  y  p   e   h  1.250  f  f   0  f  g  00260    0  -\n686    a  22.67   0.750  u  g   c   v  2.000  f  t   2  t  g  00200  394  -\n687    a  25.25  13.500  y  p  ff  ff  2.000  f  t   1  t  g  00200    1  -\n688    b  17.92   0.205  u  g  aa   v  0.040  f  f   0  f  g  00280  750  -\n689    b  35.00   3.375  u  g   c   h  8.290  f  f   0  t  g  00000    0  -\n","output_type":"stream","name":"stdout"}]},{"source":"## 4. Handling the missing values (part ii)\n<p>We replaced all the question marks with NaNs. This is going to help us in the next missing value treatment that we are going to perform.</p>\n<p>An important question that gets raised here is <em>why are we giving so much importance to missing values</em>? Can't they be just ignored? Ignoring missing values can affect the performance of a machine learning model heavily. While ignoring the missing values our machine learning model may miss out on information about the dataset that may be useful for its training. Then, there are many models which cannot handle missing values implicitly such as LDA. </p>\n<p>So, to avoid this problem, we are going to impute the missing values with a strategy called mean imputation.</p>","cell_type":"markdown","metadata":{"run_control":{"frozen":true},"editable":false,"tags":["context"],"deletable":false,"dc":{"key":"24"}}},{"source":"# Impute the missing values with mean imputation\n\ncc_apps.fillna(np.mean(cc_apps), inplace=True)\n\n# Count the number of NaNs in the dataset to verify\n# ... YOUR CODE FOR TASK 4 ...\ncc_apps.isnull().sum()","cell_type":"code","execution_count":192,"metadata":{"tags":["sample_code"],"dc":{"key":"24"},"trusted":true},"outputs":[{"data":{"text/plain":"0     12\n1     12\n2      0\n3      6\n4      6\n5      9\n6      9\n7      0\n8      0\n9      0\n10     0\n11     0\n12     0\n13    13\n14     0\n15     0\ndtype: int64"},"output_type":"execute_result","metadata":{},"execution_count":192}]},{"source":"## 5. Handling the missing values (part iii)\n<p>We have successfully taken care of the missing values present in the numeric columns. There are still some missing values to be imputed for columns 0, 1, 3, 4, 5, 6 and 13. All of these columns contain non-numeric data and this why the mean imputation strategy would not work here. This needs a different treatment. </p>\n<p>We are going to impute these missing values with the most frequent values as present in the respective columns. This is <a href=\"https://www.datacamp.com/community/tutorials/categorical-data\">good practice</a> when it comes to imputing missing values for categorical data in general.</p>","cell_type":"markdown","metadata":{"run_control":{"frozen":true},"editable":false,"tags":["context"],"deletable":false,"dc":{"key":"31"}}},{"source":"# Iterate over each column of cc_apps\nfor col in cc_apps.columns:\n    # Check if the column is of object type\n    if cc_apps[col].dtypes == 'object':\n        # Impute with the most frequent value\n        print(cc_apps[col].value_counts().index[0])\n        cc_apps = cc_apps.fillna(cc_apps[col].value_counts().index[0])\n\n# Count the number of NaNs in the dataset and print the counts to verify\n# ... YOUR CODE FOR TASK 5 ...\ncc_apps.isnull().sum()","cell_type":"code","execution_count":194,"metadata":{"tags":["sample_code"],"dc":{"key":"31"},"trusted":true},"outputs":[{"text":"b\nb\nu\ng\nc\nv\nt\nf\nf\ng\n00000\n-\n","output_type":"stream","name":"stdout"},{"data":{"text/plain":"0     0\n1     0\n2     0\n3     0\n4     0\n5     0\n6     0\n7     0\n8     0\n9     0\n10    0\n11    0\n12    0\n13    0\n14    0\n15    0\ndtype: int64"},"output_type":"execute_result","metadata":{},"execution_count":194}]},{"source":"## 6. Preprocessing the data (part i)\n<p>The missing values are now successfully handled.</p>\n<p>There is still some minor but essential data preprocessing needed before we proceed towards building our machine learning model. We are going to divide these remaining preprocessing steps into three main tasks:</p>\n<ol>\n<li>Convert the non-numeric data into numeric.</li>\n<li>Split the data into train and test sets. </li>\n<li>Scale the feature values to a uniform range.</li>\n</ol>\n<p>First, we will be converting all the non-numeric values into numeric ones. We do this because not only it results in a faster computation but also many machine learning models (like XGBoost) (and especially the ones developed using scikit-learn) require the data to be in a strictly numeric format. We will do this by using a technique called <a href=\"http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.LabelEncoder.html\">label encoding</a>.</p>","cell_type":"markdown","metadata":{"run_control":{"frozen":true},"editable":false,"tags":["context"],"deletable":false,"dc":{"key":"38"}}},{"source":"# Import LabelEncoder\n# ... YOUR CODE FOR TASK 6 ...\nfrom sklearn.preprocessing import LabelEncoder\n\n# Instantiate LabelEncoder\n# ... YOUR CODE FOR TASK 6 ...\nle = LabelEncoder()\n# Iterate over all the values of each column and extract their dtypes\nfor col in cc_apps.columns.values:\n    # Compare if the dtype is object\n    if cc_apps[col].dtype=='object':\n    # Use LabelEncoder to do the numeric transformation\n        cc_apps[col]=le.fit_transform(cc_apps[col])","cell_type":"code","execution_count":196,"metadata":{"tags":["sample_code"],"dc":{"key":"38"},"trusted":true},"outputs":[{"text":"0         b\n1     29.50\n2         2\n3         y\n4         p\n5         e\n6         h\n7         2\n8         f\n9         f\n10        0\n11        f\n12        g\n13    00256\n14       17\n15        -\nName: 673, dtype: object\n0      b\n1      a\n2      a\n3      b\n4      b\n5      b\n6      b\n7      a\n8      b\n9      b\n10     b\n11     b\n12     a\n13     b\n14     a\n15     b\n16     b\n17     a\n18     b\n19     a\n20     b\n21     b\n22     a\n23     a\n24     a\n25     a\n26     a\n27     b\n28     b\n29     b\n      ..\n660    b\n661    b\n662    a\n663    b\n664    b\n665    b\n666    a\n667    a\n668    b\n669    b\n670    b\n671    b\n672    a\n673    b\n674    a\n675    a\n676    a\n677    b\n678    a\n679    a\n680    b\n681    b\n682    b\n683    b\n684    b\n685    b\n686    a\n687    a\n688    b\n689    b\nName: 0, Length: 690, dtype: object\n0      30.83\n1      58.67\n2      24.50\n3      27.83\n4      20.17\n5      32.08\n6      33.17\n7      22.92\n8      54.42\n9      42.50\n10     22.08\n11     29.92\n12     38.25\n13     48.08\n14     45.83\n15     36.67\n16     28.25\n17     23.25\n18     21.83\n19     19.17\n20     25.00\n21     23.25\n22     47.75\n23     27.42\n24     41.17\n25     15.83\n26     47.00\n27     56.58\n28     57.42\n29     42.08\n       ...  \n660    22.25\n661    29.83\n662    23.50\n663    32.08\n664    31.08\n665    31.83\n666    21.75\n667    17.92\n668    30.33\n669    51.83\n670    47.17\n671    25.83\n672    50.25\n673    29.50\n674    37.33\n675    41.58\n676    30.58\n677    19.42\n678    17.92\n679    20.08\n680    19.50\n681    27.83\n682    17.08\n683    36.42\n684    40.58\n685    21.08\n686    22.67\n687    25.25\n688    17.92\n689    35.00\nName: 1, Length: 690, dtype: object\n0       0.000\n1       4.460\n2       0.500\n3       1.540\n4       5.625\n5       4.000\n6       1.040\n7      11.585\n8       0.500\n9       4.915\n10      0.830\n11      1.835\n12      6.000\n13      6.040\n14     10.500\n15      4.415\n16      0.875\n17      5.875\n18      0.250\n19      8.585\n20     11.250\n21      1.000\n22      8.000\n23     14.500\n24      6.500\n25      0.585\n26     13.000\n27     18.500\n28      8.500\n29      1.040\n        ...  \n660     9.000\n661     3.500\n662     1.500\n663     4.000\n664     1.500\n665     0.040\n666    11.750\n667     0.540\n668     0.500\n669     2.040\n670     5.835\n671    12.835\n672     0.835\n673     2.000\n674     2.500\n675     1.040\n676    10.665\n677     7.250\n678    10.210\n679     1.250\n680     0.290\n681     1.000\n682     3.290\n683     0.750\n684     3.290\n685    10.085\n686     0.750\n687    13.500\n688     0.205\n689     3.375\nName: 2, Length: 690, dtype: float64\n0      u\n1      u\n2      u\n3      u\n4      u\n5      u\n6      u\n7      u\n8      y\n9      y\n10     u\n11     u\n12     u\n13     u\n14     u\n15     y\n16     u\n17     u\n18     u\n19     u\n20     u\n21     u\n22     u\n23     u\n24     u\n25     u\n26     u\n27     u\n28     u\n29     u\n      ..\n660    u\n661    u\n662    u\n663    y\n664    y\n665    y\n666    u\n667    u\n668    u\n669    y\n670    u\n671    u\n672    u\n673    y\n674    u\n675    u\n676    u\n677    u\n678    u\n679    u\n680    u\n681    y\n682    u\n683    y\n684    u\n685    y\n686    u\n687    y\n688    u\n689    u\nName: 3, Length: 690, dtype: object\n0      g\n1      g\n2      g\n3      g\n4      g\n5      g\n6      g\n7      g\n8      p\n9      p\n10     g\n11     g\n12     g\n13     g\n14     g\n15     p\n16     g\n17     g\n18     g\n19     g\n20     g\n21     g\n22     g\n23     g\n24     g\n25     g\n26     g\n27     g\n28     g\n29     g\n      ..\n660    g\n661    g\n662    g\n663    p\n664    p\n665    p\n666    g\n667    g\n668    g\n669    p\n670    g\n671    g\n672    g\n673    p\n674    g\n675    g\n676    g\n677    g\n678    g\n679    g\n680    g\n681    p\n682    g\n683    p\n684    g\n685    p\n686    g\n687    p\n688    g\n689    g\nName: 4, Length: 690, dtype: object\n0       w\n1       q\n2       q\n3       w\n4       w\n5       m\n6       r\n7      cc\n8       k\n9       w\n10      c\n11      c\n12      k\n13      k\n14      q\n15      k\n16      m\n17      q\n18      d\n19     cc\n20      c\n21      c\n22      c\n23      x\n24      q\n25      c\n26      i\n27      d\n28      e\n29      w\n       ..\n660    aa\n661     c\n662     w\n663    cc\n664     w\n665     m\n666     c\n667     c\n668     d\n669    ff\n670     w\n671    cc\n672    aa\n673     e\n674     i\n675    aa\n676     q\n677     m\n678    ff\n679     c\n680     k\n681     d\n682     i\n683     d\n684     m\n685     e\n686     c\n687    ff\n688    aa\n689     c\nName: 5, Length: 690, dtype: object\n0       v\n1       h\n2       h\n3       v\n4       v\n5       v\n6       h\n7       v\n8       h\n9       v\n10      h\n11      h\n12      v\n13      v\n14      v\n15      v\n16      v\n17      v\n18      h\n19      h\n20      v\n21      v\n22      v\n23      h\n24      v\n25      h\n26     bb\n27     bb\n28      h\n29      v\n       ..\n660     v\n661     v\n662     v\n663     v\n664     v\n665     v\n666     v\n667     v\n668     h\n669    ff\n670     v\n671     v\n672     v\n673     h\n674     h\n675     v\n676     h\n677     v\n678    ff\n679     v\n680     v\n681     h\n682     v\n683     v\n684     v\n685     h\n686     v\n687    ff\n688     v\n689     h\nName: 6, Length: 690, dtype: object\n0       1.250\n1       3.040\n2       1.500\n3       3.750\n4       1.710\n5       2.500\n6       6.500\n7       0.040\n8       3.960\n9       3.165\n10      2.165\n11      4.335\n12      1.000\n13      0.040\n14      5.000\n15      0.250\n16      0.960\n17      3.170\n18      0.665\n19      0.750\n20      2.500\n21      0.835\n22      7.875\n23      3.085\n24      0.500\n25      1.500\n26      5.165\n27     15.000\n28      7.000\n29      5.000\n        ...  \n660     0.085\n661     0.165\n662     0.875\n663     1.500\n664     0.040\n665     0.040\n666     0.250\n667     1.750\n668     0.085\n669     1.500\n670     5.500\n671     0.500\n672     0.500\n673     2.000\n674     0.210\n675     0.665\n676     0.085\n677     0.040\n678     0.000\n679     0.000\n680     0.290\n681     3.000\n682     0.335\n683     0.585\n684     3.500\n685     1.250\n686     2.000\n687     2.000\n688     0.040\n689     8.290\nName: 7, Length: 690, dtype: float64\n0      t\n1      t\n2      t\n3      t\n4      t\n5      t\n6      t\n7      t\n8      t\n9      t\n10     f\n11     t\n12     t\n13     f\n14     t\n15     t\n16     t\n17     t\n18     t\n19     t\n20     t\n21     t\n22     t\n23     t\n24     t\n25     t\n26     t\n27     t\n28     t\n29     t\n      ..\n660    f\n661    f\n662    f\n663    f\n664    f\n665    f\n666    f\n667    f\n668    f\n669    f\n670    f\n671    f\n672    f\n673    f\n674    f\n675    f\n676    f\n677    f\n678    f\n679    f\n680    f\n681    f\n682    f\n683    f\n684    f\n685    f\n686    f\n687    f\n688    f\n689    f\nName: 8, Length: 690, dtype: object\n0      t\n1      t\n2      f\n3      t\n4      f\n5      f\n6      f\n7      f\n8      f\n9      f\n10     f\n11     f\n12     f\n13     f\n14     t\n15     t\n16     t\n17     t\n18     f\n19     t\n20     t\n21     f\n22     t\n23     t\n24     t\n25     t\n26     t\n27     t\n28     t\n29     t\n      ..\n660    f\n661    f\n662    f\n663    f\n664    f\n665    f\n666    f\n667    t\n668    f\n669    f\n670    f\n671    f\n672    f\n673    f\n674    f\n675    f\n676    t\n677    t\n678    f\n679    f\n680    f\n681    f\n682    f\n683    f\n684    f\n685    f\n686    t\n687    t\n688    f\n689    f\nName: 9, Length: 690, dtype: object\n0       1\n1       6\n2       0\n3       5\n4       0\n5       0\n6       0\n7       0\n8       0\n9       0\n10      0\n11      0\n12      0\n13      0\n14      7\n15     10\n16      3\n17     10\n18      0\n19      7\n20     17\n21      0\n22      6\n23      1\n24      3\n25      2\n26      9\n27     17\n28      3\n29      6\n       ..\n660     0\n661     0\n662     0\n663     0\n664     0\n665     0\n666     0\n667     1\n668     0\n669     0\n670     0\n671     0\n672     0\n673     0\n674     0\n675     0\n676    12\n677     1\n678     0\n679     0\n680     0\n681     0\n682     0\n683     0\n684     0\n685     0\n686     2\n687     1\n688     0\n689     0\nName: 10, Length: 690, dtype: int64\n0      f\n1      f\n2      f\n3      t\n4      f\n5      t\n6      t\n7      f\n8      f\n9      t\n10     t\n11     f\n12     t\n13     f\n14     t\n15     t\n16     t\n17     f\n18     t\n19     f\n20     f\n21     f\n22     t\n23     f\n24     t\n25     f\n26     t\n27     t\n28     f\n29     t\n      ..\n660    f\n661    f\n662    t\n663    t\n664    f\n665    f\n666    t\n667    t\n668    t\n669    f\n670    f\n671    f\n672    t\n673    f\n674    f\n675    f\n676    t\n677    f\n678    f\n679    f\n680    f\n681    f\n682    t\n683    f\n684    t\n685    f\n686    t\n687    t\n688    f\n689    t\nName: 11, Length: 690, dtype: object\n0      g\n1      g\n2      g\n3      g\n4      s\n5      g\n6      g\n7      g\n8      g\n9      g\n10     g\n11     g\n12     g\n13     g\n14     g\n15     g\n16     g\n17     g\n18     g\n19     g\n20     g\n21     s\n22     g\n23     g\n24     g\n25     g\n26     g\n27     g\n28     g\n29     g\n      ..\n660    g\n661    g\n662    g\n663    g\n664    s\n665    g\n666    g\n667    g\n668    s\n669    g\n670    g\n671    g\n672    g\n673    g\n674    g\n675    g\n676    g\n677    g\n678    g\n679    g\n680    g\n681    g\n682    g\n683    g\n684    s\n685    g\n686    g\n687    g\n688    g\n689    g\nName: 12, Length: 690, dtype: object\n0      00202\n1      00043\n2      00280\n3      00100\n4      00120\n5      00360\n6      00164\n7      00080\n8      00180\n9      00052\n10     00128\n11     00260\n12     00000\n13     00000\n14     00000\n15     00320\n16     00396\n17     00120\n18     00000\n19     00096\n20     00200\n21     00300\n22     00000\n23     00120\n24     00145\n25     00100\n26     00000\n27     00000\n28     00000\n29     00500\n       ...  \n660    00000\n661    00216\n662    00160\n663    00120\n664    00160\n665    00000\n666    00180\n667    00080\n668    00252\n669    00120\n670    00465\n671    00000\n672    00240\n673    00256\n674    00260\n675    00240\n676    00129\n677    00100\n678    00000\n679    00000\n680    00280\n681    00176\n682    00140\n683    00240\n684    00400\n685    00260\n686    00200\n687    00200\n688    00280\n689    00000\nName: 13, Length: 690, dtype: object\n0          0\n1        560\n2        824\n3          3\n4          0\n5          0\n6      31285\n7       1349\n8        314\n9       1442\n10         0\n11       200\n12         0\n13      2690\n14         0\n15         0\n16         0\n17       245\n18         0\n19         0\n20      1208\n21         0\n22      1260\n23        11\n24         0\n25         0\n26         0\n27         0\n28         0\n29     10000\n       ...  \n660        0\n661        0\n662        0\n663        0\n664        0\n665        0\n666        0\n667        5\n668        0\n669        1\n670      150\n671        2\n672      117\n673       17\n674      246\n675      237\n676        3\n677        1\n678       50\n679        0\n680      364\n681      537\n682        2\n683        3\n684        0\n685        0\n686      394\n687        1\n688      750\n689        0\nName: 14, Length: 690, dtype: int64\n0      +\n1      +\n2      +\n3      +\n4      +\n5      +\n6      +\n7      +\n8      +\n9      +\n10     +\n11     +\n12     +\n13     +\n14     +\n15     +\n16     +\n17     +\n18     +\n19     +\n20     +\n21     +\n22     +\n23     +\n24     +\n25     +\n26     +\n27     +\n28     +\n29     +\n      ..\n660    -\n661    -\n662    -\n663    -\n664    -\n665    -\n666    -\n667    -\n668    -\n669    -\n670    -\n671    -\n672    -\n673    -\n674    -\n675    -\n676    -\n677    -\n678    -\n679    -\n680    -\n681    -\n682    -\n683    -\n684    -\n685    -\n686    -\n687    -\n688    -\n689    -\nName: 15, Length: 690, dtype: object\n","output_type":"stream","name":"stdout"}]},{"source":"## 7. Splitting the dataset into train and test sets\n<p>We have successfully converted all the non-numeric values to numeric ones.</p>\n<p>Now, we will split our data into train set and test set to prepare our data for two different phases of machine learning modeling: training and testing. Ideally, no information from the test data should be used to scale the training data or should be used to direct the training process of a machine learning model. Hence, we first split the data and then apply the scaling.</p>\n<p>Also, features like <code>DriversLicense</code> and <code>ZipCode</code> are not as important as the other features in the dataset for predicting credit card approvals. We should drop them to design our machine learning model with the best set of features. In Data Science literature, this is often referred to as <em>feature selection</em>. </p>","cell_type":"markdown","metadata":{"run_control":{"frozen":true},"editable":false,"tags":["context"],"deletable":false,"dc":{"key":"45"}}},{"source":"# Import train_test_split\n# ... YOUR CODE FOR TASK 7 ...\nfrom sklearn.model_selection import train_test_split\n# Drop the features 11 and 13 and convert the DataFrame to a NumPy array\ncc_apps = cc_apps.drop([11, 13], axis=1)\ncc_apps = cc_apps.values\n\n# Segregate features and labels into separate variables\nX,y = cc_apps[:,0:13] , cc_apps[:,13]\n\n# Split into train and test sets\nX_train, X_test, y_train, y_test = train_test_split(X,\n                                y,\n                                test_size=0.33,\n                                random_state=42)","cell_type":"code","execution_count":198,"metadata":{"tags":["sample_code"],"dc":{"key":"45"},"trusted":true},"outputs":[]},{"source":"## 8. Preprocessing the data (part ii)\n<p>The data is now split into two separate sets - train and test sets respectively. We are only left with one final preprocessing step of scaling before we can fit a machine learning model to the data. </p>\n<p>Now, let's try to understand what these scaled values mean in the real world. Let's use <code>CreditScore</code> as an example. The credit score of a person is their creditworthiness based on their credit history. The higher this number, the more financially trustworthy a person is considered to be. So, a <code>CreditScore</code> of 1 is the highest since we're rescaling all the values to the range of 0-1.</p>","cell_type":"markdown","metadata":{"run_control":{"frozen":true},"editable":false,"tags":["context"],"deletable":false,"dc":{"key":"52"}}},{"source":"# Import MinMaxScaler\n# ... YOUR CODE FOR TASK 8 ...\nfrom sklearn.preprocessing import MinMaxScaler\n# Instantiate MinMaxScaler and use it to rescale X_train and X_test\nscaler = MinMaxScaler(feature_range=(0, 1))\nrescaledX_train = scaler.fit_transform(X_train)\nrescaledX_test = scaler.fit_transform(X_test)","cell_type":"code","execution_count":200,"metadata":{"tags":["sample_code"],"collapsed":true,"dc":{"key":"52"},"trusted":true},"outputs":[]},{"source":"## 9. Fitting a logistic regression model to the train set\n<p>Essentially, predicting if a credit card application will be approved or not is a <a href=\"https://en.wikipedia.org/wiki/Statistical_classification\">classification</a> task. <a href=\"http://archive.ics.uci.edu/ml/machine-learning-databases/credit-screening/crx.names\">According to UCI</a>, our dataset contains more instances that correspond to \"Denied\" status than instances corresponding to \"Approved\" status. Specifically, out of 690 instances, there are 383 (55.5%) applications that got denied and 307 (44.5%) applications that got approved. </p>\n<p>This gives us a benchmark. A good machine learning model should be able to accurately predict the status of the applications with respect to these statistics.</p>\n<p>Which model should we pick? A question to ask is: <em>are the features that affect the credit card approval decision process correlated with each other?</em> Although we can measure correlation, that is outside the scope of this notebook, so we'll rely on our intuition that they indeed are correlated for now. Because of this correlation, we'll take advantage of the fact that generalized linear models perform well in these cases. Let's start our machine learning modeling with a Logistic Regression model (a generalized linear model).</p>","cell_type":"markdown","metadata":{"run_control":{"frozen":true},"editable":false,"tags":["context"],"deletable":false,"dc":{"key":"59"}}},{"source":"# Import LogisticRegression\n# ... YOUR CODE FOR TASK 9 ...\nfrom sklearn.linear_model import LogisticRegression\n# Instantiate a LogisticRegression classifier with default parameter values\nlogreg = LogisticRegression()\n\n# Fit logreg to the train set\n# ... YOUR CODE FOR TASK 9 ...\nlogreg.fit(X_train,y_train)","cell_type":"code","execution_count":202,"metadata":{"tags":["sample_code"],"dc":{"key":"59"},"trusted":true},"outputs":[{"data":{"text/plain":"LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n          intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n          penalty='l2', random_state=None, solver='liblinear', tol=0.0001,\n          verbose=0, warm_start=False)"},"output_type":"execute_result","metadata":{},"execution_count":202}]},{"source":"## 10. Making predictions and evaluating performance\n<p>But how well does our model perform? </p>\n<p>We will now evaluate our model on the test set with respect to <a href=\"https://developers.google.com/machine-learning/crash-course/classification/accuracy\">classification accuracy</a>. But we will also take a look the model's <a href=\"http://www.dataschool.io/simple-guide-to-confusion-matrix-terminology/\">confusion matrix</a>. In the case of predicting credit card applications, it is equally important to see if our machine learning model is able to predict the approval status of the applications as denied that originally got denied. If our model is not performing well in this aspect, then it might end up approving the application that should have been approved. The confusion matrix helps us to view our model's performance from these aspects.  </p>","cell_type":"markdown","metadata":{"run_control":{"frozen":true},"editable":false,"tags":["context"],"deletable":false,"dc":{"key":"66"}}},{"source":"# Import confusion_matrix\n# ... YOUR CODE FOR TASK 10 ...\nfrom sklearn.metrics import confusion_matrix\n# Use logreg to predict instances from the test set and store it\ny_pred = logreg.predict(rescaledX_test)\n\n# Get the accuracy score of logreg model and print it\nprint(\"Accuracy of logistic regression classifier: \", logreg.score(rescaledX_test,y_test))\n\n# Print the confusion matrix of the logreg model\n# ... YOUR CODE FOR TASK 10 ...\nconfusion_matrix(y_test,y_pred)","cell_type":"code","execution_count":204,"metadata":{"tags":["sample_code"],"dc":{"key":"66"},"trusted":true},"outputs":[{"text":"Accuracy of logistic regression classifier:  0.8421052631578947\n","output_type":"stream","name":"stdout"},{"data":{"text/plain":"array([[95,  8],\n       [28, 97]])"},"output_type":"execute_result","metadata":{},"execution_count":204}]},{"source":"## 11. Grid searching and making the model perform better\n<p>Our model was pretty good! It was able to yield an accuracy score of almost 84%.</p>\n<p>For the confusion matrix, the first element of the of the first row of the confusion matrix denotes the true negatives meaning the number of negative instances (denied applications) predicted by the model correctly. And the last element of the second row of the confusion matrix denotes the true positives meaning the number of positive instances (approved applications) predicted by the model correctly.</p>\n<p>Let's see if we can do better. We can perform a <a href=\"https://machinelearningmastery.com/how-to-tune-algorithm-parameters-with-scikit-learn/\">grid search</a> of the model parameters to improve the model's ability to predict credit card approvals.</p>\n<p><a href=\"http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html\">scikit-learn's implementation of logistic regression</a> consists of different hyperparameters but we will grid search over the following two:</p>\n<ul>\n<li>tol</li>\n<li>max_iter</li>\n</ul>","cell_type":"markdown","metadata":{"run_control":{"frozen":true},"editable":false,"tags":["context"],"deletable":false,"dc":{"key":"73"}}},{"source":"# Import GridSearchCV\n# ... YOUR CODE FOR TASK 11 ...\nfrom sklearn.model_selection import GridSearchCV\n# Define the grid of values for tol and max_iter\ntol = [0.01,0.001,0.0001]\nmax_iter = [100,150,200]\n\n# Create a dictionary where tol and max_iter are keys and the lists of their values are corresponding values\nparam_grid = dict(tol=tol, max_iter=max_iter)","cell_type":"code","execution_count":207,"metadata":{"tags":["sample_code"],"dc":{"key":"73"},"trusted":true},"outputs":[]},{"source":"## 12. Finding the best performing model\n<p>We have defined the grid of hyperparameter values and converted them into a single dictionary format which <code>GridSearchCV()</code> expects as one of its parameters. Now, we will begin the grid search to see which values perform best.</p>\n<p>We will instantiate <code>GridSearchCV()</code> with our earlier <code>logreg</code> model with all the data we have. Instead of passing train and test sets separately, we will supply <code>X</code> (scaled version) and <code>y</code>. We will also instruct <code>GridSearchCV()</code> to perform a <a href=\"https://www.dataschool.io/machine-learning-with-scikit-learn/\">cross-validation</a> of five folds.</p>\n<p>We'll end the notebook by storing the best-achieved score and the respective best parameters.</p>\n<p>While building this credit card predictor, we tackled some of the most widely-known preprocessing steps such as <strong>scaling</strong>, <strong>label encoding</strong>, and <strong>missing value imputation</strong>. We finished with some <strong>machine learning</strong> to predict if a person's application for a credit card would get approved or not given some information about that person.</p>","cell_type":"markdown","metadata":{"run_control":{"frozen":true},"editable":false,"tags":["context"],"deletable":false,"dc":{"key":"80"}}},{"source":"# Instantiate GridSearchCV with the required parameters\ngrid_model = GridSearchCV(estimator=logreg, param_grid=param_grid, cv=5)\n\n# Use scaler to rescale X and assign it to rescaledX\nrescaledX = scaler.fit_transform(X)\n\n# Fit data to grid_model\ngrid_model_result = grid_model.fit(res, ...)\n\n# Summarize results\nbest_score, best_params = ...\nprint(\"Best: %f using %s\" % (..., ...))","cell_type":"code","execution_count":null,"metadata":{"tags":["sample_code"],"collapsed":true,"dc":{"key":"80"},"trusted":true},"outputs":[]}],"nbformat":4,"metadata":{"language_info":{"mimetype":"text/x-python","file_extension":".py","pygments_lexer":"ipython3","name":"python","nbconvert_exporter":"python","version":"3.5.2","codemirror_mode":{"version":3,"name":"ipython"}},"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"}}}