Lab 8_ Training a Linear Classifier with Numerical and Categorical Features.ipynb

{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"name":"Lab 8_ Training a Linear Classifier with Numerical and Categorical Features.ipynb","version":"0.3.2","views":{},"default_view":{},"provenance":[{"file_id":"/v2/external/notebooks/intro_to_ml_semester_course/Lab_8__Training_a_Linear_Classifier_with_Numerical_and_Categorical_Features.ipynb","timestamp":1526419933234},{"file_id":"0BxDag_sVV0QbaHdJbTRENU95N28","timestamp":1505169321861},{"file_id":"0BxDag_sVV0QbNlZGN1VOeTgwdzg","timestamp":1505157544448},{"file_id":"0BxDag_sVV0Qbc1BJRUg1Zml3UnM","timestamp":1504899671600},{"file_id":"0BxDag_sVV0QbQ1IwNHNPaU56YjA","timestamp":1504728550259},{"file_id":"0BxDag_sVV0QbeHl6TnVGaERiRWs","timestamp":1504141803468},{"file_id":"0BxDag_sVV0QbM0RHakNBcGtFdlE","timestamp":1503350093341},{"file_id":"0BxDag_sVV0QbZnBLa1FqNEE3bTg","timestamp":1500399420475},{"file_id":"0BxDag_sVV0QbYVM3cV9XVWhrRjg","timestamp":1500234272739},{"file_id":"0B1ymeC_TeE1dQUdHcVpFbC10YmM","timestamp":1463102040711}],"collapsed_sections":["JndnmDMp66FL"]}},"cells":[{"metadata":{"id":"JndnmDMp66FL","colab_type":"text"},"cell_type":"markdown","source":["#### Copyright 2017 Google LLC."]},{"metadata":{"id":"hMqWDc_m6rUC","colab_type":"code","colab":{"autoexec":{"startup":false,"wait_interval":0}},"cellView":"both"},"cell_type":"code","source":["# Licensed under the Apache License, Version 2.0 (the \"License\");\n","# you may not use this file except in compliance with the License.\n","# You may obtain a copy of the License at\n","#\n","# https://www.apache.org/licenses/LICENSE-2.0\n","#\n","# Unless required by applicable law or agreed to in writing, software\n","# distributed under the License is distributed on an \"AS IS\" BASIS,\n","# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n","# See the License for the specific language governing permissions and\n","# limitations under the License."],"execution_count":0,"outputs":[]},{"metadata":{"id":"4f3CKqFUqL2-","colab_type":"text","slideshow":{"slide_type":"slide"}},"cell_type":"markdown","source":["# Lab 8: Training a Linear Classifier With Numerical and Categorical Features\n","**Learning Objectives:**\n","  * Introduce logistic regression to train a binary classifier.\n","  * Understand metrics such as ROC curves, AUC, log loss, classification errors.\n","  * Train a `LinearClassifier` using the raw numerical and categorical features.\n"]},{"metadata":{"id":"UHU_WK5Z2gsw","colab_type":"text"},"cell_type":"markdown","source":["### Standard Set-up\n","\n","We begin with the standard set-up as seen in the last lab, however, we use a different data set based on 1990 census data from California. Since this data set has a header row, we don't need to provide the column names."]},{"metadata":{"id":"P5_-Nmc52gsy","colab_type":"code","slideshow":{"slide_type":"-"},"colab":{"autoexec":{"startup":false,"wait_interval":0},"test":{"output":"ignore","timeout":600}},"cellView":"both"},"cell_type":"code","source":["import math\n","\n","from IPython import display\n","from matplotlib import cm\n","from matplotlib import gridspec\n","from matplotlib import pyplot as plt\n","import numpy as np\n","import pandas as pd\n","from mpl_toolkits.mplot3d import Axes3D\n","from sklearn import metrics\n","import tensorflow as tf\n","from tensorflow.contrib.learn.python.learn import learn_io, estimator\n","\n","# This line increases the amount of logging when there is an error.  You can\n","# remove it if you want less logging.\n","tf.logging.set_verbosity(tf.logging.ERROR)\n","\n","# Set the output display to have two digits for decimal places, for display\n","# readability only and limit it to printing 15 rows.\n","pd.options.display.float_format = '{:.2f}'.format\n","pd.options.display.max_rows = 15\n"],"execution_count":0,"outputs":[]},{"metadata":{"id":"RG4WjoRq7we3","colab_type":"text"},"cell_type":"markdown","source":["In this lab, you will solve a binary classification problem: Given census data about a person such as age, gender, education and occupation, we will predict whether or not the person earns more than 50,000 dollars a year (the target label). We will train a linear classifier that given the feature values for an individual outputs a number between 0 and 1, which can be interpreted as the probability that the individual has an annual income of over 50K.\n","\n","The dataset we'll be using is the [Census Income Dataset](https://archive.ics.uci.edu/ml/datasets/Census+Income). We load the [training data](https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.data) durectly from here. Most of the column names should be self explanatory if not you can read more about it at the website linked above. The number of people the census takers believe that each observation represents (sample weight) could be used to weight the examples during training but for the purpose of this lab you can ignore it.  You don't want to use it as a feature."]},{"metadata":{"id":"WrtTmerb7wOT","colab_type":"code","colab":{"autoexec":{"startup":false,"wait_interval":0},"base_uri":"https://localhost:8080/","height":370},"outputId":"2131ad65-88f9-4b40-8578-0d0177561b5e","executionInfo":{"status":"ok","timestamp":1526654149087,"user_tz":240,"elapsed":1184,"user":{"displayName":"Airi C.","photoUrl":"//lh6.googleusercontent.com/-mRsT-p21xZI/AAAAAAAAAAI/AAAAAAAAHeU/RHc7qaRZabg/s50-c-k-no/photo.jpg","userId":"109705951704145205723"}}},"cell_type":"code","source":["import tempfile\n","import urllib\n","train_file = tempfile.NamedTemporaryFile()\n","urllib.urlretrieve(\"http://mlr.cs.umass.edu/ml/machine-learning-databases/adult/adult.data\", train_file.name)\n","\n","COLUMNS = [\"age\", \"workclass\", \"sample_weight\", \"education\", \"education_num\",\n","           \"marital_status\", \"occupation\", \"relationship\", \"race\", \"gender\",\n","           \"capital_gain\", \"capital_loss\", \"hours_per_week\", \"native_country\",\n","           \"income_bracket\"]\n","census_df = pd.read_csv(train_file, names=COLUMNS, skipinitialspace=True)\n","# Show the first 5 rows of the table.\n","census_df.head(5)"],"execution_count":2,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>age</th>\n","      <th>workclass</th>\n","      <th>sample_weight</th>\n","      <th>education</th>\n","      <th>education_num</th>\n","      <th>marital_status</th>\n","      <th>occupation</th>\n","      <th>relationship</th>\n","      <th>race</th>\n","      <th>gender</th>\n","      <th>capital_gain</th>\n","      <th>capital_loss</th>\n","      <th>hours_per_week</th>\n","      <th>native_country</th>\n","      <th>income_bracket</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>39</td>\n","      <td>State-gov</td>\n","      <td>77516</td>\n","      <td>Bachelors</td>\n","      <td>13</td>\n","      <td>Never-married</td>\n","      <td>Adm-clerical</td>\n","      <td>Not-in-family</td>\n","      <td>White</td>\n","      <td>Male</td>\n","      <td>2174</td>\n","      <td>0</td>\n","      <td>40</td>\n","      <td>United-States</td>\n","      <td>&lt;=50K</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>50</td>\n","      <td>Self-emp-not-inc</td>\n","      <td>83311</td>\n","      <td>Bachelors</td>\n","      <td>13</td>\n","      <td>Married-civ-spouse</td>\n","      <td>Exec-managerial</td>\n","      <td>Husband</td>\n","      <td>White</td>\n","      <td>Male</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>13</td>\n","      <td>United-States</td>\n","      <td>&lt;=50K</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>38</td>\n","      <td>Private</td>\n","      <td>215646</td>\n","      <td>HS-grad</td>\n","      <td>9</td>\n","      <td>Divorced</td>\n","      <td>Handlers-cleaners</td>\n","      <td>Not-in-family</td>\n","      <td>White</td>\n","      <td>Male</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>40</td>\n","      <td>United-States</td>\n","      <td>&lt;=50K</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>53</td>\n","      <td>Private</td>\n","      <td>234721</td>\n","      <td>11th</td>\n","      <td>7</td>\n","      <td>Married-civ-spouse</td>\n","      <td>Handlers-cleaners</td>\n","      <td>Husband</td>\n","      <td>Black</td>\n","      <td>Male</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>40</td>\n","      <td>United-States</td>\n","      <td>&lt;=50K</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>28</td>\n","      <td>Private</td>\n","      <td>338409</td>\n","      <td>Bachelors</td>\n","      <td>13</td>\n","      <td>Married-civ-spouse</td>\n","      <td>Prof-specialty</td>\n","      <td>Wife</td>\n","      <td>Black</td>\n","      <td>Female</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>40</td>\n","      <td>Cuba</td>\n","      <td>&lt;=50K</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["   age         workclass  sample_weight  education  education_num  \\\n","0   39         State-gov          77516  Bachelors             13   \n","1   50  Self-emp-not-inc          83311  Bachelors             13   \n","2   38           Private         215646    HS-grad              9   \n","3   53           Private         234721       11th              7   \n","4   28           Private         338409  Bachelors             13   \n","\n","       marital_status         occupation   relationship   race  gender  \\\n","0       Never-married       Adm-clerical  Not-in-family  White    Male   \n","1  Married-civ-spouse    Exec-managerial        Husband  White    Male   \n","2            Divorced  Handlers-cleaners  Not-in-family  White    Male   \n","3  Married-civ-spouse  Handlers-cleaners        Husband  Black    Male   \n","4  Married-civ-spouse     Prof-specialty           Wife  Black  Female   \n","\n","   capital_gain  capital_loss  hours_per_week native_country income_bracket  \n","0          2174             0              40  United-States          <=50K  \n","1             0             0              13  United-States          <=50K  \n","2             0             0              40  United-States          <=50K  \n","3             0             0              40  United-States          <=50K  \n","4             0             0              40           Cuba          <=50K  "]},"metadata":{"tags":[]},"execution_count":2}]},{"metadata":{"id":"2BokkZ4g2gs4","colab_type":"text"},"cell_type":"markdown","source":["### Setting Up the Feature Columns and Input Function for TensorFlow\n","As in the past labs, we define `input_fn` to define a `FeatureColumn` for each categorical and numerical feature, and then define `train_input_fn` to use the training data, `eval_input_fn` to use the validation data, and `test_input_fn` to use the test data."]},{"metadata":{"id":"OwnMzAkn2gs6","colab_type":"code","colab":{"autoexec":{"startup":false,"wait_interval":0}}},"cell_type":"code","source":["CATEGORICAL_COLUMNS = [\"workclass\", \"education\", \"marital_status\", \"occupation\",\n","                       \"relationship\", \"race\", \"gender\", \"native_country\"]\n","NUMERICAL_COLUMNS = [\"age\", \"education_num\", \"capital_gain\", \"capital_loss\",\n","                      \"hours_per_week\"]\n","LABEL = \"income_over_50k\"\n","\n","def input_fn(dataframe):\n","  \"\"\"Constructs a dictionary for the feature columns.\n","\n","  Args:\n","    dataframe: The Pandas DataFrame to use for the input.\n","  Returns:\n","    The feature columns and the associated labels for the provided input.\n","  \"\"\"\n","  # Creates a dictionary mapping each numerical feature column name (k) to\n","  # the values of that column stored in a constant Tensor.\n","  numerical_cols = {k: tf.constant(dataframe[k].values) \n","                    for k in NUMERICAL_COLUMNS}\n","  # Creates a dictionary mapping each categorical feature column name (k)\n","  # to the values of that column stored in a tf.SparseTensor.\n","  categorical_cols = {k: tf.SparseTensor(\n","      indices=[[i, 0] for i in range(dataframe[k].size)],\n","      values=dataframe[k].values,\n","      dense_shape=[dataframe[k].size, 1])\n","                      for k in CATEGORICAL_COLUMNS}\n","  # Merges the two dictionaries into one.\n","  feature_cols = dict(numerical_cols.items() + categorical_cols.items())\n","  # Converts the label column into a constant Tensor.\n","  label = tf.constant(dataframe[LABEL].values)\n","  # Returns the feature columns and the label.\n","  return feature_cols, label\n","\n","def train_input_fn():\n","  return input_fn(training_examples)\n","\n","def eval_input_fn():\n","  return input_fn(validation_examples)\n","\n","def test_input_fn():\n","  return input_fn(test_examples)"],"execution_count":0,"outputs":[]},{"metadata":{"id":"K83U9tm72yF7","colab_type":"text"},"cell_type":"markdown","source":["##Prepare Features\n","\n","Here is a basic implementation of `PrepareFeatures` for you to use.  Feel free to modify this to use feature normalization other than just linear scaling.  \n","\n","Note that for linear classification with just two labels, the labels must be 0 (think of this as false) and 1 (think of this as true).  Since `income_brackets` is a string, we must convert it to an integer. This can be done using a lambda function that outputs a Boolean  value, and then casts it to an integer. We have provided this for you."]},{"metadata":{"id":"JETiyZLS2gtY","colab_type":"code","colab":{"autoexec":{"startup":false,"wait_interval":0}}},"cell_type":"code","source":["# Linearly rescales to the range [0, 1]\n","def linear_scale(series):\n","  min_val = series.min()\n","  max_val = series.max()\n","  scale = 1.0 * (max_val - min_val)\n","  return series.apply(lambda x:((x - min_val) / scale))\n","\n","def prepare_features(dataframe):\n","  \"\"\"Prepares the features for provided dataset.\n","\n","  Args:\n","    dataframe: A Pandas DataFrame expected to contain the data.\n","  Returns:\n","    A new DataFrame that contains the features to be used for the model.\n","  \"\"\"\n","  processed_features = dataframe.copy()\n","  for feature in NUMERICAL_COLUMNS:\n","    processed_features[feature] = linear_scale(dataframe[feature])\n","    \n","  # Convert the output target to 0 (for <=50k) and 1 (> 50k)\n","  processed_features[LABEL] = dataframe[\"income_bracket\"].apply(\n","      lambda x: \">50K\" in x).astype(int) #Is x equal to str \">50k\" returns boolean.\n","  \n","  return processed_features"],"execution_count":0,"outputs":[]},{"metadata":{"id":"O4tSQbQmGo7m","colab_type":"text"},"cell_type":"markdown","source":["###Divide the provided data for training our model into training and validation sets\n","\n","As we've done in the past, let's divide the data into a ***training set*** and ***validation set***.  There are 16281 examples so let's set aside 4000 for our validation data.  Let's not forget to randomize the order before splitting the data so that our validation set is a representative sample."]},{"metadata":{"id":"pr9J6wVUGoQX","colab_type":"code","colab":{"autoexec":{"startup":false,"wait_interval":0}}},"cell_type":"code","source":["census_df = census_df.reindex(np.random.permutation(census_df.index))\n","training_examples = prepare_features(census_df.head(12281))\n","validation_examples = prepare_features(census_df.tail(4000))"],"execution_count":0,"outputs":[]},{"metadata":{"id":"27I6NhglBnUB","colab_type":"text"},"cell_type":"markdown","source":["### Load the Test Data\n","\n","Like with the housing data, this data set has a specified [test data](https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.test) that we can use at the end to look at the performance for our final classifier.  We will load it and do the pre-processing here but this should not be used in training or selecting any of the hyperparameters."]},{"metadata":{"id":"XK2--38hBnl3","colab_type":"code","colab":{"autoexec":{"startup":false,"wait_interval":0}}},"cell_type":"code","source":["test_file = tempfile.NamedTemporaryFile()\n","urllib.urlretrieve(\"http://mlr.cs.umass.edu/ml/machine-learning-databases/adult/adult.test\", test_file.name)\n","\n","census_df_test = pd.read_csv(test_file, names=COLUMNS, skipinitialspace=True, skiprows=1)\n","test_examples = prepare_features(census_df_test)"],"execution_count":0,"outputs":[]},{"metadata":{"id":"v5XS0Z_-GevD","colab_type":"text"},"cell_type":"markdown","source":["### Functions to help visualize our results\n","\n","Since this is a classification problem, the calibration plot is not a good visualization tool. Instead we use an **ROC curve** in which the x-axis is the false positive rate and the y-axis is the true positive rate.  An ROC curve is a very good way to visualize the quality of a binary classifier and also to pick a threshold when making a binary prediction.  Recall that the line `x=y` corresponds to a random classifier.  **AUC** (the area under the ROC curve) has the nice *probabilistic interpretation that a random positive example is predicted to be more likely to be positive than a random negative example*.\n","\n","Our implementation of `make_roc_curve` uses the [sklearn metrics](http://scikit-learn.org/stable/modules/classes.html#module-sklearn.metrics) class. There are a lot of tools already available within Python libraries so be sure and look for those. "]},{"metadata":{"id":"kiaRYgOSDCFw","colab_type":"text"},"cell_type":"markdown","source":[""]},{"metadata":{"id":"YIXTkIa6GB0s","colab_type":"code","colab":{"autoexec":{"startup":false,"wait_interval":0}}},"cell_type":"code","source":["from sklearn.metrics import confusion_matrix\n","from sklearn import metrics\n","\n","def make_roc_curve(predictions, targets):\n","  \"\"\" Plots an ROC curve for the provided predictions and targets.\n","\n","  Args:\n","    predictions: the probability that the example has label 1.\n","    targets: a list of the target values being predicted that must be the\n","             same length as predictions.\n","  \"\"\"  \n","  # Put your code here to get false positives and true positives\n","  # Hint: check sklearn's metrics to find anything related to roc_curve!\n","  \n","  \n","  fpr, tpr, threshords = metrics.roc_curve (predictions, targets, pos_label=1)\n","  fpr\n","  # Put your code here to plot the results\n","\n","  \n","def plot_learning_curve(training_losses, validation_losses):\n","  \"\"\" Plot the learning curve.\n","  \n","  Args:\n","    training_loses: a list of training losses to plot.\n","    validation_losses: a list of validation losses to plot.\n","  \"\"\"   \n","  # Plot the training and validation losses\n","  # x-axis should be the training steps\n","  # y-axis should be the loss\n"],"execution_count":0,"outputs":[]},{"metadata":{"id":"G2F3zbi2CjM7","colab_type":"text"},"cell_type":"markdown","source":["### Compute Loss\n","\n","For classification problems, we generally would like our output to be a probability distribution over the possible classes.  When we have two classes the **log loss** is a measure of how close the predicted distribution is to the target distribution, and that is the metric that we will optimize. Again, we use the [sklearn metrics](http://scikit-learn.org/stable/modules/classes.html#module-sklearn.metrics) class. "]},{"metadata":{"id":"sPHDJH0JCiWP","colab_type":"code","colab":{"autoexec":{"startup":false,"wait_interval":0},"base_uri":"https://localhost:8080/","height":153},"outputId":"93a0cce7-0951-4e0d-c5a9-2174f772f192","executionInfo":{"status":"error","timestamp":1526498887184,"user_tz":240,"elapsed":470,"user":{"displayName":"Airi C.","photoUrl":"//lh6.googleusercontent.com/-mRsT-p21xZI/AAAAAAAAAAI/AAAAAAAAHeU/RHc7qaRZabg/s50-c-k-no/photo.jpg","userId":"109705951704145205723"}}},"cell_type":"code","source":["def compute_loss(model, input_fn, targets):\n","  \"\"\" Computes the log loss for training a linear classifier.\n","  \n","  Args:\n","    model: the trained model to use for making the predictions.\n","    input_fn: the input_fn to use to make the predicitons.\n","    targets: a list of the target values being predicted that must be the\n","             same length as predictions.\n","    \n","  Returns:\n","    The log loss for the provided predictions and targets.\n","  \"\"\"      \n","  # Use the model to calculate the output probabilities \n","  # Hint, use the input_fn. Store the results in a numpy array\n","  prediction = ...\n","  \n","  # Calculate the log loss between targets and your predictions\n","  # Hint: check the sklearn metrics!\n","  log_loss = ...\n","  return log_loss"],"execution_count":11,"outputs":[{"output_type":"error","ename":"SyntaxError","evalue":"ignored","traceback":["\u001b[0;36m  File \u001b[0;32m\"<ipython-input-11-6a92acbbc410>\"\u001b[0;36m, line \u001b[0;32m15\u001b[0m\n\u001b[0;31m    prediction = ...\u001b[0m\n\u001b[0m                 ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m invalid syntax\n"]}]},{"metadata":{"id":"LolSOay4Vnj0","colab_type":"text"},"cell_type":"markdown","source":["##Setting up the Features\n","\n","We will set things up showing you an example of how to set up each of the kind of features you will be using.  Then you can add in additional features.\n","\n","####Categorical Feauture Columns with known values.\n","\n","When the values are known you can simply use a line like below.  If you would view the weights, index 0 will be the first key provided, index 1, the next key,.....\n","\n","```\n","      gender = tf.contrib.layers.sparse_column_with_keys(column_name=\"gender\", keys=[\"Female\", \"Male\"])\n","  \n","  ```\n","####Categorical Feature Columns without known values\n","\n","Since you don't always know the possible values you can instead assign an index to each possible value via hashing where `hash_bucket_size` is the number of hash buckets used.\n","\n","```\n","      education = tf.contrib.layers.sparse_column_with_hash_bucket(\"education\", hash_bucket_size=100)\n","```\n","\n","####Numerical Feature Columns\n","As we have seen in past labs, we can directly use numerical features as long as appropriate scaling has been applied. The provided implementation of `prepare_features` linearly scales all numerical featuers to fall in [0,1]\n","```\n","   age = tf.contrib.layers.real_valued_column(\"age\") \n","```"]},{"metadata":{"id":"uq6e_T_1F94U","colab_type":"code","colab":{"autoexec":{"startup":false,"wait_interval":0}}},"cell_type":"code","source":["def construct_feature_columns():\n","  \"\"\"Construct TensorFlow Feature Columns for features\n","  \n","  Returns:\n","    A set of feature columns.\n","  \"\"\"\n","  \n","  # Sample of creating a categorical column with known values.\n","  gender = tf.contrib.layers.sparse_column_with_keys(\n","    column_name=\"gender\", keys=[\"Female\", \"Male\"])\n","  \n","  # Sample of creating a categorical columns with a hash bucket.    \n","  education = tf.contrib.layers.sparse_column_with_hash_bucket(\n","      \"education\", hash_bucket_size=50)\n","  \n","  # Sample of creating a real-valued column that can be used for numeric data.\n","  age = tf.contrib.layers.real_valued_column(\"age\") \n","\n","  # Return the set of all feature columns generated.\n","  feature_columns=[gender, education, age]\n"," \n","  return feature_columns"],"execution_count":0,"outputs":[]},{"metadata":{"id":"ZLfq6L3qR5gT","colab_type":"text"},"cell_type":"markdown","source":["### Train Model\n","\n","For the most part `define_linear_classifier` is like `define_linear_regressor` with the changes of using the log loss to optimize and the ROC curve to visualize the model quality.  As before we plot a learning curve to see if the model is converging, to help tune the learning rate, and to check if we are overfitting by looking at the loss on the validation data."]},{"metadata":{"id":"n7WDRxUo2gtN","colab_type":"code","colab":{"autoexec":{"startup":false,"wait_interval":0}}},"cell_type":"code","source":["def define_linear_classifier(learning_rate):\n","  \"\"\" Defines a linear classifer to predict the target.\n","  \n","  Args:\n","    learning_rate: A `float`, the learning rate.\n","    \n","  Returns:\n","    A linear classifier created with the given parameters.\n","  \"\"\"\n","  linear_classifier = tf.contrib.learn.LinearClassifier(\n","    feature_columns=construct_feature_columns(),\n","    optimizer=tf.train.GradientDescentOptimizer(learning_rate=learning_rate),\n","    gradient_clip_norm=5.0\n","  )  \n","  return linear_classifier\n","\n","def train_model(model, steps):\n","  \"\"\"Trains a linear classifier.\n","  \n","  Args:\n","    model: The model to train.\n","    steps: A non-zero `int`, the total number of training steps.\n","    \n","  Returns:\n","    The trained model.\n","  \"\"\"\n","  # In order to see how the model evolves as we train it, we divide the\n","  # steps into periods and show the model after each period.\n","  periods = 10\n","  steps_per_period = steps / periods\n","  \n","  # Train the model, but do so inside a loop so that we can periodically assess\n","  # loss metrics.  We store the training and validation losses so we can\n","  # generate a learning curve.\n","  print \"Training model...\"\n","  training_losses = []\n","  validation_losses = []\n","\n","  for period in range (0, periods):\n","    # Call fit to train the model for steps_per_period steps.\n","    model.fit(input_fn=train_input_fn, steps=steps_per_period)\n","    \n","    # Compute the loss between the predictions and the correct labels, append\n","    # the training and validation loss to the list of losses used to generate\n","    # the learning curve after training is complete and print the current\n","    # training loss.\n","    training_loss = compute_loss(model, train_input_fn,\n","                                 training_examples[LABEL])\n","    validation_loss = compute_loss(model, eval_input_fn,\n","                                   validation_examples[LABEL])\n","    training_losses.append(training_loss) \n","    validation_losses.append(validation_loss) \n","    print \"  Training loss after period %02d : %0.3f\" % (period, training_loss)\n","      \n","  # Now that training is done print the final training and validation losses.  \n","  print \"Final Training Loss: %0.3f\" % training_loss\n","  print \"Final Validation Loss: %0.3f\" % validation_loss \n","  \n","  # Generate a figure with the learning curve on the left and an ROC curve on\n","  # the right.\n","  plt.figure(figsize=(10, 5))\n","  plt.subplot(1, 2, 1)\n","  plt.title(\"Learning Curve (Loss vs time)\")\n","  plot_learning_curve(training_losses, validation_losses)\n","  \n","  plt.subplot(1, 2, 2)\n","  plt.tight_layout(pad=1.1, w_pad=3.0, h_pad=3.0) \n","  plt.title(\"ROC Curve on Validation Data\")\n","  validation_probabilities = np.array(list(model.predict_proba(\n","    input_fn=eval_input_fn)))\n","  # ROC curve uses the probability that the label is 1.\n","  make_roc_curve(validation_probabilities[:, 1], validation_examples[LABEL])\n","   \n","  return model"],"execution_count":0,"outputs":[]},{"metadata":{"id":"S0aDPY3NRT3F","colab_type":"text"},"cell_type":"markdown","source":["## Task 1 - Train a Linear Classifier (1/2 point)\n","\n","You will need to fix the following functions for the training to work:\n","```\n","make_roc_curve\n","plot_learning_curve\n","compute_loss\n","```\n","\n","Let's start by just training the model with the three features already set-up in `construct_feature_columns`. Without changing anything but the learning_rate and number of steps to train, train the best model you can."]},{"metadata":{"id":"siQuxXfIRQS-","colab_type":"code","colab":{"autoexec":{"startup":false,"wait_interval":0}}},"cell_type":"code","source":["LEARNING_RATE = 0.1\n","STEPS = 250\n","\n","linear_classifier = define_linear_classifier(learning_rate = LEARNING_RATE)\n","linear_classifier = train_model(linear_classifier, steps=STEPS)"],"execution_count":0,"outputs":[]},{"metadata":{"id":"_yeU3ZA8dnNJ","colab_type":"text"},"cell_type":"markdown","source":["You can get a variety of evaluation metrics on the validation data to help you understand the quality of the models you have trained.  As we mentioned above,  **AUC** (the area under the ROC curve) has the nice *probabilistic interpretation that a random positive example is predicted to be more likely to be positive than a random negative example*.  The accuracy is the fraction of examples (in the provided data set) for which the predicted value is the same as the target label when a threshold of 0.5 on the output (the probability that the example is positive) is used as a threshold for classifying an example as positive or negative.  Finally the loss is the log loss (which is the metric being optimized)."]},{"metadata":{"id":"VpL7pAwrdnNL","colab_type":"code","colab":{"autoexec":{"startup":false,"wait_interval":0}}},"cell_type":"code","source":["evaluation_metrics = linear_classifier.evaluate(\n"," input_fn=eval_input_fn, steps=1)\n","\n","print \"AUC on the validation set: %0.2f\" % evaluation_metrics['auc']\n","print \"Accuracy on the validation set: %0.2f\" % evaluation_metrics['accuracy']\n","print \"Loss on the validation set: %0.2f\" % evaluation_metrics['loss']"],"execution_count":0,"outputs":[]},{"metadata":{"id":"91yJ6WebbC57","colab_type":"text"},"cell_type":"markdown","source":["## Task 2 - Train a Better Linear Classifier (4 points)\n","\n","Using your choice of categorical and numerical features and hyperparameters, train the best linear classifier you can.\n","\n","For your convenience we have duplicated the definition of `construct_feature_columns` and included it below with the code to train the model. **So far no changes have been made -- that's what you need to do here.**\n","\n","Feel free to duplicate the code box below if you want to see the results of several different options at once.\n","\n","**WARNING: As discussed in the slides, because the log loss has a gradient that goes to infinity as your prediction approaches the target value, when training a logistic regression model with a lot of features and thus the possibility to overfit the training data, you can get a gradient that is so large that your model overflows.  If you see an error indicating that you divided by zero or a loss of NaN, then most likely this situation has occured. The way to address this problem is to introduce regularization (which you will learn how to do in TensorFlow soon). For now, the solution is to reduce the learning rate and/or the number of training steps even if that means that your model is undertrained.**"]},{"metadata":{"id":"xwp3sn3cBoD6","colab_type":"code","colab":{"autoexec":{"startup":false,"wait_interval":0}}},"cell_type":"code","source":["def construct_feature_columns():\n","  \"\"\"Construct TensorFlow Feature Columns for features\n","  \n","  Returns:\n","    A set of feature columns.\n","  \"\"\"\n","  \n","  # Sample of creating a categorical column with known values.\n","  gender = tf.contrib.layers.sparse_column_with_keys(\n","    column_name=\"gender\", keys=[\"Female\", \"Male\"])\n","  \n","  # Sample of creating a categorical columns with a hash bucket.    \n","  education = tf.contrib.layers.sparse_column_with_hash_bucket(\n","      \"education\", hash_bucket_size=50)\n","  \n","  # Sample of creating a real-valued column that can be used for numerical data.\n","  age = tf.contrib.layers.real_valued_column(\"age\") \n","\n","  # Return the set of all feature columns generated.\n","  feature_columns=[gender, education, age]\n"," \n","  return feature_columns\n","\n","LEARNING_RATE = 0.1\n","STEPS = 100\n","\n","linear_classifier = define_linear_classifier(learning_rate = LEARNING_RATE)\n","linear_classifier = train_model(linear_classifier, steps=STEPS)"],"execution_count":0,"outputs":[]},{"metadata":{"id":"OnRD9iHKZun4","colab_type":"text"},"cell_type":"markdown","source":["Let's look at the evaluation metrics for your trained model."]},{"metadata":{"id":"EqdJBt0FZuTB","colab_type":"code","colab":{"autoexec":{"startup":false,"wait_interval":0}}},"cell_type":"code","source":["evaluation_metrics = linear_classifier.evaluate(\n"," input_fn=eval_input_fn, steps=1)\n","\n","print \"AUC on the validation set: %0.2f\" % evaluation_metrics['auc']\n","print \"Accuracy on the validation set: %0.2f\" % evaluation_metrics['accuracy']\n","print \"Loss on the validation set: %0.2f\" % evaluation_metrics['loss']"],"execution_count":0,"outputs":[]},{"metadata":{"id":"ftNnIpjRG3cb","colab_type":"text"},"cell_type":"markdown","source":["Once you've picked the hyperparameters (using the validation data), you can compute these metrics on the test data."]},{"metadata":{"id":"KwSQH2BuDJ-e","colab_type":"text"},"cell_type":"markdown","source":["## Task 3 - Look at Evaluation Metrics on the Test Data (1/2 point)\n","\n","In the first code box below put and execute code to compute the accuracy and AUC of your model on the test data.  Then in your own words interpret what these numbers are telling you about how your model will perform.\n"]},{"metadata":{"id":"CF6gEgZFS16Z","colab_type":"code","colab":{"autoexec":{"startup":false,"wait_interval":0}}},"cell_type":"code","source":["# Put code here to compute accuracy and AUC on the test input. Use what is done for the validation data to guide you."],"execution_count":0,"outputs":[]},{"metadata":{"id":"DcsXs5vhCz7n","colab_type":"code","colab":{"autoexec":{"startup":false,"wait_interval":0}}},"cell_type":"code","source":["\"\"\"\n","Explain in your own words what the accuracy and AUC values are saying about how your model will perform.\n","\n","\n","\"\"\""],"execution_count":0,"outputs":[]}]}