estimatorapi.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introducing tf.estimator\n",
    "\n",
    "**Learning Objectives**\n",
    "  - Understand where the `tf.estimator` module sits in the hierarchy of Tensorflow APIs\n",
    "  - Understand the workflow of creating a `tf.estimator` model\n",
    "    1. Create Feature Columns\n",
    "    2. Create Input Functions\n",
    "    3. Create Estimator\n",
    "    4. Train/Evaluate/Predict\n",
    "  - Understand how to swap in/out different types of Estimators\n",
    "  \n",
    "## Introduction \n",
    "Tensorflow is a hierarchical framework. The further down the hierarchy you go, the more flexibility you have, but that more code you have to write. Generally one starts at the highest level of abstraction. Then if you need additional flexibility drop down one layer.\n",
    "\n",
    "<img src='../assets/TFHierarchy.png' width='50%'>\n",
    "<sup>(image: https://www.tensorflow.org/guide/premade_estimators)</sup>\n",
    "\n",
    "In this notebook we will be operating at the highest level of Tensorflow abstraction, using the Estimator API to predict taxifare prices on the sampled dataset we created previously."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting tensorflow==1.13.1\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/ca/f2/0931c194bb98398017d52c94ee30e5e1a4082ab6af76e204856ff1fdb33e/tensorflow-1.13.1-cp35-cp35m-manylinux1_x86_64.whl (92.5MB)\n",
      "\u001b[K    100% |████████████████████████████████| 92.5MB 283kB/s eta 0:00:01  3% |█▏                              | 3.4MB 15.9MB/s eta 0:00:06    6% |██▏                             | 6.1MB 31.4MB/s eta 0:00:03    7% |██▌                             | 7.3MB 13.8MB/s eta 0:00:07    10% |███▍                            | 9.9MB 29.1MB/s eta 0:00:03    12% |████                            | 11.5MB 26.4MB/s eta 0:00:04    13% |████▍                           | 12.7MB 17.8MB/s eta 0:00:05    15% |████▉                           | 14.0MB 22.2MB/s eta 0:00:04    16% |█████▍                          | 15.5MB 29.1MB/s eta 0:00:03    19% |██████▍                         | 18.4MB 34.4MB/s eta 0:00:03    33% |██████████▉                     | 31.3MB 20.5MB/s eta 0:00:03    38% |████████████▏                   | 35.3MB 5.3MB/s eta 0:00:11    40% |█████████████                   | 37.4MB 27.7MB/s eta 0:00:02    43% |█████████████▉                  | 40.0MB 30.8MB/s eta 0:00:02    46% |██████████████▊                 | 42.6MB 27.1MB/s eta 0:00:02    47% |███████████████▏                | 43.9MB 27.7MB/s eta 0:00:02    50% |████████████████                | 46.6MB 26.4MB/s eta 0:00:02    55% |█████████████████▋              | 51.1MB 26.3MB/s eta 0:00:02    59% |███████████████████             | 54.8MB 27.7MB/s eta 0:00:02    62% |███████████████████▉            | 57.4MB 28.2MB/s eta 0:00:02    80% |█████████████████████████▋      | 74.2MB 14.3MB/s eta 0:00:02    83% |██████████████████████████▉     | 77.5MB 5.7MB/s eta 0:00:03    87% |████████████████████████████    | 81.2MB 36.1MB/s eta 0:00:01    89% |████████████████████████████▌   | 82.5MB 26.4MB/s eta 0:00:01    90% |█████████████████████████████   | 83.8MB 27.4MB/s eta 0:00:01    94% |██████████████████████████████▎ | 87.5MB 29.6MB/s eta 0:00:01\n",
      "\u001b[?25hCollecting tensorflow-estimator<1.14.0rc0,>=1.13.0 (from tensorflow==1.13.1)\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/bb/48/13f49fc3fa0fdf916aa1419013bb8f2ad09674c275b4046d5ee669a46873/tensorflow_estimator-1.13.0-py2.py3-none-any.whl (367kB)\n",
      "\u001b[K    100% |████████████████████████████████| 368kB 18.5MB/s ta 0:00:01\n",
      "\u001b[?25hRequirement already satisfied: wheel>=0.26 in /usr/local/envs/py3env/lib/python3.5/site-packages (from tensorflow==1.13.1) (0.31.1)\n",
      "Requirement already satisfied: gast>=0.2.0 in /usr/local/envs/py3env/lib/python3.5/site-packages (from tensorflow==1.13.1) (0.2.2)\n",
      "Requirement already satisfied: protobuf>=3.6.1 in /usr/local/envs/py3env/lib/python3.5/site-packages (from tensorflow==1.13.1) (3.6.1)\n",
      "Requirement already satisfied: grpcio>=1.8.6 in /usr/local/envs/py3env/lib/python3.5/site-packages (from tensorflow==1.13.1) (1.17.1)\n",
      "Collecting keras-preprocessing>=1.0.5 (from tensorflow==1.13.1)\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/c0/bf/0315ef6a9fd3fc2346e85b0ff1f5f83ca17073f2c31ac719ab2e4da0d4a3/Keras_Preprocessing-1.0.9-py2.py3-none-any.whl (59kB)\n",
      "\u001b[K    100% |████████████████████████████████| 61kB 5.2MB/s ta 0:00:01\n",
      "\u001b[?25hRequirement already satisfied: absl-py>=0.1.6 in /usr/local/envs/py3env/lib/python3.5/site-packages (from tensorflow==1.13.1) (0.6.1)\n",
      "Requirement already satisfied: six>=1.10.0 in /usr/local/envs/py3env/lib/python3.5/site-packages (from tensorflow==1.13.1) (1.10.0)\n",
      "Collecting tensorboard<1.14.0,>=1.13.0 (from tensorflow==1.13.1)\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/0f/39/bdd75b08a6fba41f098b6cb091b9e8c7a80e1b4d679a581a0ccd17b10373/tensorboard-1.13.1-py3-none-any.whl (3.2MB)\n",
      "\u001b[K    100% |████████████████████████████████| 3.2MB 5.7MB/s eta 0:00:01\n",
      "\u001b[?25hRequirement already satisfied: termcolor>=1.1.0 in /usr/local/envs/py3env/lib/python3.5/site-packages (from tensorflow==1.13.1) (1.1.0)\n",
      "Requirement already satisfied: numpy>=1.13.3 in /usr/local/envs/py3env/lib/python3.5/site-packages (from tensorflow==1.13.1) (1.14.0)\n",
      "Requirement already satisfied: astor>=0.6.0 in /usr/local/envs/py3env/lib/python3.5/site-packages (from tensorflow==1.13.1) (0.7.1)\n",
      "Collecting keras-applications>=1.0.6 (from tensorflow==1.13.1)\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/90/85/64c82949765cfb246bbdaf5aca2d55f400f792655927a017710a78445def/Keras_Applications-1.0.7-py2.py3-none-any.whl (51kB)\n",
      "\u001b[K    100% |████████████████████████████████| 61kB 9.0MB/s eta 0:00:01\n",
      "\u001b[?25hRequirement already satisfied: mock>=2.0.0 in /usr/local/envs/py3env/lib/python3.5/site-packages (from tensorflow-estimator<1.14.0rc0,>=1.13.0->tensorflow==1.13.1) (2.0.0)\n",
      "Requirement already satisfied: setuptools in /usr/local/envs/py3env/lib/python3.5/site-packages (from protobuf>=3.6.1->tensorflow==1.13.1) (40.2.0)\n",
      "Requirement already satisfied: markdown>=2.6.8 in /usr/local/envs/py3env/lib/python3.5/site-packages (from tensorboard<1.14.0,>=1.13.0->tensorflow==1.13.1) (2.6.11)\n",
      "Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/envs/py3env/lib/python3.5/site-packages (from tensorboard<1.14.0,>=1.13.0->tensorflow==1.13.1) (0.14.1)\n",
      "Requirement already satisfied: h5py in /usr/local/envs/py3env/lib/python3.5/site-packages (from keras-applications>=1.0.6->tensorflow==1.13.1) (2.7.1)\n",
      "Requirement already satisfied: pbr>=0.11 in /usr/local/envs/py3env/lib/python3.5/site-packages (from mock>=2.0.0->tensorflow-estimator<1.14.0rc0,>=1.13.0->tensorflow==1.13.1) (4.2.0)\n",
      "Installing collected packages: tensorflow-estimator, keras-preprocessing, tensorboard, keras-applications, tensorflow\n",
      "  Found existing installation: tensorboard 1.8.0\n",
      "    Uninstalling tensorboard-1.8.0:\n",
      "      Successfully uninstalled tensorboard-1.8.0\n",
      "  Found existing installation: tensorflow 1.8.0\n",
      "    Uninstalling tensorflow-1.8.0:\n",
      "      Successfully uninstalled tensorflow-1.8.0\n",
      "Successfully installed keras-applications-1.0.7 keras-preprocessing-1.0.9 tensorboard-1.13.1 tensorflow-1.13.1 tensorflow-estimator-1.13.0\n"
     ]
    }
   ],
   "source": [
    "# Ensure that we have Tensorflow 1.13.1 installed.\n",
    "!pip3 freeze | grep tensorflow==1.13.1 || pip3 install tensorflow==1.13.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/envs/py3env/lib/python3.5/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n",
      "  from ._conv import register_converters as _register_converters\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.13.1\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "import pandas as pd\n",
    "import shutil\n",
    "\n",
    "print(tf.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load raw data \n",
    "\n",
    "First let's download the raw .csv data. These are the same files created in the `create_datasets.ipynb` notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Updates are available for some Cloud SDK components.  To install them,\n",
      "please run:\n",
      "  $ gcloud components update\n",
      "\n",
      "Copying gs://cloud-training-demos/taxifare/small/taxi-test.csv...\n",
      "Copying gs://cloud-training-demos/taxifare/small/taxi-train.csv...              \n",
      "Copying gs://cloud-training-demos/taxifare/small/taxi-valid.csv...              \n",
      "/ [3 files][ 10.9 MiB/ 10.9 MiB]                                                \n",
      "Operation completed over 3 objects/10.9 MiB.                                     \n",
      "-rw-r--r-- 1 root root 1799474 May 15 01:14 taxi-test.csv\n",
      "-rw-r--r-- 1 root root 7986353 May 15 01:14 taxi-train.csv\n",
      "-rw-r--r-- 1 root root 1673742 May 15 01:14 taxi-valid.csv\n"
     ]
    }
   ],
   "source": [
    "!gsutil cp gs://cloud-training-demos/taxifare/small/*.csv .\n",
    "!ls -l *.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because the files are small we can load them into in-memory Pandas dataframes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['fare_amount', 'dayofweek', 'hourofday', 'pickuplon', 'pickuplat', 'dropofflon', 'dropofflat']\n"
     ]
    }
   ],
   "source": [
    "df_train = pd.read_csv(filepath_or_buffer = \"./taxi-train.csv\")\n",
    "df_valid = pd.read_csv(filepath_or_buffer = \"./taxi-valid.csv\")\n",
    "df_test = pd.read_csv(filepath_or_buffer = \"./taxi-test.csv\")\n",
    "\n",
    "CSV_COLUMN_NAMES = list(df_train)\n",
    "print(CSV_COLUMN_NAMES)\n",
    "\n",
    "FEATURE_NAMES = CSV_COLUMN_NAMES[1:] # all but first column\n",
    "LABEL_NAME = CSV_COLUMN_NAMES[0] # first column"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create feature columns\n",
    "\n",
    "Feature columns make it easy to perform common type of feature engineering on your raw data. For example you can one-hot encode categorical data, create feature crosses, embeddings and more. We'll cover these later in the course, but if you want to a sneak peak browse the official TensorFlow [feature columns guide](https://www.tensorflow.org/guide/feature_columns).\n",
    "\n",
    "In our case we won't do any feature engineering. However we still need to create a list of feature columns because the Estimator we will use requires one. To specify the numeric values should be passed on without modification we use `tf.feature_column.numeric_column()`\n",
    "\n",
    "#### **Exercise 1**\n",
    "\n",
    "Use a [python list comprehension](https://www.pythonforbeginners.com/basics/list-comprehensions-in-python) or a `for` loop to create the feature columns for all features in `FEATURE_NAMES`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[NumericColumn(key='dayofweek', shape=(1,), default_value=None, dtype=tf.float32, normalizer_fn=None), NumericColumn(key='hourofday', shape=(1,), default_value=None, dtype=tf.float32, normalizer_fn=None), NumericColumn(key='pickuplon', shape=(1,), default_value=None, dtype=tf.float32, normalizer_fn=None), NumericColumn(key='pickuplat', shape=(1,), default_value=None, dtype=tf.float32, normalizer_fn=None), NumericColumn(key='dropofflon', shape=(1,), default_value=None, dtype=tf.float32, normalizer_fn=None), NumericColumn(key='dropofflat', shape=(1,), default_value=None, dtype=tf.float32, normalizer_fn=None)]\n"
     ]
    }
   ],
   "source": [
    "feature_columns = [tf.feature_column.numeric_column(feat_str) for feat_str in FEATURE_NAMES]\n",
    "print(feature_columns)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define input function\n",
    "\n",
    "Now that your estimator knows what type of data to expect and how to intepret it, you need to actually pass the data to it! This is the job of the input function.\n",
    "\n",
    "The input function returns a new batch of (features, label) tuples each time it is called by the Estimator.\n",
    "\n",
    "- features: A python dictionary. Each key is a feature column name and its value is the tensor containing the data for that feature\n",
    "- label: A Tensor containing the labels\n",
    "\n",
    "So how do we get from our current Pandas dataframes to (features, label) tuples that return one batch at a time?\n",
    "\n",
    "The `tf.data` module contains a collection of classes that allows you to easily load data, manipulate it, and pipe it into your model. https://www.tensorflow.org/guide/datasets_for_estimators"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### **Exercise 2**\n",
    "\n",
    "The code cell below has a few TODOs for you to complete. \n",
    "\n",
    "The first TODO in the `train_input_fn` asks you to create a tf.dataset using the [tf.data.Dataset](https://www.tensorflow.org/api_docs/python/tf/data/Dataset) API for input pipelines. Complete the code so that the variable `dataset` creates a tf.data.Dataset element using the [tf.from_tensor_slices method](https://www.tensorflow.org/api_docs/python/tf/data/Dataset#from_tensor_slices). The argument `tensors` should be a tuple of a dict of the features and the label taken from the Pandas dataframe. \n",
    "\n",
    "The second TODO in the `train_input_fn` asks you to add a shuffle, repeat and batch operation to the dataset object you created above. Have a look at [the usage of these methods in the tf.data.Datasets API](https://www.tensorflow.org/api_docs/python/tf/data/Dataset#methods)\n",
    "\n",
    "The next TODO is in the `eval_input_fn`. Here you are asked to create a dataset object for the validation data. It should look similar to the pipeline you created for the `train_input_fn`. Note that for the `eval_input_fn` we don't add a shuffle or repeat step as we'll just evaluation a given batch during each validation step.\n",
    "\n",
    "The last TODO is in the `predict_input_fn` where you are asked to once again use the Tensorflow Dataset API to set up a dataset for the prediction stage using the same `from_tensor_slices` as before. Note, during `PREDICT` we don't have the label, only features. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_input_fn(df, batch_size = 128):\n",
    "    #1. Convert dataframe into correct (features, label) format for Estimator API\n",
    "    dataset = tf.data.Dataset.from_tensor_slices(tensors =  ( dict(df[FEATURE_NAMES]) , df[LABEL_NAME] ) )\n",
    "    #dataset = tf.estimator.inputs.pandas_input_fn(x=df, y=df[LABEL_NAME])\n",
    "    \n",
    "    # Note:\n",
    "    # If we returned now, the Dataset would iterate over the data once  \n",
    "    # in a fixed order, and only produce a single element at a time.\n",
    "    \n",
    "    #2. Shuffle, repeat, and batch the examples.\n",
    "    dataset = dataset.shuffle(buffer_size = 1000).repeat(count=None).batch(batch_size = batch_size)\n",
    "    #dataset = tf.estimator.inputs.pandas_input_fn(x=df, y=df[LABEL_NAME], batch_size=128, num_epochs=10, shuffle=True, queue_capacity=1000)\n",
    "   \n",
    "    return dataset\n",
    "\n",
    "def eval_input_fn(df, batch_size = 128):\n",
    "    #1. Convert dataframe into correct (features, label) format for Estimator API\n",
    "    dataset = tf.data.Dataset.from_tensor_slices(tensors =  ( dict(df[FEATURE_NAMES]) , df[LABEL_NAME] ) )\n",
    "    \n",
    "    #2.Batch the examples.\n",
    "    dataset = dataset.batch(batch_size = batch_size)\n",
    "   \n",
    "    return dataset\n",
    "\n",
    "def predict_input_fn(df, batch_size = 128):\n",
    "    #1. Convert dataframe into correct (features) format for Estimator API\n",
    "    dataset = tf.data.Dataset.from_tensor_slices(tensors =  ( dict(df[FEATURE_NAMES]) , df[LABEL_NAME] ) )\n",
    "    \n",
    "    #2.Batch the examples.\n",
    "    dataset = dataset.batch(batch_size = batch_size)\n",
    "   \n",
    "    return dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Choose Estimator\n",
    "\n",
    "Tensorflow has several premade estimators for you to choose from:\n",
    "\n",
    "- LinearClassifier/Regressor\n",
    "- BoostedTreesClassifier/Regressor\n",
    "- DNNClassifier/Regressor\n",
    "- DNNLinearCombinedClassifier/Regressor\n",
    "\n",
    "If none of these meet your needs you can implement a custom estimator using `tf.Keras`. We'll cover that later in the course.\n",
    "\n",
    "For now we will use the premade LinearRegressor. To instantiate an estimator simply pass it what feature columns to expect and specify an directory for it to output checkpoint files to."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### **Exercise 3**\n",
    "\n",
    "Comlete the code in the cell below to define a Linear Regression model using the TF Estimator API. Have a [look at the documentation](https://www.tensorflow.org/api_docs/python/tf/estimator/LinearRegressor#__init__) to see what variables you must pass to initialize a `LinearRegressor` instance. You'll want to add values for `feature_columns`, `model_dir` and `config`. When setting up `config`, have a look at the [documentation for tf.estimator.RunConfig](https://www.tensorflow.org/api_docs/python/tf/estimator/RunConfig) and be sure to set `tf.random_seed` to ensure reproducibility."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Using config: {'_save_summary_steps': 100, '_eval_distribute': None, '_train_distribute': None, '_num_worker_replicas': 1, '_experimental_distribute': None, '_protocol': None, '_model_dir': 'taxi_trained', '_save_checkpoints_secs': 600, '_device_fn': None, '_session_config': allow_soft_placement: true\n",
      "graph_options {\n",
      "  rewrite_options {\n",
      "    meta_optimizer_iterations: ONE\n",
      "  }\n",
      "}\n",
      ", '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f541dad6d30>, '_global_id_in_cluster': 0, '_log_step_count_steps': 100, '_tf_random_seed': 1, '_task_id': 0, '_num_ps_replicas': 0, '_is_chief': True, '_keep_checkpoint_every_n_hours': 10000, '_evaluation_master': '', '_service': None, '_master': '', '_keep_checkpoint_max': 5, '_task_type': 'worker', '_save_checkpoints_steps': None}\n"
     ]
    }
   ],
   "source": [
    "OUTDIR = \"taxi_trained\"\n",
    "\n",
    "model = tf.estimator.LinearRegressor(\n",
    "feature_columns = feature_columns, \n",
    "model_dir = OUTDIR,\n",
    "config = tf.estimator.RunConfig(tf_random_seed = 1) # for reproducibility\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train\n",
    "\n",
    "Simply invoke the estimator's `train()` function. Specify the `input_fn` which tells it how to load in data, and specify the number of steps to train for.\n",
    "\n",
    "By default estimators check the output directory for checkpoint files before beginning training, so it can pickup where it last left off. To prevent this we'll delete the output directory before starting training each time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /usr/local/envs/py3env/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Colocations handled automatically by placer.\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "WARNING:tensorflow:From /usr/local/envs/py3env/lib/python3.5/site-packages/tensorflow/python/feature_column/feature_column_v2.py:2703: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.cast instead.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Saving checkpoints for 0 into taxi_trained/model.ckpt.\n",
      "INFO:tensorflow:loss = 21212.258, step = 1\n",
      "INFO:tensorflow:global_step/sec: 319.274\n",
      "INFO:tensorflow:loss = 5491.35, step = 101 (0.318 sec)\n",
      "INFO:tensorflow:global_step/sec: 407.108\n",
      "INFO:tensorflow:loss = 18175.975, step = 201 (0.245 sec)\n",
      "INFO:tensorflow:global_step/sec: 330.958\n",
      "INFO:tensorflow:loss = 13306.74, step = 301 (0.304 sec)\n",
      "INFO:tensorflow:global_step/sec: 392.307\n",
      "INFO:tensorflow:loss = 7328.732, step = 401 (0.253 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 500 into taxi_trained/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 12039.231.\n",
      "CPU times: user 5.93 s, sys: 322 ms, total: 6.25 s\n",
      "Wall time: 6.87 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "tf.logging.set_verbosity(tf.logging.INFO) # so loss is printed during training\n",
    "shutil.rmtree(path = OUTDIR, ignore_errors = True) # start fresh each time\n",
    "\n",
    "model.train(\n",
    "    input_fn = lambda: train_input_fn(df = df_train), \n",
    "    steps = 500)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluate\n",
    "\n",
    "Estimators similarly have an `evaluate()` function. In this case we don't need to specify the number of steps to train because we didn't tell our input function to repeat the data. Once the input function reaches the end of the data evaluation will end. \n",
    "\n",
    "Loss is reported as MSE by default so we take the square root before printing."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### **Exercise 4**\n",
    "\n",
    "Complete the code in the cell below to run evaluation on the model you just trained. You'll use the `evaluate` method of the `LinearRegressor` model you created and trained above. Have a look at [the documentation of the evaluate method](https://www.tensorflow.org/api_docs/python/tf/estimator/LinearRegressor#evaluate) here to see what it expects. Note you'll need to pass the evaluation input function as a lambda function processing the Pandas dataframe `df_valid`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Starting evaluation at 2019-05-15T02:31:23Z\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "WARNING:tensorflow:From /usr/local/envs/py3env/lib/python3.5/site-packages/tensorflow/python/training/saver.py:1266: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use standard file APIs to check for files with this prefix.\n",
      "INFO:tensorflow:Restoring parameters from taxi_trained/model.ckpt-500\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Finished evaluation at 2019-05-15-02:31:24\n",
      "INFO:tensorflow:Saving dict for global step 500: average_loss = 88.96323, global_step = 500, label/mean = 11.229713, loss = 11368.714, prediction/mean = 11.383156\n",
      "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 500: taxi_trained/model.ckpt-500\n",
      "RMSE on dataset = 9.432031929460342\n"
     ]
    }
   ],
   "source": [
    "def print_rmse(model, df):\n",
    "    metrics = model.evaluate(\n",
    "        input_fn = lambda : eval_input_fn( df = df_valid)\n",
    "    )\n",
    "    print(\"RMSE on dataset = {}\".format(metrics[\"average_loss\"]**.5))\n",
    "print_rmse(model = model, df = df_valid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "RMSE of 9.43 is  worse than our rules based benchmark (RMSE of $7.70). However given that we haven't done any feature engineering or hyperparameter tuning, and we're training on a small dataset using a simple linear model, we shouldn't yet expect good performance. \n",
    "\n",
    "The goal at this point is to demonstrate the mechanics of the Estimator API. In subsequent notebooks we'll improve on the model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Predict\n",
    "\n",
    "To run prediction on the test set `df_test` we use the `predict_input_fn` you created above, passsing the `df_test` dataframe for prediction. We'll use our model to make predicitons on the first 10 elements of the `df_test` dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Restoring parameters from taxi_trained/model.ckpt-500\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "{'predictions': array([7.487807], dtype=float32)}\n",
      "{'predictions': array([7.6602535], dtype=float32)}\n",
      "{'predictions': array([7.5467978], dtype=float32)}\n",
      "{'predictions': array([7.488025], dtype=float32)}\n",
      "{'predictions': array([7.4881253], dtype=float32)}\n",
      "{'predictions': array([7.832441], dtype=float32)}\n",
      "{'predictions': array([7.8321295], dtype=float32)}\n",
      "{'predictions': array([7.656979], dtype=float32)}\n",
      "{'predictions': array([7.4861827], dtype=float32)}\n",
      "{'predictions': array([7.8308024], dtype=float32)}\n"
     ]
    }
   ],
   "source": [
    "predictions = model.predict(input_fn = lambda: predict_input_fn(df = df_test[:10]))\n",
    "for items in predictions:\n",
    "    print(items)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Further evidence of the primitiveness of our model, it predicts almost the same amount for every trip!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Change Estimator type\n",
    "\n",
    "One of the payoffs for using the Estimator API is we can swap in a different model type with just a few lines of code. Let's try a DNN. Note how now we need to specify the number of neurons in each hidden layer. Have a look at [the documentation for the DNN Regressor](https://www.tensorflow.org/api_docs/python/tf/estimator/DNNRegressor) to see what other variables you can set. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Using config: {'_save_summary_steps': 100, '_eval_distribute': None, '_train_distribute': None, '_num_worker_replicas': 1, '_experimental_distribute': None, '_protocol': None, '_model_dir': 'taxi_trained', '_save_checkpoints_secs': 600, '_device_fn': None, '_session_config': allow_soft_placement: true\n",
      "graph_options {\n",
      "  rewrite_options {\n",
      "    meta_optimizer_iterations: ONE\n",
      "  }\n",
      "}\n",
      ", '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f541e3339b0>, '_global_id_in_cluster': 0, '_log_step_count_steps': 100, '_tf_random_seed': 1, '_task_id': 0, '_num_ps_replicas': 0, '_is_chief': True, '_keep_checkpoint_every_n_hours': 10000, '_evaluation_master': '', '_service': None, '_master': '', '_keep_checkpoint_max': 5, '_task_type': 'worker', '_save_checkpoints_steps': None}\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Saving checkpoints for 0 into taxi_trained/model.ckpt.\n",
      "INFO:tensorflow:loss = 858535.1, step = 1\n",
      "INFO:tensorflow:global_step/sec: 369.842\n",
      "INFO:tensorflow:loss = 5354.631, step = 101 (0.275 sec)\n",
      "INFO:tensorflow:global_step/sec: 305.843\n",
      "INFO:tensorflow:loss = 18081.992, step = 201 (0.327 sec)\n",
      "INFO:tensorflow:global_step/sec: 462.568\n",
      "INFO:tensorflow:loss = 13290.943, step = 301 (0.216 sec)\n",
      "INFO:tensorflow:global_step/sec: 476.849\n",
      "INFO:tensorflow:loss = 7439.8286, step = 401 (0.210 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 500 into taxi_trained/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 11707.673.\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Starting evaluation at 2019-05-15T02:34:43Z\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Restoring parameters from taxi_trained/model.ckpt-500\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Finished evaluation at 2019-05-15-02:34:44\n",
      "INFO:tensorflow:Saving dict for global step 500: average_loss = 85.75572, global_step = 500, label/mean = 11.229713, loss = 10958.823, prediction/mean = 11.0708475\n",
      "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 500: taxi_trained/model.ckpt-500\n",
      "RMSE on dataset = 9.260438545009542\n",
      "CPU times: user 6.16 s, sys: 187 ms, total: 6.35 s\n",
      "Wall time: 7.04 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "tf.logging.set_verbosity(tf.logging.INFO)\n",
    "shutil.rmtree(path = OUTDIR, ignore_errors = True)\n",
    "\n",
    "model = tf.estimator.DNNRegressor(\n",
    "    hidden_units = [10,10], # specify neural architecture\n",
    "    feature_columns = feature_columns, \n",
    "    model_dir = OUTDIR,\n",
    "    config = tf.estimator.RunConfig(tf_random_seed = 1)\n",
    ")\n",
    "model.train(\n",
    "    input_fn = lambda: train_input_fn(df = df_train), \n",
    "    steps = 500)\n",
    "print_rmse(model = model, df = df_valid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our performance is only slightly better at 9.26, and still far worse than our rules based model.  This illustrates an important tenant of machine learning: A more complex model can't outrun bad data. \n",
    "\n",
    "Currently since we're not doing any feature engineering our input data has very little signal to learn from, so using a DNN doesn't help much."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Results summary\n",
    "\n",
    "We can summarize our results in a table here.\n",
    "\n",
    "#### **Exercise 5** \n",
    "\n",
    "Insert the results you found for the `LinearRegressor` and `DNNRegressor` model performance here.\n",
    "   \n",
    "|Model | RMSE on validation set|\n",
    "|------|-----------------|\n",
    "|Rules Based Benchmark| 7.76|\n",
    "|Linear Model| 9.4320319 |\n",
    "|DNN Model|9.2604385 |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Challenge exercise\n",
    "\n",
    "Create a neural network that is capable of finding the volume of a cylinder given the radius of its base (r) and its height (h). Assume that the radius and height of the cylinder are both in the range 0.5 to 2.0. Simulate the necessary training dataset.\n",
    "<p>\n",
    "Hint (highlight to see):\n",
    "<p style='color:white'>\n",
    "The input features will be r and h and the label will be $\\pi r^2 h$\n",
    "Create random values for r and h and compute V.\n",
    "Your dataset will consist of r, h and V.\n",
    "Then, use a DNN regressor.\n",
    "Make sure to generate enough data.\n",
    "</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright 2019 Google Inc. Licensed under the Apache License, Version 2.0 (the \"License\"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}