intro_to_sparse_data_and_embeddings.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "intro_to_sparse_data_and_embeddings.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": [
        "JndnmDMp66FL",
        "mNCLhxsXyOIS",
        "eQS5KQzBybTY"
      ],
      "include_colab_link": true
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "[View in Colaboratory](https://colab.research.google.com/github/ArunkumarRamanan/Exercises-Machine-Learning-Crash-Course-Google-Developers/blob/master/intro_to_sparse_data_and_embeddings.ipynb)"
      ]
    },
    {
      "metadata": {
        "id": "JndnmDMp66FL",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#### Copyright 2017 Google LLC."
      ]
    },
    {
      "metadata": {
        "id": "hMqWDc_m6rUC",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "PTaAdgy3LS8W",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Intro to Sparse Data and Embeddings\n",
        "\n",
        "**Learning Objectives:**\n",
        "* Convert movie-review string data to a sparse feature vector\n",
        "* Implement a sentiment-analysis linear model using a sparse feature vector\n",
        "* Implement a sentiment-analysis DNN model using an embedding that projects data into two dimensions\n",
        "* Visualize the embedding to see what the model has learned about the relationships between words\n",
        "\n",
        "In this exercise, we'll explore sparse data and work with embeddings using text data from movie reviews (from the [ACL 2011 IMDB dataset](http://ai.stanford.edu/~amaas/data/sentiment/)). This data has already been processed into `tf.Example` format.  "
      ]
    },
    {
      "metadata": {
        "id": "2AKGtmwNosU8",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Setup\n",
        "\n",
        "Let's import our dependencies and download the training and test data. [`tf.keras`](https://www.tensorflow.org/api_docs/python/tf/keras) includes a file download and caching tool that we can use to retrieve the data sets."
      ]
    },
    {
      "metadata": {
        "id": "jGWqDqFFL_NZ",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from __future__ import print_function\n",
        "\n",
        "import collections\n",
        "import io\n",
        "import math\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "import tensorflow as tf\n",
        "from IPython import display\n",
        "from sklearn import metrics\n",
        "\n",
        "tf.logging.set_verbosity(tf.logging.ERROR)\n",
        "train_url = 'https://dl.google.com/mlcc/mledu-datasets/sparse-data-embedding/train.tfrecord'\n",
        "train_path = tf.keras.utils.get_file(train_url.split('/')[-1], train_url)\n",
        "test_url = 'https://dl.google.com/mlcc/mledu-datasets/sparse-data-embedding/test.tfrecord'\n",
        "test_path = tf.keras.utils.get_file(test_url.split('/')[-1], test_url)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "6W7aZ9qspZVj",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Building a Sentiment Analysis Model"
      ]
    },
    {
      "metadata": {
        "id": "jieA0k_NLS8a",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Let's train a sentiment-analysis model on this data that predicts if a review is generally *favorable* (label of 1) or *unfavorable* (label of 0).\n",
        "\n",
        "To do so, we'll turn our string-value `terms` into feature vectors by using a *vocabulary*, a list of each term we expect to see in our data. For the purposes of this exercise, we've created a small vocabulary that focuses on a limited set of terms. Most of these terms were found to be strongly indicative of *favorable* or *unfavorable*, but some were just added because they're interesting.\n",
        "\n",
        "Each term in the vocabulary is mapped to a coordinate in our feature vector. To convert the string-value `terms` for an example into this vector format, we encode such that each coordinate gets a value of 0 if the vocabulary term does not appear in the example string, and a value of 1 if it does. Terms in an example that don't appear in the vocabulary are thrown away."
      ]
    },
    {
      "metadata": {
        "id": "2HSfklfnLS8b",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "**NOTE:** *We could of course use a larger vocabulary, and there are special tools for creating these. In addition, instead of just dropping terms that are not in the vocabulary, we can introduce a small number of OOV (out-of-vocabulary) buckets to which you can hash the terms not in the vocabulary. We can also use a __feature hashing__ approach that hashes each term, instead of creating an explicit vocabulary. This works well in practice, but loses interpretability, which is useful for this exercise. See see the tf.feature_column module for tools handling this.*"
      ]
    },
    {
      "metadata": {
        "id": "Uvoa2HyDtgqe",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Building the Input Pipeline"
      ]
    },
    {
      "metadata": {
        "id": "O20vMEOurDol",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "First, let's configure the input pipeline to import our data into a TensorFlow model. We can use the following function to parse the training and test data (which is in [TFRecord](https://www.tensorflow.org/programmers_guide/datasets) format) and return a dict of the features and the corresponding labels."
      ]
    },
    {
      "metadata": {
        "id": "SxxNIEniPq2z",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def _parse_function(record):\n",
        "  \"\"\"Extracts features and labels.\n",
        "  \n",
        "  Args:\n",
        "    record: File path to a TFRecord file    \n",
        "  Returns:\n",
        "    A `tuple` `(labels, features)`:\n",
        "      features: A dict of tensors representing the features\n",
        "      labels: A tensor with the corresponding labels.\n",
        "  \"\"\"\n",
        "  features = {\n",
        "    \"terms\": tf.VarLenFeature(dtype=tf.string), # terms are strings of varying lengths\n",
        "    \"labels\": tf.FixedLenFeature(shape=[1], dtype=tf.float32) # labels are 0 or 1\n",
        "  }\n",
        "  \n",
        "  parsed_features = tf.parse_single_example(record, features)\n",
        "  \n",
        "  terms = parsed_features['terms'].values\n",
        "  labels = parsed_features['labels']\n",
        "\n",
        "  return  {'terms':terms}, labels"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "SXhTeeYMrp-l",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "To confirm our function is working as expected, let's construct a `TFRecordDataset` for the training data, and map the data to features and labels using the function above."
      ]
    },
    {
      "metadata": {
        "id": "oF4YWXR0Omt0",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Create the Dataset object.\n",
        "ds = tf.data.TFRecordDataset(train_path)\n",
        "# Map features and labels with the parse function.\n",
        "ds = ds.map(_parse_function)\n",
        "\n",
        "ds"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "bUoMvK-9tVXP",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Run the following cell to retrieve the first example from the training data set."
      ]
    },
    {
      "metadata": {
        "id": "Z6QE2DWRUc4E",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "n = ds.make_one_shot_iterator().get_next()\n",
        "sess = tf.Session()\n",
        "sess.run(n)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "jBU39UeFty9S",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Now, let's build a formal input function that we can pass to the `train()` method of a TensorFlow Estimator object."
      ]
    },
    {
      "metadata": {
        "id": "5_C5-ueNYIn_",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Create an input_fn that parses the tf.Examples from the given files,\n",
        "# and split them into features and targets.\n",
        "def _input_fn(input_filenames, num_epochs=None, shuffle=True):\n",
        "  \n",
        "  # Same code as above; create a dataset and map features and labels.\n",
        "  ds = tf.data.TFRecordDataset(input_filenames)\n",
        "  ds = ds.map(_parse_function)\n",
        "\n",
        "  if shuffle:\n",
        "    ds = ds.shuffle(10000)\n",
        "\n",
        "  # Our feature data is variable-length, so we pad and batch\n",
        "  # each field of the dataset structure to whatever size is necessary.\n",
        "  ds = ds.padded_batch(25, ds.output_shapes)\n",
        "  \n",
        "  ds = ds.repeat(num_epochs)\n",
        "\n",
        "  \n",
        "  # Return the next batch of data.\n",
        "  features, labels = ds.make_one_shot_iterator().get_next()\n",
        "  return features, labels"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "Y170tVlrLS8c",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Task 1: Use a Linear Model with Sparse Inputs and an Explicit Vocabulary\n",
        "\n",
        "For our first model, we'll build a [`LinearClassifier`](https://www.tensorflow.org/api_docs/python/tf/estimator/LinearClassifier) model using 50 informative terms; always start simple!\n",
        "\n",
        "The following code constructs the feature column for our terms. The [`categorical_column_with_vocabulary_list`](https://www.tensorflow.org/api_docs/python/tf/feature_column/categorical_column_with_vocabulary_list) function creates a feature column with the string-to-feature-vector mapping."
      ]
    },
    {
      "metadata": {
        "id": "B5gdxuWsvPcx",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# 50 informative terms that compose our model vocabulary \n",
        "informative_terms = (\"bad\", \"great\", \"best\", \"worst\", \"fun\", \"beautiful\",\n",
        "                     \"excellent\", \"poor\", \"boring\", \"awful\", \"terrible\",\n",
        "                     \"definitely\", \"perfect\", \"liked\", \"worse\", \"waste\",\n",
        "                     \"entertaining\", \"loved\", \"unfortunately\", \"amazing\",\n",
        "                     \"enjoyed\", \"favorite\", \"horrible\", \"brilliant\", \"highly\",\n",
        "                     \"simple\", \"annoying\", \"today\", \"hilarious\", \"enjoyable\",\n",
        "                     \"dull\", \"fantastic\", \"poorly\", \"fails\", \"disappointing\",\n",
        "                     \"disappointment\", \"not\", \"him\", \"her\", \"good\", \"time\",\n",
        "                     \"?\", \".\", \"!\", \"movie\", \"film\", \"action\", \"comedy\",\n",
        "                     \"drama\", \"family\")\n",
        "\n",
        "terms_feature_column = tf.feature_column.categorical_column_with_vocabulary_list(key=\"terms\", vocabulary_list=informative_terms)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "eTiDwyorwd3P",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Next, we'll construct the `LinearClassifier`, train it on the training set, and evaluate it on the evaluation set. After you read through the code, run it and see how you do."
      ]
    },
    {
      "metadata": {
        "id": "HYKKpGLqLS8d",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "my_optimizer = tf.train.AdagradOptimizer(learning_rate=0.1)\n",
        "my_optimizer = tf.contrib.estimator.clip_gradients_by_norm(my_optimizer, 5.0)\n",
        "\n",
        "feature_columns = [ terms_feature_column ]\n",
        "\n",
        "\n",
        "classifier = tf.estimator.LinearClassifier(\n",
        "  feature_columns=feature_columns,\n",
        "  optimizer=my_optimizer,\n",
        ")\n",
        "\n",
        "classifier.train(\n",
        "  input_fn=lambda: _input_fn([train_path]),\n",
        "  steps=1000)\n",
        "\n",
        "evaluation_metrics = classifier.evaluate(\n",
        "  input_fn=lambda: _input_fn([train_path]),\n",
        "  steps=1000)\n",
        "print(\"Training set metrics:\")\n",
        "for m in evaluation_metrics:\n",
        "  print(m, evaluation_metrics[m])\n",
        "print(\"---\")\n",
        "\n",
        "evaluation_metrics = classifier.evaluate(\n",
        "  input_fn=lambda: _input_fn([test_path]),\n",
        "  steps=1000)\n",
        "\n",
        "print(\"Test set metrics:\")\n",
        "for m in evaluation_metrics:\n",
        "  print(m, evaluation_metrics[m])\n",
        "print(\"---\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "J0ubn9gULS8g",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Task 2: Use a Deep Neural Network (DNN) Model\n",
        "\n",
        "The above model is a linear model.  It works quite well.  But can we do better with a DNN model?\n",
        "\n",
        "Let's swap in a [`DNNClassifier`](https://www.tensorflow.org/api_docs/python/tf/estimator/DNNClassifier) for the `LinearClassifier`. Run the following cell, and see how you do."
      ]
    },
    {
      "metadata": {
        "id": "jcgOPfEALS8h",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "##################### Here's what we changed ##################################\n",
        "classifier = tf.estimator.DNNClassifier(                                      #\n",
        "  feature_columns=[tf.feature_column.indicator_column(terms_feature_column)], #\n",
        "  hidden_units=[20,20],                                                       #\n",
        "  optimizer=my_optimizer,                                                     #\n",
        ")                                                                             #\n",
        "###############################################################################\n",
        "\n",
        "try:\n",
        "  classifier.train(\n",
        "    input_fn=lambda: _input_fn([train_path]),\n",
        "    steps=1000)\n",
        "\n",
        "  evaluation_metrics = classifier.evaluate(\n",
        "    input_fn=lambda: _input_fn([train_path]),\n",
        "    steps=1)\n",
        "  print(\"Training set metrics:\")\n",
        "  for m in evaluation_metrics:\n",
        "    print(m, evaluation_metrics[m])\n",
        "  print(\"---\")\n",
        "\n",
        "  evaluation_metrics = classifier.evaluate(\n",
        "    input_fn=lambda: _input_fn([test_path]),\n",
        "    steps=1)\n",
        "\n",
        "  print(\"Test set metrics:\")\n",
        "  for m in evaluation_metrics:\n",
        "    print(m, evaluation_metrics[m])\n",
        "  print(\"---\")\n",
        "except ValueError as err:\n",
        "  print(err)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "cZz68luxLS8j",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Task 3: Use an Embedding with a DNN Model\n",
        "\n",
        "In this task, we'll implement our DNN model using an embedding column. An embedding column takes sparse data as input and returns a lower-dimensional dense vector as output."
      ]
    },
    {
      "metadata": {
        "id": "AliRzhvJLS8k",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "**NOTE:** *An embedding_column is usually the computationally most efficient option to use for training a model on sparse data. In an [optional section](#scrollTo=XDMlGgRfKSVz) at the end of this exercise, we'll discuss in more depth the implementational differences between using an `embedding_column` and an `indicator_column`, and the tradeoffs of selecting one over the other.*"
      ]
    },
    {
      "metadata": {
        "id": "F-as3PtALS8l",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "In the following code, do the following:\n",
        "\n",
        "* Define the feature columns for the model using an `embedding_column` that projects the data into 2 dimensions (see the [TF docs](https://www.tensorflow.org/api_docs/python/tf/feature_column/embedding_column) for more details on the function signature for `embedding_column`).\n",
        "* Define a `DNNClassifier` with the following specifications:\n",
        "  * Two hidden layers of 20 units each\n",
        "  * Adagrad optimization with a learning rate of 0.1\n",
        "  * A `gradient_clip_norm` of 5.0"
      ]
    },
    {
      "metadata": {
        "id": "UlPZ-Q9bLS8m",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "**NOTE:** *In practice, we might project to dimensions higher than 2, like 50 or 100.  But for now, 2 dimensions is easy to visualize.*"
      ]
    },
    {
      "metadata": {
        "id": "mNCLhxsXyOIS",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Hint"
      ]
    },
    {
      "metadata": {
        "id": "L67xYD7hLS8m",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Here's a example code snippet you might use to define the feature columns:\n",
        "\n",
        "terms_embedding_column = tf.feature_column.embedding_column(terms_feature_column, dimension=2)\n",
        "feature_columns = [ terms_embedding_column ]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "iv1UBsJxyV37",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Complete the Code Below"
      ]
    },
    {
      "metadata": {
        "id": "5PG_yhNGLS8u",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "########################## YOUR CODE HERE ######################################\n",
        "terms_embedding_column = # Define the embedding column\n",
        "feature_columns = # Define the feature columns\n",
        "\n",
        "classifier = # Define the DNNClassifier\n",
        "################################################################################\n",
        "\n",
        "classifier.train(\n",
        "  input_fn=lambda: _input_fn([train_path]),\n",
        "  steps=1000)\n",
        "\n",
        "evaluation_metrics = classifier.evaluate(\n",
        "  input_fn=lambda: _input_fn([train_path]),\n",
        "  steps=1000)\n",
        "print(\"Training set metrics:\")\n",
        "for m in evaluation_metrics:\n",
        "  print(m, evaluation_metrics[m])\n",
        "print(\"---\")\n",
        "\n",
        "evaluation_metrics = classifier.evaluate(\n",
        "  input_fn=lambda: _input_fn([test_path]),\n",
        "  steps=1000)\n",
        "\n",
        "print(\"Test set metrics:\")\n",
        "for m in evaluation_metrics:\n",
        "  print(m, evaluation_metrics[m])\n",
        "print(\"---\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "eQS5KQzBybTY",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Solution\n",
        "\n",
        "Click below for a solution."
      ]
    },
    {
      "metadata": {
        "id": "R5xOdYeQydi5",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "########################## SOLUTION CODE ########################################\n",
        "terms_embedding_column = tf.feature_column.embedding_column(terms_feature_column, dimension=2)\n",
        "feature_columns = [ terms_embedding_column ]\n",
        "\n",
        "my_optimizer = tf.train.AdagradOptimizer(learning_rate=0.1)\n",
        "my_optimizer = tf.contrib.estimator.clip_gradients_by_norm(my_optimizer, 5.0)\n",
        "\n",
        "classifier = tf.estimator.DNNClassifier(\n",
        "  feature_columns=feature_columns,\n",
        "  hidden_units=[20,20],\n",
        "  optimizer=my_optimizer\n",
        ")\n",
        "#################################################################################\n",
        "\n",
        "classifier.train(\n",
        "  input_fn=lambda: _input_fn([train_path]),\n",
        "  steps=1000)\n",
        "\n",
        "evaluation_metrics = classifier.evaluate(\n",
        "  input_fn=lambda: _input_fn([train_path]),\n",
        "  steps=1000)\n",
        "print(\"Training set metrics:\")\n",
        "for m in evaluation_metrics:\n",
        "  print(m, evaluation_metrics[m])\n",
        "print(\"---\")\n",
        "\n",
        "evaluation_metrics = classifier.evaluate(\n",
        "  input_fn=lambda: _input_fn([test_path]),\n",
        "  steps=1000)\n",
        "\n",
        "print(\"Test set metrics:\")\n",
        "for m in evaluation_metrics:\n",
        "  print(m, evaluation_metrics[m])\n",
        "print(\"---\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "aiHnnVtzLS8w",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Task 4: Convince yourself there's actually an embedding in there\n",
        "\n",
        "The above model used an `embedding_column`, and it seemed to work, but this doesn't tell us much about what's going on internally. How can we check that the model is actually using an embedding inside?\n",
        "\n",
        "To start, let's look at the tensors in the model:"
      ]
    },
    {
      "metadata": {
        "id": "h1jNgLdQLS8w",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "classifier.get_variable_names()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "Sl4-VctMLS8z",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Okay, we can see that there is an embedding layer in there: `'dnn/input_from_feature_columns/input_layer/terms_embedding/...'`. (What's interesting here, by the way, is that this layer is trainable along with the rest of the model just as any hidden layer is.)\n",
        "\n",
        "Is the embedding layer the correct shape? Run the following code to find out."
      ]
    },
    {
      "metadata": {
        "id": "JNFxyQUiLS80",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "**NOTE:** *Remember, in our case, the embedding is a matrix that allows us to project a 50-dimensional vector down to 2 dimensions.*"
      ]
    },
    {
      "metadata": {
        "id": "1xMbpcEjLS80",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "classifier.get_variable_value('dnn/input_from_feature_columns/input_layer/terms_embedding/embedding_weights').shape"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "MnLCIogjLS82",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Spend some time manually checking the various layers and shapes to make sure everything is connected the way you would expect it would be."
      ]
    },
    {
      "metadata": {
        "id": "rkKAaRWDLS83",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Task 5: Examine the Embedding\n",
        "\n",
        "Let's now take a look at the actual embedding space, and see where the terms end up in it. Do the following:\n",
        "1. Run the following code to see the embedding we trained in **Task 3**. Do things end up where you'd expect?\n",
        "\n",
        "2. Re-train the model by rerunning the code in **Task 3**, and then run the embedding visualization below again. What stays the same? What changes?\n",
        "\n",
        "3. Finally, re-train the model again using only 10 steps (which will yield a terrible model). Run the embedding visualization below again. What do you see now, and why?"
      ]
    },
    {
      "metadata": {
        "id": "s4NNu7KqLS84",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "embedding_matrix = classifier.get_variable_value('dnn/input_from_feature_columns/input_layer/terms_embedding/embedding_weights')\n",
        "\n",
        "for term_index in range(len(informative_terms)):\n",
        "  # Create a one-hot encoding for our term. It has 0s everywhere, except for\n",
        "  # a single 1 in the coordinate that corresponds to that term.\n",
        "  term_vector = np.zeros(len(informative_terms))\n",
        "  term_vector[term_index] = 1\n",
        "  # We'll now project that one-hot vector into the embedding space.\n",
        "  embedding_xy = np.matmul(term_vector, embedding_matrix)\n",
        "  plt.text(embedding_xy[0],\n",
        "           embedding_xy[1],\n",
        "           informative_terms[term_index])\n",
        "\n",
        "# Do a little setup to make sure the plot displays nicely.\n",
        "plt.rcParams[\"figure.figsize\"] = (15, 15)\n",
        "plt.xlim(1.2 * embedding_matrix.min(), 1.2 * embedding_matrix.max())\n",
        "plt.ylim(1.2 * embedding_matrix.min(), 1.2 * embedding_matrix.max())\n",
        "plt.show() "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "pUb3L7pqLS86",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Task 6:  Try to improve the model's performance\n",
        "\n",
        "See if you can refine the model to improve performance. A couple things you may want to try:\n",
        "\n",
        "* **Changing hyperparameters**, or **using a different optimizer** like Adam (you may only gain one or two accuracy percentage points following these strategies).\n",
        "* **Adding additional terms to `informative_terms`.** There's a full vocabulary file with all 30,716 terms for this data set that you can use at: https://dl.google.com/mlcc/mledu-datasets/sparse-data-embedding/terms.txt You can pick out additional terms from this vocabulary file, or use the whole thing via the `categorical_column_with_vocabulary_file` feature column."
      ]
    },
    {
      "metadata": {
        "id": "6-b3BqXvLS86",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Download the vocabulary file.\n",
        "terms_url = 'https://dl.google.com/mlcc/mledu-datasets/sparse-data-embedding/terms.txt'\n",
        "terms_path = tf.keras.utils.get_file(terms_url.split('/')[-1], terms_url)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "0jbJlwW5LS8-",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Create a feature column from \"terms\", using a full vocabulary file.\n",
        "informative_terms = None\n",
        "with io.open(terms_path, 'r', encoding='utf8') as f:\n",
        "  # Convert it to a set first to remove duplicates.\n",
        "  informative_terms = list(set(f.read().split()))\n",
        "  \n",
        "terms_feature_column = tf.feature_column.categorical_column_with_vocabulary_list(key=\"terms\", \n",
        "                                                                                 vocabulary_list=informative_terms)\n",
        "\n",
        "terms_embedding_column = tf.feature_column.embedding_column(terms_feature_column, dimension=2)\n",
        "feature_columns = [ terms_embedding_column ]\n",
        "\n",
        "my_optimizer = tf.train.AdagradOptimizer(learning_rate=0.1)\n",
        "my_optimizer = tf.contrib.estimator.clip_gradients_by_norm(my_optimizer, 5.0)\n",
        "\n",
        "classifier = tf.estimator.DNNClassifier(\n",
        "  feature_columns=feature_columns,\n",
        "  hidden_units=[10,10],\n",
        "  optimizer=my_optimizer\n",
        ")\n",
        "\n",
        "classifier.train(\n",
        "  input_fn=lambda: _input_fn([train_path]),\n",
        "  steps=1000)\n",
        "\n",
        "evaluation_metrics = classifier.evaluate(\n",
        "  input_fn=lambda: _input_fn([train_path]),\n",
        "  steps=1000)\n",
        "print(\"Training set metrics:\")\n",
        "for m in evaluation_metrics:\n",
        "  print(m, evaluation_metrics[m])\n",
        "print(\"---\")\n",
        "\n",
        "evaluation_metrics = classifier.evaluate(\n",
        "  input_fn=lambda: _input_fn([test_path]),\n",
        "  steps=1000)\n",
        "\n",
        "print(\"Test set metrics:\")\n",
        "for m in evaluation_metrics:\n",
        "  print(m, evaluation_metrics[m])\n",
        "print(\"---\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "ew3kwGM-LS9B",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## A Final Word\n",
        "\n",
        "We may have gotten a DNN solution with an embedding that was better than our original linear model, but the linear model was also pretty good and was quite a bit faster to train. Linear models train more quickly because they do not have nearly as many parameters to update or layers to backprop through.\n",
        "\n",
        "In some applications, the speed of linear models may be a game changer, or linear models may be perfectly sufficient from a quality standpoint. In other areas, the additional model complexity and capacity provided by DNNs might be more important. When defining your model architecture, remember to explore your problem sufficiently so that you know which space you're in."
      ]
    },
    {
      "metadata": {
        "id": "9MquXy9zLS9B",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### *Optional Discussion:* Trade-offs between `embedding_column` and `indicator_column`\n",
        "\n",
        "Conceptually when training a `LinearClassifier` or a `DNNClassifier`, there is an adapter needed to use a sparse column. TF provides two options: `embedding_column` or `indicator_column`.\n",
        "\n",
        "When training a LinearClassifier (as in **Task 1**), an `embedding_column` in used under the hood. As seen in **Task 2**, when training a `DNNClassifier`, you must explicitly choose either `embedding_column` or `indicator_column`. This section discusses the distinction between the two, and the trade-offs of using one over the other, by looking at a simple example."
      ]
    },
    {
      "metadata": {
        "id": "M_3XuZ_LLS9C",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Suppose we have sparse data containing the values `\"great\"`, `\"beautiful\"`, `\"excellent\"`. Since the vocabulary size we're using here is $V = 50$, each unit (neuron) in the first layer will have 50 weights. We denote the number of terms in a sparse input using $s$. So for this example sparse data, $s = 3$. For an input layer with $V$ possible values, a hidden layer with $d$ units needs to do a vector-matrix multiply: $(1 \\times V) * (V \\times d)$.  This has $O(V * d)$ computational cost. Note that this cost is proportional to the number of weights in that hidden layer and independent of $s$.\n",
        "\n",
        "If the inputs are one-hot encoded (a Boolean vector of length $V$ with a 1 for the terms present and a 0 for the rest) using an [`indicator_column`](https://www.tensorflow.org/api_docs/python/tf/feature_column/indicator_column), this means multiplying and adding a lot of zeros."
      ]
    },
    {
      "metadata": {
        "id": "I7mR4Wa2LS9C",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "When we achieve the exact same results by using an [`embedding_column`](https://www.tensorflow.org/api_docs/python/tf/feature_column/embedding_column) of size $d$, we look up and add up just the embeddings corresponding to the three features present in our example input of \"`great`\", \"`beautiful`\", \"`excellent`\": $(1 \\times d) + (1 \\times d) + (1 \\times d)$. Since the weights for the features that are absent are multiplied by zero in the vector-matrix multiply, they do not contribute to the result. Weights for the features that are present are multiplied by 1 in the vector-matrix multiply. Thus, adding the weights obtained via the embedding lookup will lead to the same result as in the vector-matrix-multiply.\n",
        "\n",
        "When using an embedding, computing the embedding lookup is an $O(s * d)$ computation, which is computationally much more efficient than the $O(V * d)$ cost for the `indicator_column` in sparse data for which $s$ is much smaller than $V$. (Remember, these embeddings are being learned. In any given training iteration it is the current weights that are being looked up.)"
      ]
    },
    {
      "metadata": {
        "id": "etZ9qf0kLS9D",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "As we saw in **Task 3**, by using an `embedding_column` in training the `DNNClassifier`, our model learns a low-dimensional representation for the features, where the dot product defines a similarity metric tailored to the desired task. In this example, terms that are used similarly in the context of movie reviews (e.g., `\"great\"` and `\"excellent\"`) will be closer to each other the embedding space (i.e., have a large dot product), and terms that are dissimilar (e.g., `\"great\"` and `\"bad\"`) will be farther away from each other in the embedding space (i.e., have a small dot product)."
      ]
    }
  ]
}