g3doc/tutorials/graph_keras_lstm_imdb.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "24gYiJcWNlpA"
      },
      "source": [
        "##### Copyright 2019 The TensorFlow Neural Structured Learning Authors"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "ioaprt5q5US7"
      },
      "outputs": [],
      "source": [
        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ItXfxkxvosLH"
      },
      "source": [
        "# Graph regularization for sentiment classification using synthesized graphs\n",
        "\n",
        "\u003ctable class=\"tfo-notebook-buttons\" align=\"left\"\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca target=\"_blank\" href=\"https://www.tensorflow.org/neural_structured_learning/tutorials/graph_keras_lstm_imdb\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/tf_logo_32px.png\" /\u003eView on TensorFlow.org\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/neural-structured-learning/blob/master/g3doc/tutorials/graph_keras_lstm_imdb.ipynb\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" /\u003eRun in Google Colab\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca target=\"_blank\" href=\"https://github.com/tensorflow/neural-structured-learning/blob/master/g3doc/tutorials/graph_keras_lstm_imdb.ipynb\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" /\u003eView source on GitHub\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca href=\"https://storage.googleapis.com/tensorflow_docs/neural-structured-learning/g3doc/tutorials/graph_keras_lstm_imdb.ipynb\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/download_logo_32px.png\" /\u003eDownload notebook\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca href=\"https://tfhub.dev/\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/hub_logo_32px.png\" /\u003eSee TF Hub model\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "\u003c/table\u003e"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "z3otbdCMmJiJ"
      },
      "source": [
        "## Overview"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Eg62Pmz3o83v"
      },
      "source": [
        "This notebook classifies movie reviews as *positive* or *negative* using the\n",
        "text of the review. This is an example of *binary* classification, an important\n",
        "and widely applicable kind of machine learning problem.\n",
        "\n",
        "We will demonstrate the use of graph regularization in this notebook by building\n",
        "a graph from the given input. The general recipe for building a\n",
        "graph-regularized model using the Neural Structured Learning (NSL) framework\n",
        "when the input does not contain an explicit graph is as follows:\n",
        "\n",
        "1.  Create embeddings for each text sample in the input. This can be done using\n",
        "    pre-trained models such as [word2vec](https://arxiv.org/pdf/1310.4546.pdf),\n",
        "    [Swivel](https://arxiv.org/abs/1602.02215),\n",
        "    [BERT](https://arxiv.org/abs/1810.04805) etc.\n",
        "2.  Build a graph based on these embeddings by using a similarity metric such as\n",
        "    the 'L2' distance, 'cosine' distance, etc. Nodes in the graph correspond to\n",
        "    samples and edges in the graph correspond to similarity between pairs of\n",
        "    samples.\n",
        "3.  Generate training data from the above synthesized graph and sample features.\n",
        "    The resulting training data will contain neighbor features in addition to\n",
        "    the original node features.\n",
        "4.  Create a neural network as a base model using the Keras sequential,\n",
        "    functional, or subclass API.\n",
        "5.  Wrap the base model with the GraphRegularization wrapper class, which is\n",
        "    provided by the NSL framework, to create a new graph Keras model. This new\n",
        "    model will include a graph regularization loss as the regularization term in\n",
        "    its training objective.\n",
        "6.  Train and evaluate the graph Keras model.\n",
        "\n",
        "**Note**: We expect that it would take readers about 1 hour to go through this\n",
        "tutorial."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nDOFbB34KY1R"
      },
      "source": [
        "## Requirements\n",
        "\n",
        "1.  Install the Neural Structured Learning package.\n",
        "2.  Install tensorflow-hub."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "uVnjPmOaQlnH"
      },
      "outputs": [],
      "source": [
        "!pip install --quiet neural-structured-learning\n",
        "!pip install --quiet tensorflow-hub"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "x6FJ64qMNLez"
      },
      "source": [
        "## Dependencies and imports"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2ew7HTbPpCJH"
      },
      "outputs": [],
      "source": [
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "\n",
        "import neural_structured_learning as nsl\n",
        "\n",
        "import tensorflow as tf\n",
        "import tensorflow_hub as hub\n",
        "\n",
        "# Resets notebook state\n",
        "tf.keras.backend.clear_session()\n",
        "\n",
        "print(\"Version: \", tf.__version__)\n",
        "print(\"Eager mode: \", tf.executing_eagerly())\n",
        "print(\"Hub version: \", hub.__version__)\n",
        "print(\n",
        "    \"GPU is\",\n",
        "    \"available\" if tf.config.list_physical_devices(\"GPU\") else \"NOT AVAILABLE\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nGwwFd99n42P"
      },
      "source": [
        "## IMDB dataset\n",
        "\n",
        "The\n",
        "[IMDB dataset](https://www.tensorflow.org/api_docs/python/tf/keras/datasets/imdb)\n",
        "contains the text of 50,000 movie reviews from the\n",
        "[Internet Movie Database](https://www.imdb.com/). These are split into 25,000\n",
        "reviews for training and 25,000 reviews for testing. The training and testing\n",
        "sets are *balanced*, meaning they contain an equal number of positive and\n",
        "negative reviews.\n",
        "\n",
        "In this tutorial, we will use a preprocessed version of the IMDB dataset."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iAsKG535pHep"
      },
      "source": [
        "### Download preprocessed IMDB dataset\n",
        "\n",
        "The IMDB dataset comes packaged with TensorFlow. It has already been\n",
        "preprocessed such that the reviews (sequences of words) have been converted to\n",
        "sequences of integers, where each integer represents a specific word in a\n",
        "dictionary.\n",
        "\n",
        "The following code downloads the IMDB dataset (or uses a cached copy if it has\n",
        "already been downloaded):"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "zXXx5Oc3pOmN"
      },
      "outputs": [],
      "source": [
        "imdb = tf.keras.datasets.imdb\n",
        "(pp_train_data, pp_train_labels), (pp_test_data, pp_test_labels) = (\n",
        "    imdb.load_data(num_words=10000))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "odr-KlzO-lkL"
      },
      "source": [
        "The argument `num_words=10000` keeps the top 10,000 most frequently occurring words in the training data. The rare words are discarded to keep the size of the vocabulary manageable."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "l50X3GfjpU4r"
      },
      "source": [
        "### Explore the data\n",
        "\n",
        "Let's take a moment to understand the format of the data. The dataset comes preprocessed: each example is an array of integers representing the words of the movie review. Each label is an integer value of either 0 or 1, where 0 is a negative review, and 1 is a positive review."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "y8qCnve_-lkO"
      },
      "outputs": [],
      "source": [
        "print('Training entries: {}, labels: {}'.format(\n",
        "    len(pp_train_data), len(pp_train_labels)))\n",
        "training_samples_count = len(pp_train_data)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RnKvHWW4-lkW"
      },
      "source": [
        "The text of reviews have been converted to integers, where each integer represents a specific word in a dictionary. Here's what the first review looks like:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "QtTS4kpEpjbi"
      },
      "outputs": [],
      "source": [
        "print(pp_train_data[0])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hIE4l_72x7DP"
      },
      "source": [
        "Movie reviews may be different lengths. The below code shows the number of words in the first and second reviews. Since inputs to a neural network must be the same length, we'll need to resolve this later."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "X-6Ii9Pfx6Nr"
      },
      "outputs": [],
      "source": [
        "len(pp_train_data[0]), len(pp_train_data[1])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4wJg2FiYpuoX"
      },
      "source": [
        "### Convert the integers back to words\n",
        "\n",
        "It may be useful to know how to convert integers back to the corresponding text.\n",
        "Here, we'll create a helper function to query a dictionary object that contains\n",
        "the integer to string mapping:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "tr5s_1alpzop"
      },
      "outputs": [],
      "source": [
        "def build_reverse_word_index():\n",
        "  # A dictionary mapping words to an integer index\n",
        "  word_index = imdb.get_word_index()\n",
        "\n",
        "  # The first indices are reserved\n",
        "  word_index = {k: (v + 3) for k, v in word_index.items()}\n",
        "  word_index['\u003cPAD\u003e'] = 0\n",
        "  word_index['\u003cSTART\u003e'] = 1\n",
        "  word_index['\u003cUNK\u003e'] = 2  # unknown\n",
        "  word_index['\u003cUNUSED\u003e'] = 3\n",
        "  return dict((value, key) for (key, value) in word_index.items())\n",
        "\n",
        "reverse_word_index = build_reverse_word_index()\n",
        "\n",
        "def decode_review(text):\n",
        "  return ' '.join([reverse_word_index.get(i, '?') for i in text])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U3CNRvEZVppl"
      },
      "source": [
        "Now we can use the `decode_review` function to display the text for the first review:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "s_OqxmH6-lkn"
      },
      "outputs": [],
      "source": [
        "decode_review(pp_train_data[0])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rVmqL-zcWm8v"
      },
      "source": [
        "## Graph construction\n",
        "\n",
        "Graph construction involves creating embeddings for text samples and then using\n",
        "a similarity function to compare the embeddings.\n",
        "\n",
        "Before proceeding further, we first create a directory to store artifacts\n",
        "created by this tutorial."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "wZicFxFOeL2J"
      },
      "outputs": [],
      "source": [
        "!mkdir -p /tmp/imdb"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uUyHEa-3TB2X"
      },
      "source": [
        "### Create sample embeddings"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qCe9vOy7-Br9"
      },
      "source": [
        "We will use pretrained Swivel embeddings to create embeddings in the\n",
        "`tf.train.Example` format for each sample in the input. We will store the\n",
        "resulting embeddings in the `TFRecord` format along with an additional feature\n",
        "that represents the ID of each sample. This is important and will allow us match\n",
        "sample embeddings with corresponding nodes in the graph later."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "nq2Ohd9CuZv_"
      },
      "outputs": [],
      "source": [
        "pretrained_embedding = 'https://tfhub.dev/google/tf2-preview/gnews-swivel-20dim/1'\n",
        "\n",
        "hub_layer = hub.KerasLayer(\n",
        "    pretrained_embedding, input_shape=[], dtype=tf.string, trainable=True)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "wXJ3RaboTSKQ"
      },
      "outputs": [],
      "source": [
        "def _int64_feature(value):\n",
        "  \"\"\"Returns int64 tf.train.Feature.\"\"\"\n",
        "  return tf.train.Feature(int64_list=tf.train.Int64List(value=value.tolist()))\n",
        "\n",
        "\n",
        "def _bytes_feature(value):\n",
        "  \"\"\"Returns bytes tf.train.Feature.\"\"\"\n",
        "  return tf.train.Feature(\n",
        "      bytes_list=tf.train.BytesList(value=[value.encode('utf-8')]))\n",
        "\n",
        "\n",
        "def _float_feature(value):\n",
        "  \"\"\"Returns float tf.train.Feature.\"\"\"\n",
        "  return tf.train.Feature(float_list=tf.train.FloatList(value=value.tolist()))\n",
        "\n",
        "\n",
        "def create_embedding_example(word_vector, record_id):\n",
        "  \"\"\"Create tf.Example containing the sample's embedding and its ID.\"\"\"\n",
        "\n",
        "  text = decode_review(word_vector)\n",
        "\n",
        "  # Shape = [batch_size,].\n",
        "  sentence_embedding = hub_layer(tf.reshape(text, shape=[-1,]))\n",
        "\n",
        "  # Flatten the sentence embedding back to 1-D.\n",
        "  sentence_embedding = tf.reshape(sentence_embedding, shape=[-1])\n",
        "\n",
        "  features = {\n",
        "      'id': _bytes_feature(str(record_id)),\n",
        "      'embedding': _float_feature(sentence_embedding.numpy())\n",
        "  }\n",
        "  return tf.train.Example(features=tf.train.Features(feature=features))\n",
        "\n",
        "\n",
        "def create_embeddings(word_vectors, output_path, starting_record_id):\n",
        "  record_id = int(starting_record_id)\n",
        "  with tf.io.TFRecordWriter(output_path) as writer:\n",
        "    for word_vector in word_vectors:\n",
        "      example = create_embedding_example(word_vector, record_id)\n",
        "      record_id = record_id + 1\n",
        "      writer.write(example.SerializeToString())\n",
        "  return record_id\n",
        "\n",
        "\n",
        "# Persist TF.Example features containing embeddings for training data in\n",
        "# TFRecord format.\n",
        "create_embeddings(pp_train_data, '/tmp/imdb/embeddings.tfr', 0)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "C8s06RuI_vKs"
      },
      "source": [
        "### Build a graph\n",
        "\n",
        "Now that we have the sample embeddings, we will use them to build a similarity\n",
        "graph, i.e, nodes in this graph will correspond to samples and edges in this\n",
        "graph will correspond to similarity between pairs of nodes.\n",
        "\n",
        "Neural Structured Learning provides a graph building library to build a graph\n",
        "based on sample embeddings. It uses\n",
        "[**cosine similarity**](https://en.wikipedia.org/wiki/Cosine_similarity) as the\n",
        "similarity measure to compare embeddings and build edges between them. It also\n",
        "allows us to specify a similarity threshold, which can be used to discard\n",
        "dissimilar edges from the final graph. In this example, using 0.99 as the\n",
        "similarity threshold and 12345 as the random seed, we end up with a graph that\n",
        "has 429,415 bi-directional edges. Here we're using the graph builder's support\n",
        "for [locality-sensitive hashing](https://en.wikipedia.org/wiki/Locality-sensitive_hashing)\n",
        "(LSH) to speed up graph building. For details on using the graph builder's LSH\n",
        "support, see the\n",
        "[`build_graph_from_config`](https://www.tensorflow.org/neural_structured_learning/api_docs/python/nsl/tools/build_graph_from_config)\n",
        "API documentation."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "DY6lqhNkBh2Q"
      },
      "outputs": [],
      "source": [
        "graph_builder_config = nsl.configs.GraphBuilderConfig(\n",
        "    similarity_threshold=0.99, lsh_splits=32, lsh_rounds=15, random_seed=12345)\n",
        "nsl.tools.build_graph_from_config(['/tmp/imdb/embeddings.tfr'],\n",
        "                                  '/tmp/imdb/graph_99.tsv',\n",
        "                                  graph_builder_config)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4dk9xfQcK553"
      },
      "source": [
        "Each bi-directional edge is represented by two directed edges in the output TSV\n",
        "file, so that file contains 429,415 * 2 = 858,830 total lines:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "dDPwTpZcJ3zF"
      },
      "outputs": [],
      "source": [
        "!wc -l /tmp/imdb/graph_99.tsv"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "06QrEVCIlTvV"
      },
      "source": [
        "**Note:** Graph quality and by extension, embedding quality, are very important\n",
        "for graph regularization. While we have used Swivel embeddings in this notebook,\n",
        "using BERT embeddings for instance, will likely capture review semantics more\n",
        "accurately. We encourage users to use embeddings of their choice and as\n",
        "appropriate to their needs."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_USkfut69gNW"
      },
      "source": [
        "## Sample features\n",
        "\n",
        "We create sample features for our problem using the `tf.train.Example` format\n",
        "and persist them in the `TFRecord` format. Each sample will include the\n",
        "following three features:\n",
        "\n",
        "1.  **id**: The node ID of the sample.\n",
        "2.  **words**: An int64 list containing word IDs.\n",
        "3.  **label**: A singleton int64 identifying the target class of the review."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "9PcUF4_B9grB"
      },
      "outputs": [],
      "source": [
        "def create_example(word_vector, label, record_id):\n",
        "  \"\"\"Create tf.Example containing the sample's word vector, label, and ID.\"\"\"\n",
        "  features = {\n",
        "      'id': _bytes_feature(str(record_id)),\n",
        "      'words': _int64_feature(np.asarray(word_vector)),\n",
        "      'label': _int64_feature(np.asarray([label])),\n",
        "  }\n",
        "  return tf.train.Example(features=tf.train.Features(feature=features))\n",
        "\n",
        "def create_records(word_vectors, labels, record_path, starting_record_id):\n",
        "  record_id = int(starting_record_id)\n",
        "  with tf.io.TFRecordWriter(record_path) as writer:\n",
        "    for word_vector, label in zip(word_vectors, labels):\n",
        "      example = create_example(word_vector, label, record_id)\n",
        "      record_id = record_id + 1\n",
        "      writer.write(example.SerializeToString())\n",
        "  return record_id\n",
        "\n",
        "# Persist TF.Example features (word vectors and labels) for training and test\n",
        "# data in TFRecord format.\n",
        "next_record_id = create_records(pp_train_data, pp_train_labels,\n",
        "                                '/tmp/imdb/train_data.tfr', 0)\n",
        "create_records(pp_test_data, pp_test_labels, '/tmp/imdb/test_data.tfr',\n",
        "               next_record_id)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rhFO9sZ8Aa_g"
      },
      "source": [
        "## Augment training data with graph neighbors\n",
        "\n",
        "Since we have the sample features and the synthesized graph, we can generate the\n",
        "augmented training data for Neural Structured Learning. The NSL framework\n",
        "provides a library to combine the graph and the sample features to produce\n",
        "the final training data for graph regularization. The resulting training data\n",
        "will include original sample features as well as features of their corresponding\n",
        "neighbors.\n",
        "\n",
        "In this tutorial, we consider undirected edges and use a maximum of 3 neighbors\n",
        "per sample to augment training data with graph neighbors."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "lSCHj4rIBj_A"
      },
      "outputs": [],
      "source": [
        "nsl.tools.pack_nbrs(\n",
        "    '/tmp/imdb/train_data.tfr',\n",
        "    '',\n",
        "    '/tmp/imdb/graph_99.tsv',\n",
        "    '/tmp/imdb/nsl_train_data.tfr',\n",
        "    add_undirected_edges=True,\n",
        "    max_nbrs=3)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AzBWdWkBqlMy"
      },
      "source": [
        "## Base model\n",
        "\n",
        "We are now ready to build a base model without graph regularization. In order to\n",
        "build this model, we can either use embeddings that were used in building the\n",
        "graph, or we can learn new embeddings jointly along with the classification\n",
        "task. For the purpose of this notebook, we will do the latter."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kLSbRFguBUNl"
      },
      "source": [
        "### Global variables"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "zsA8HuvvwGri"
      },
      "outputs": [],
      "source": [
        "NBR_FEATURE_PREFIX = 'NL_nbr_'\n",
        "NBR_WEIGHT_SUFFIX = '_weight'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s8gMVBw6t6CI"
      },
      "source": [
        "### Hyperparameters\n",
        "\n",
        "We will use an instance of `HParams` to inclue various hyperparameters and\n",
        "constants used for training and evaluation. We briefly describe each of them\n",
        "below:\n",
        "\n",
        "-   **num_classes**: There are 2 classes -- *positive* and *negative*.\n",
        "\n",
        "-   **max_seq_length**: This is the maximum number of words considered from each\n",
        "    movie review in this example.\n",
        "\n",
        "-   **vocab_size**: This is the size of the vocabulary considered for this\n",
        "    example.\n",
        "\n",
        "-   **distance_type**: This is the distance metric used to regularize the sample\n",
        "    with its neighbors.\n",
        "\n",
        "-   **graph_regularization_multiplier**: This controls the relative weight of\n",
        "    the graph regularization term in the overall loss function.\n",
        "\n",
        "-   **num_neighbors**: The number of neighbors used for graph regularization.\n",
        "    This value has to be less than or equal to the `max_nbrs` argument used\n",
        "    above when invoking `nsl.tools.pack_nbrs`.\n",
        "\n",
        "-   **num_fc_units**: The number of units in the fully connected layer of the\n",
        "    neural network.\n",
        "\n",
        "-   **train_epochs**: The number of training epochs.\n",
        "\n",
        "-   **batch_size**: Batch size used for training and evaluation.\n",
        "\n",
        "-   **eval_steps**: The number of batches to process before deeming evaluation\n",
        "    is complete. If set to `None`, all instances in the test set are evaluated."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "YlTmug7auQ2r"
      },
      "outputs": [],
      "source": [
        "class HParams(object):\n",
        "  \"\"\"Hyperparameters used for training.\"\"\"\n",
        "  def __init__(self):\n",
        "    ### dataset parameters\n",
        "    self.num_classes = 2\n",
        "    self.max_seq_length = 256\n",
        "    self.vocab_size = 10000\n",
        "    ### neural graph learning parameters\n",
        "    self.distance_type = nsl.configs.DistanceType.L2\n",
        "    self.graph_regularization_multiplier = 0.1\n",
        "    self.num_neighbors = 2\n",
        "    ### model architecture\n",
        "    self.num_embedding_dims = 16\n",
        "    self.num_lstm_dims = 64\n",
        "    self.num_fc_units = 64\n",
        "    ### training parameters\n",
        "    self.train_epochs = 10\n",
        "    self.batch_size = 128\n",
        "    ### eval parameters\n",
        "    self.eval_steps = None  # All instances in the test set are evaluated.\n",
        "\n",
        "HPARAMS = HParams()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lFP_XKVRp4_S"
      },
      "source": [
        "### Prepare the data\n",
        "\n",
        "The reviews—the arrays of integers—must be converted to tensors before being fed\n",
        "into the neural network. This conversion can be done a couple of ways:\n",
        "\n",
        "*   Convert the arrays into vectors of `0`s and `1`s indicating word occurrence,\n",
        "    similar to a one-hot encoding. For example, the sequence `[3, 5]` would         become a `10000`-dimensional vector that is all zeros except for indices        `3` and `5`, which are ones. Then, make this the first layer in our             network—a `Dense` layer—that can handle floating point vector data. This        approach is memory intensive, though, requiring a `num_words * num_reviews`     size matrix.\n",
        "\n",
        "*   Alternatively, we can pad the arrays so they all have the same length, then\n",
        "    create an integer tensor of shape `max_length * num_reviews`. We can use an\n",
        "    embedding layer capable of handling this shape as the first layer in our\n",
        "    network.\n",
        "\n",
        "In this tutorial, we will use the second approach.\n",
        "\n",
        "Since the movie reviews must be the same length, we will use the `pad_sequence`\n",
        "function defined below to standardize the lengths."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "J5lkZVynuHWs"
      },
      "outputs": [],
      "source": [
        "def make_dataset(file_path, training=False):\n",
        "  \"\"\"Creates a `tf.data.TFRecordDataset`.\n",
        "\n",
        "  Args:\n",
        "    file_path: Name of the file in the `.tfrecord` format containing\n",
        "      `tf.train.Example` objects.\n",
        "    training: Boolean indicating if we are in training mode.\n",
        "\n",
        "  Returns:\n",
        "    An instance of `tf.data.TFRecordDataset` containing the `tf.train.Example`\n",
        "    objects.\n",
        "  \"\"\"\n",
        "\n",
        "  def pad_sequence(sequence, max_seq_length):\n",
        "    \"\"\"Pads the input sequence (a `tf.SparseTensor`) to `max_seq_length`.\"\"\"\n",
        "    pad_size = tf.maximum([0], max_seq_length - tf.shape(sequence)[0])\n",
        "    padded = tf.concat(\n",
        "        [sequence.values,\n",
        "         tf.fill((pad_size), tf.cast(0, sequence.dtype))],\n",
        "        axis=0)\n",
        "    # The input sequence may be larger than max_seq_length. Truncate down if\n",
        "    # necessary.\n",
        "    return tf.slice(padded, [0], [max_seq_length])\n",
        "\n",
        "  def parse_example(example_proto):\n",
        "    \"\"\"Extracts relevant fields from the `example_proto`.\n",
        "\n",
        "    Args:\n",
        "      example_proto: An instance of `tf.train.Example`.\n",
        "\n",
        "    Returns:\n",
        "      A pair whose first value is a dictionary containing relevant features\n",
        "      and whose second value contains the ground truth labels.\n",
        "    \"\"\"\n",
        "    # The 'words' feature is a variable length word ID vector.\n",
        "    feature_spec = {\n",
        "        'words': tf.io.VarLenFeature(tf.int64),\n",
        "        'label': tf.io.FixedLenFeature((), tf.int64, default_value=-1),\n",
        "    }\n",
        "    # We also extract corresponding neighbor features in a similar manner to\n",
        "    # the features above during training.\n",
        "    if training:\n",
        "      for i in range(HPARAMS.num_neighbors):\n",
        "        nbr_feature_key = '{}{}_{}'.format(NBR_FEATURE_PREFIX, i, 'words')\n",
        "        nbr_weight_key = '{}{}{}'.format(NBR_FEATURE_PREFIX, i,\n",
        "                                         NBR_WEIGHT_SUFFIX)\n",
        "        feature_spec[nbr_feature_key] = tf.io.VarLenFeature(tf.int64)\n",
        "\n",
        "        # We assign a default value of 0.0 for the neighbor weight so that\n",
        "        # graph regularization is done on samples based on their exact number\n",
        "        # of neighbors. In other words, non-existent neighbors are discounted.\n",
        "        feature_spec[nbr_weight_key] = tf.io.FixedLenFeature(\n",
        "            [1], tf.float32, default_value=tf.constant([0.0]))\n",
        "\n",
        "    features = tf.io.parse_single_example(example_proto, feature_spec)\n",
        "\n",
        "    # Since the 'words' feature is a variable length word vector, we pad it to a\n",
        "    # constant maximum length based on HPARAMS.max_seq_length\n",
        "    features['words'] = pad_sequence(features['words'], HPARAMS.max_seq_length)\n",
        "    if training:\n",
        "      for i in range(HPARAMS.num_neighbors):\n",
        "        nbr_feature_key = '{}{}_{}'.format(NBR_FEATURE_PREFIX, i, 'words')\n",
        "        features[nbr_feature_key] = pad_sequence(features[nbr_feature_key],\n",
        "                                                 HPARAMS.max_seq_length)\n",
        "\n",
        "    labels = features.pop('label')\n",
        "    return features, labels\n",
        "\n",
        "  dataset = tf.data.TFRecordDataset([file_path])\n",
        "  if training:\n",
        "    dataset = dataset.shuffle(10000)\n",
        "  dataset = dataset.map(parse_example)\n",
        "  dataset = dataset.batch(HPARAMS.batch_size)\n",
        "  return dataset\n",
        "\n",
        "\n",
        "train_dataset = make_dataset('/tmp/imdb/nsl_train_data.tfr', True)\n",
        "test_dataset = make_dataset('/tmp/imdb/test_data.tfr')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LLC02j2g-llC"
      },
      "source": [
        "### Build the model\n",
        "\n",
        "A neural network is created by stacking layers—this requires two main architectural decisions:\n",
        "\n",
        "* How many layers to use in the model?\n",
        "* How many *hidden units* to use for each layer?\n",
        "\n",
        "In this example, the input data consists of an array of word-indices. The labels to predict are either 0 or 1.\n",
        "\n",
        "We will use a bi-directional LSTM as our base model in this tutorial."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "xpKOoWgu-llD"
      },
      "outputs": [],
      "source": [
        "# This function exists as an alternative to the bi-LSTM model used in this\n",
        "# notebook.\n",
        "def make_feed_forward_model():\n",
        "  \"\"\"Builds a simple 2 layer feed forward neural network.\"\"\"\n",
        "  inputs = tf.keras.Input(\n",
        "      shape=(HPARAMS.max_seq_length,), dtype='int64', name='words')\n",
        "  embedding_layer = tf.keras.layers.Embedding(HPARAMS.vocab_size, 16)(inputs)\n",
        "  pooling_layer = tf.keras.layers.GlobalAveragePooling1D()(embedding_layer)\n",
        "  dense_layer = tf.keras.layers.Dense(16, activation='relu')(pooling_layer)\n",
        "  outputs = tf.keras.layers.Dense(1)(dense_layer)\n",
        "  return tf.keras.Model(inputs=inputs, outputs=outputs)\n",
        "\n",
        "\n",
        "def make_bilstm_model():\n",
        "  \"\"\"Builds a bi-directional LSTM model.\"\"\"\n",
        "  inputs = tf.keras.Input(\n",
        "      shape=(HPARAMS.max_seq_length,), dtype='int64', name='words')\n",
        "  embedding_layer = tf.keras.layers.Embedding(HPARAMS.vocab_size,\n",
        "                                              HPARAMS.num_embedding_dims)(\n",
        "                                                  inputs)\n",
        "  lstm_layer = tf.keras.layers.Bidirectional(\n",
        "      tf.keras.layers.LSTM(HPARAMS.num_lstm_dims))(\n",
        "          embedding_layer)\n",
        "  dense_layer = tf.keras.layers.Dense(\n",
        "      HPARAMS.num_fc_units, activation='relu')(\n",
        "          lstm_layer)\n",
        "  outputs = tf.keras.layers.Dense(1)(dense_layer)\n",
        "  return tf.keras.Model(inputs=inputs, outputs=outputs)\n",
        "\n",
        "\n",
        "# Feel free to use an architecture of your choice.\n",
        "model = make_bilstm_model()\n",
        "model.summary()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6PbKQ6mucuKL"
      },
      "source": [
        "The layers are effectively stacked sequentially to build the classifier:\n",
        "\n",
        "1.  The first layer is an `Input` layer which takes the integer-encoded\n",
        "    vocabulary.\n",
        "2.  The next layer is an `Embedding` layer, which takes the integer-encoded\n",
        "    vocabulary and looks up the embedding vector for each word-index. These\n",
        "    vectors are learned as the model trains. The vectors add a dimension to the\n",
        "    output array. The resulting dimensions are: `(batch, sequence, embedding)`.\n",
        "3.  Next, a bidirectional LSTM layer returns a fixed-length output vector for\n",
        "    each example.\n",
        "4.  This fixed-length output vector is piped through a fully-connected (`Dense`)\n",
        "    layer with 64 hidden units.\n",
        "5.  The last layer is densely connected with a single output node. Using the\n",
        "    `sigmoid` activation function, this value is a float between 0 and 1,\n",
        "    representing a probability, or confidence level."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0XMwnDOp-llH"
      },
      "source": [
        "### Hidden units\n",
        "\n",
        "The above model has two intermediate or \"hidden\" layers, between the input and\n",
        "output, and excluding the `Embedding` layer. The number of outputs (units,\n",
        "nodes, or neurons) is the dimension of the representational space for the layer.\n",
        "In other words, the amount of freedom the network is allowed when learning an\n",
        "internal representation.\n",
        "\n",
        "If a model has more hidden units (a higher-dimensional representation space),\n",
        "and/or more layers, then the network can learn more complex representations.\n",
        "However, it makes the network more computationally expensive and may lead to\n",
        "learning unwanted patterns—patterns that improve performance on training data\n",
        "but not on the test data. This is called *overfitting*."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "L4EqVWg4-llM"
      },
      "source": [
        "### Loss function and optimizer\n",
        "\n",
        "A model needs a loss function and an optimizer for training. Since this is a\n",
        "binary classification problem and the model outputs a probability (a single-unit\n",
        "layer with a sigmoid activation), we'll use the `binary_crossentropy` loss\n",
        "function."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Mr0GP-cQ-llN"
      },
      "outputs": [],
      "source": [
        "model.compile(\n",
        "    optimizer='adam',\n",
        "    loss=tf.keras.losses.BinaryCrossentropy(from_logits=True),\n",
        "    metrics=['accuracy'])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hCWYwkug-llQ"
      },
      "source": [
        "### Create a validation set\n",
        "\n",
        "When training, we want to check the accuracy of the model on data it hasn't seen\n",
        "before. Create a *validation set* by setting apart a fraction of the original\n",
        "training data. (Why not use the testing set now? Our goal is to develop and tune\n",
        "our model using only the training data, then use the test data just once to\n",
        "evaluate our accuracy).\n",
        "\n",
        "In this tutorial, we take roughly 10% of the initial training samples (10% of 25000) as labeled data for training and the remaining as validation data. Since the initial train/test split was 50/50 (25000 samples each), the effective train/validation/test split we now have is 5/45/50.\n",
        "\n",
        "Note that 'train_dataset' has already been batched and shuffled. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "oYTf7zkZQ-Dl"
      },
      "outputs": [],
      "source": [
        "validation_fraction = 0.9\n",
        "validation_size = int(validation_fraction *\n",
        "                      int(training_samples_count / HPARAMS.batch_size))\n",
        "print(validation_size)\n",
        "validation_dataset = train_dataset.take(validation_size)\n",
        "train_dataset = train_dataset.skip(validation_size)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "35jv_fzP-llU"
      },
      "source": [
        "### Train the model\n",
        "\n",
        "Train the model in mini-batches. While training, monitor the model's loss and accuracy on the validation set:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "BLWzgfF1xpDu"
      },
      "outputs": [],
      "source": [
        "history = model.fit(\n",
        "    train_dataset,\n",
        "    validation_data=validation_dataset,\n",
        "    epochs=HPARAMS.train_epochs,\n",
        "    verbose=1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9EEGuDVuzb5r"
      },
      "source": [
        "### Evaluate the model\n",
        "\n",
        "Now, let's see how the model performs. Two values will be returned. Loss (a number which represents our error, lower values are better), and accuracy."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6q7CoDfoCJ5h"
      },
      "outputs": [],
      "source": [
        "results = model.evaluate(test_dataset, steps=HPARAMS.eval_steps)\n",
        "print(results)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5KggXVeL-llZ"
      },
      "source": [
        "### Create a graph of accuracy/loss over time\n",
        "\n",
        "`model.fit()` returns a `History` object that contains a dictionary with everything that happened during training:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "VcvSXvhp-llb"
      },
      "outputs": [],
      "source": [
        "history_dict = history.history\n",
        "history_dict.keys()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nRKsqL40-lle"
      },
      "source": [
        "There are four entries: one for each monitored metric during training and validation. We can use these to plot the training and validation loss for comparison, as well as the training and validation accuracy:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "nGoYf2Js-lle"
      },
      "outputs": [],
      "source": [
        "acc = history_dict['accuracy']\n",
        "val_acc = history_dict['val_accuracy']\n",
        "loss = history_dict['loss']\n",
        "val_loss = history_dict['val_loss']\n",
        "\n",
        "epochs = range(1, len(acc) + 1)\n",
        "\n",
        "# \"-r^\" is for solid red line with triangle markers.\n",
        "plt.plot(epochs, loss, '-r^', label='Training loss')\n",
        "# \"-b0\" is for solid blue line with circle markers.\n",
        "plt.plot(epochs, val_loss, '-bo', label='Validation loss')\n",
        "plt.title('Training and validation loss')\n",
        "plt.xlabel('Epochs')\n",
        "plt.ylabel('Loss')\n",
        "plt.legend(loc='best')\n",
        "\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6hXx-xOv-llh"
      },
      "outputs": [],
      "source": [
        "plt.clf()   # clear figure\n",
        "\n",
        "plt.plot(epochs, acc, '-r^', label='Training acc')\n",
        "plt.plot(epochs, val_acc, '-bo', label='Validation acc')\n",
        "plt.title('Training and validation accuracy')\n",
        "plt.xlabel('Epochs')\n",
        "plt.ylabel('Accuracy')\n",
        "plt.legend(loc='best')\n",
        "\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oFEmZ5zq-llk"
      },
      "source": [
        "Notice the training loss *decreases* with each epoch and the training accuracy\n",
        "*increases* with each epoch. This is expected when using a gradient descent\n",
        "optimization—it should minimize the desired quantity on every iteration."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SymtYWWiMUum"
      },
      "source": [
        "## Graph regularization\n",
        "\n",
        "We are now ready to try graph regularization using the base model that we built\n",
        "above. We will use the `GraphRegularization` wrapper class provided by the\n",
        "Neural Structured Learning framework to wrap the base (bi-LSTM) model to include\n",
        "graph regularization. The rest of the steps for training and evaluating the\n",
        "graph-regularized model are similar to that of the base model."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pCIkVe_QFX38"
      },
      "source": [
        "### Create graph-regularized model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vuIGN8KQH0jR"
      },
      "source": [
        "To assess the incremental benefit of graph regularization, we will create a new\n",
        "base model instance. This is because `model` has already been trained for a few\n",
        "iterations, and reusing this trained model to create a graph-regularized model\n",
        "will not be a fair comparison for `model`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "WOEElnbtPzSr"
      },
      "outputs": [],
      "source": [
        "# Build a new base LSTM model.\n",
        "base_reg_model = make_bilstm_model()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "XGaDeyjEOMLC"
      },
      "outputs": [],
      "source": [
        "# Wrap the base model with graph regularization.\n",
        "graph_reg_config = nsl.configs.make_graph_reg_config(\n",
        "    max_neighbors=HPARAMS.num_neighbors,\n",
        "    multiplier=HPARAMS.graph_regularization_multiplier,\n",
        "    distance_type=HPARAMS.distance_type,\n",
        "    sum_over_axis=-1)\n",
        "graph_reg_model = nsl.keras.GraphRegularization(base_reg_model,\n",
        "                                                graph_reg_config)\n",
        "graph_reg_model.compile(\n",
        "    optimizer='adam',\n",
        "    loss=tf.keras.losses.BinaryCrossentropy(from_logits=True),\n",
        "    metrics=['accuracy'])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gSZSqJOKFdgX"
      },
      "source": [
        "### Train the model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "aONZhwc9FWoo"
      },
      "outputs": [],
      "source": [
        "graph_reg_history = graph_reg_model.fit(\n",
        "    train_dataset,\n",
        "    validation_data=validation_dataset,\n",
        "    epochs=HPARAMS.train_epochs,\n",
        "    verbose=1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xD1oHiGHFjPB"
      },
      "source": [
        "### Evaluate the model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "vdFMEfe2e5JY"
      },
      "outputs": [],
      "source": [
        "graph_reg_results = graph_reg_model.evaluate(test_dataset, steps=HPARAMS.eval_steps)\n",
        "print(graph_reg_results)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3BshURAbF49R"
      },
      "source": [
        "### Create a graph of accuracy/loss over time"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "kHxshrYLah9v"
      },
      "outputs": [],
      "source": [
        "graph_reg_history_dict = graph_reg_history.history\n",
        "graph_reg_history_dict.keys()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yBrp0Y0jHu5k"
      },
      "source": [
        "There are five entries in total in the dictionary: training loss, training\n",
        "accuracy, training graph loss, validation loss, and validation accuracy. We can\n",
        "plot them all together for comparison. Note that the graph loss is only computed\n",
        "during training."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "YhjhH4n_aprb"
      },
      "outputs": [],
      "source": [
        "acc = graph_reg_history_dict['accuracy']\n",
        "val_acc = graph_reg_history_dict['val_accuracy']\n",
        "loss = graph_reg_history_dict['loss']\n",
        "graph_loss = graph_reg_history_dict['scaled_graph_loss']\n",
        "val_loss = graph_reg_history_dict['val_loss']\n",
        "\n",
        "epochs = range(1, len(acc) + 1)\n",
        "\n",
        "plt.clf()   # clear figure\n",
        "\n",
        "# \"-r^\" is for solid red line with triangle markers.\n",
        "plt.plot(epochs, loss, '-r^', label='Training loss')\n",
        "# \"-gD\" is for solid green line with diamond markers.\n",
        "plt.plot(epochs, graph_loss, '-gD', label='Training graph loss')\n",
        "# \"-b0\" is for solid blue line with circle markers.\n",
        "plt.plot(epochs, val_loss, '-bo', label='Validation loss')\n",
        "plt.title('Training and validation loss')\n",
        "plt.xlabel('Epochs')\n",
        "plt.ylabel('Loss')\n",
        "plt.legend(loc='best')\n",
        "\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "NE0vcDiqa1Id"
      },
      "outputs": [],
      "source": [
        "plt.clf()   # clear figure\n",
        "\n",
        "plt.plot(epochs, acc, '-r^', label='Training acc')\n",
        "plt.plot(epochs, val_acc, '-bo', label='Validation acc')\n",
        "plt.title('Training and validation accuracy')\n",
        "plt.xlabel('Epochs')\n",
        "plt.ylabel('Accuracy')\n",
        "plt.legend(loc='best')\n",
        "\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Su1TOgT3mgrk"
      },
      "source": [
        "## The power of semi-supervised learning\n",
        "\n",
        "Semi-supervised learning and more specifically, graph regularization in the\n",
        "context of this tutorial, can be really powerful when the amount of training\n",
        "data is small. The lack of training data is compensated by leveraging similarity\n",
        "among the training samples, which is not possible in traditional supervised\n",
        "learning.\n",
        "\n",
        "We define ***supervision ratio*** as the ratio of training samples to the total\n",
        "number of samples which includes training, validation, and test samples. In this\n",
        "notebook, we have used a supervision ratio of 0.05 (i.e, 5% of the labeled data)\n",
        "for training both the base model as well as the graph-regularized model. We\n",
        "illustrate the impact of the supervision ratio on model accuracy in the cell\n",
        "below."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "nWWa384R5vSm"
      },
      "outputs": [],
      "source": [
        "# Accuracy values for both the Bi-LSTM model and the feed forward NN model have\n",
        "# been precomputed for the following supervision ratios.\n",
        "\n",
        "supervision_ratios = [0.3, 0.15, 0.05, 0.03, 0.02, 0.01, 0.005]\n",
        "\n",
        "model_tags = ['Bi-LSTM model', 'Feed Forward NN model']\n",
        "base_model_accs = [[84, 84, 83, 80, 65, 52, 50], [87, 86, 76, 74, 67, 52, 51]]\n",
        "graph_reg_model_accs = [[84, 84, 83, 83, 65, 63, 50],\n",
        "                        [87, 86, 80, 75, 67, 52, 50]]\n",
        "\n",
        "plt.clf()  # clear figure\n",
        "\n",
        "fig, axes = plt.subplots(1, 2)\n",
        "fig.set_size_inches((12, 5))\n",
        "\n",
        "for ax, model_tag, base_model_acc, graph_reg_model_acc in zip(\n",
        "    axes, model_tags, base_model_accs, graph_reg_model_accs):\n",
        "\n",
        "  # \"-r^\" is for solid red line with triangle markers.\n",
        "  ax.plot(base_model_acc, '-r^', label='Base model')\n",
        "  # \"-gD\" is for solid green line with diamond markers.\n",
        "  ax.plot(graph_reg_model_acc, '-gD', label='Graph-regularized model')\n",
        "  ax.set_title(model_tag)\n",
        "  ax.set_xlabel('Supervision ratio')\n",
        "  ax.set_ylabel('Accuracy(%)')\n",
        "  ax.set_ylim((25, 100))\n",
        "  ax.set_xticks(range(len(supervision_ratios)))\n",
        "  ax.set_xticklabels(supervision_ratios)\n",
        "  ax.legend(loc='best')\n",
        "\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tukoIryKugX_"
      },
      "source": [
        "It can be observed that as the superivision ratio decreases, model accuracy also\n",
        "decreases. This is true for both the base model and for the graph-regularized\n",
        "model, regardless of the model architecture used. However, notice that the\n",
        "graph-regularized model performs better than the base model for both the\n",
        "architectures. In particular, for the Bi-LSTM model, when the supervision ratio\n",
        "is 0.01, the accuracy of the graph-regularized model is **~20%** higher than\n",
        "that of the base model. This is primarily because of semi-supervised learning\n",
        "for the graph-regularized model, where structural similarity among training\n",
        "samples is used in addition to the training samples themselves."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8X4zCEyPhIp-"
      },
      "source": [
        "## Conclusion\n",
        "\n",
        "We have demonstrated the use of graph regularization using the Neural Structured\n",
        "Learning (NSL) framework even when the input does not contain an explicit graph.\n",
        "We considered the task of sentiment classification of IMDB movie reviews for\n",
        "which we synthesized a similarity graph based on review embeddings. We encourage\n",
        "users to experiment further by varying hyperparameters, the amount of\n",
        "supervision, and by using different model architectures."
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "collapsed_sections": [
        "24gYiJcWNlpA"
      ],
      "name": "Graph regularization for sentiment classification using synthesized graphs",
      "private_outputs": true,
      "provenance": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}