05B_NLP_Complete.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction to Natural Language Processing (NLP) in TensorFlow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Word Embeddings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Word embeddings, or word vectors, provide a way of mapping words from a vocabulary into a low-dimensional space, where words with similar meanings are close together. Let's play around with a set of pre-trained word vectors, to get used to their properties. There exist many sets of pretrained word embeddings; here, we use ConceptNet Numberbatch, which provides a relatively small download in an easy-to-work-with format (h5)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Download word vectors\n",
    "from urllib.request import urlretrieve\n",
    "import os\n",
    "if not os.path.isfile('mini.h5'):\n",
    "    print(\"Downloading Conceptnet Numberbatch word embeddings...\")\n",
    "    conceptnet_url = 'http://conceptnet.s3.amazonaws.com/precomputed-data/2016/numberbatch/17.06/mini.h5'\n",
    "    urlretrieve(conceptnet_url, 'mini.h5')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To read an `h5` file, we'll need to use the `h5py` package. Below, we use the package to open the `mini.h5` file we just downloaded. We extract from the file a list of utf-8-encoded words, as well as their $300$-dimensional vectors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: h5py in c:\\users\\kevin_000\\anaconda3\\envs\\tensorflow\\lib\\site-packages (2.7.1)\n",
      "Requirement already satisfied: numpy>=1.7 in c:\\users\\kevin_000\\anaconda3\\envs\\tensorflow\\lib\\site-packages (from h5py) (1.14.3)\n",
      "Requirement already satisfied: six in c:\\users\\kevin_000\\anaconda3\\envs\\tensorflow\\lib\\site-packages (from h5py) (1.11.0)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "tensorboard 1.8.0 has requirement bleach==1.5.0, but you'll have bleach 2.1.3 which is incompatible.\n",
      "tensorboard 1.8.0 has requirement html5lib==0.9999999, but you'll have html5lib 1.0.1 which is incompatible.\n"
     ]
    }
   ],
   "source": [
    "!pip install h5py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\kevin_000\\Anaconda3\\envs\\tensorflow\\lib\\site-packages\\h5py\\__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n",
      "  from ._conv import register_converters as _register_converters\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "all_words dimensions: 362891\n",
      "all_embeddings dimensions: (362891, 300)\n",
      "/c/de/aufmachung\n"
     ]
    }
   ],
   "source": [
    "# Load the file and pull out words and embeddings\n",
    "import h5py\n",
    "\n",
    "with h5py.File('mini.h5', 'r') as f:\n",
    "    all_words = [word.decode('utf-8') for word in f['mat']['axis1'][:]]\n",
    "    all_embeddings = f['mat']['block0_values'][:]\n",
    "    \n",
    "print(\"all_words dimensions: {0}\".format(len(all_words)))\n",
    "print(\"all_embeddings dimensions: {0}\".format(all_embeddings.shape))\n",
    "\n",
    "print(all_words[1337])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, `all_words` is a list of $V$ strings (what we call our *vocabulary*), and `all_embeddings` is a $V \\times 300$ matrix. The strings are of the form `/c/language_code/word`—for example, `/c/en/cat` and `/c/es/gato`.\n",
    "\n",
    "We are interested only in the English words. We use Python list comprehensions to pull out the indices of the English words, then extract just the English words (stripping the six-character `/c/en/` prefix) and their embeddings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "all_words dimensions: 150875\n",
      "all_embeddings dimensions: (150875, 300)\n",
      "activated_carbon\n"
     ]
    }
   ],
   "source": [
    "# Restrict our vocabulary to just the English words\n",
    "english_words = [word[6:] for word in all_words if word.startswith('/c/en/')]\n",
    "english_word_indices = [i for i, word in enumerate(all_words) if word.startswith('/c/en/')]\n",
    "english_embeddings = all_embeddings[english_word_indices]\n",
    "\n",
    "print(\"all_words dimensions: {0}\".format(len(english_words)))\n",
    "print(\"all_embeddings dimensions: {0}\".format(english_embeddings.shape))\n",
    "\n",
    "print(english_words[1337])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The magnitude of a word vector is less important than its direction; the magnitude can be thought of as representing frequency of use, independent of the semantics of the word. \n",
    "Here, we will be interested in semantics, so we *normalize* our vectors, dividing each by its length. \n",
    "The result is that all of our word vectors are length 1, and as such, lie on a unit circle. \n",
    "The dot product of two vectors is proportional to the cosine of the angle between them, and provides a measure of similarity (the bigger the cosine, the smaller the angle).\n",
    "\n",
    "<img src=\"Figures/cosine_similarity.png\" alt=\"cosine\" style=\"width: 500px;\"/>\n",
    "<center>Figure adapted from *[Mastering Machine Learning with Spark 2.x](https://www.safaribooksonline.com/library/view/mastering-machine-learning/9781785283451/ba8bef27-953e-42a4-8180-cea152af8118.xhtml)*</center>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "norms = np.linalg.norm(english_embeddings, axis=1)\n",
    "normalized_embeddings = english_embeddings.astype('float32') / norms.astype('float32').reshape([-1, 1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We want to look up words easily, so we create a dictionary that maps us from a word to its index in the word embeddings matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "index = {word: i for i, word in enumerate(english_words)}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we are ready to measure the similarity between pairs of words. We use numpy to take dot products."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cat\tcat\t 1.0000001\n",
      "cat\tfeline\t 0.8199548\n",
      "cat\tdog\t 0.590724\n",
      "cat\tmoo\t 0.0039538303\n",
      "cat\tfreeze\t -0.030225191\n",
      "antonyms\topposites\t 0.3941065\n",
      "antonyms\tsynonyms\t 0.46883982\n"
     ]
    }
   ],
   "source": [
    "def similarity_score(w1, w2):\n",
    "    score = np.dot(normalized_embeddings[index[w1], :], normalized_embeddings[index[w2], :])\n",
    "    return score\n",
    "\n",
    "# A word is as similar with itself as possible:\n",
    "print('cat\\tcat\\t', similarity_score('cat', 'cat'))\n",
    "\n",
    "# Closely related words still get high scores:\n",
    "print('cat\\tfeline\\t', similarity_score('cat', 'feline'))\n",
    "print('cat\\tdog\\t', similarity_score('cat', 'dog'))\n",
    "\n",
    "# Unrelated words, not so much\n",
    "print('cat\\tmoo\\t', similarity_score('cat', 'moo'))\n",
    "print('cat\\tfreeze\\t', similarity_score('cat', 'freeze'))\n",
    "\n",
    "# Antonyms are still considered related, sometimes more so than synonyms\n",
    "print('antonyms\\topposites\\t', similarity_score('antonym', 'opposite'))\n",
    "print('antonyms\\tsynonyms\\t', similarity_score('antonym', 'synonym'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also find, for instance, the most similar words to a given word."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def closest_to_vector(v, n):\n",
    "    all_scores = np.dot(normalized_embeddings, v)\n",
    "    best_words = map(lambda i: english_words[i], reversed(np.argsort(all_scores)))\n",
    "    return [next(best_words) for _ in range(n)]\n",
    "\n",
    "def most_similar(w, n):\n",
    "    return closest_to_vector(normalized_embeddings[index[w], :], n)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['cat', 'humane_society', 'kitten', 'feline', 'colocolo', 'cats', 'kitty', 'maine_coon', 'housecat', 'sharp_teeth']\n",
      "['dog', 'dogs', 'wire_haired_dachshund', 'doggy_paddle', 'lhasa_apso', 'good_friend', 'puppy_dog', 'bichon_frise', 'woof_woof', 'golden_retrievers']\n",
      "['duke', 'dukes', 'duchess', 'duchesses', 'ducal', 'dukedom', 'duchy', 'voivode', 'princes', 'prince']\n"
     ]
    }
   ],
   "source": [
    "print(most_similar('cat', 10))\n",
    "print(most_similar('dog', 10))\n",
    "print(most_similar('duke', 10))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also use `closest_to_vector` to find words \"nearby\" vectors that we create ourselves. This allows us to solve analogies. For example, in order to solve the analogy \"man : brother :: woman : ?\", we can compute a new vector `brother - man + woman`: the meaning of brother, minus the meaning of man, plus the meaning of woman. We can then ask which words are closest, in the embedding space, to that new vector."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['sister', 'brother', 'sisters', 'kid_sister', 'younger_brother', 'niece', 'nieces', 'sistren', 'stepsister', 'daughter']\n",
      "['wife', 'husband', 'husbands', 'spouse', 'wifes', 'wifey', 'et_ux', 'hubby', 'hotwife', 'wives']\n",
      "['paris', 'france', 'le_havre', 'in_france', 'montmartre', 'marseille', 'loire_valley', 'saone', 'lyonnais', 'jacques_chirac']\n"
     ]
    }
   ],
   "source": [
    "def solve_analogy(a1, b1, a2):\n",
    "    b2 = normalized_embeddings[index[b1], :] - normalized_embeddings[index[a1], :] + normalized_embeddings[index[a2], :]\n",
    "    return closest_to_vector(b2, 10)\n",
    "\n",
    "print(solve_analogy(\"man\", \"brother\", \"woman\"))\n",
    "print(solve_analogy(\"man\", \"husband\", \"woman\"))\n",
    "print(solve_analogy(\"spain\", \"madrid\", \"france\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These three results are quite good, but in general, the results of these analogies can be disappointing. Try experimenting with other analogies, and see if you can think of ways to get around the problems you notice (i.e., modifications to the solve_analogy algorithm)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using word embeddings in deep models\n",
    "Word embeddings are fun to play around with, but their primary use is that they allow us to think of words as existing in a continuous, Euclidean space; we can then use an existing arsenal of techniques for machine learning with continuous numerical data (like logistic regression or neural networks) to process text.\n",
    "\n",
    "Let's take a look at an especially simple version of this. We'll perform *sentiment analysis* on a set of movie reviews: in particular, we will attempt to classify a movie review as positive or negative based on its text.\n",
    "\n",
    "We will use a [Simple Word Embedding Model](http://people.ee.duke.edu/~lcarin/acl2018_swem.pdf) (SWEM, Shen et al. 2018) to do so. We will represent a review as the *mean* of the embeddings of the words in the review. Then we'll train a three-layer MLP (a neural network) to classify the review as positive or negative.\n",
    "\n",
    "Download the `movie-simple.txt` file from Google Classroom into this directory. Each line of that file contains \n",
    "\n",
    "1. the numeral 0 (for negative) or the numeral 1 (for positive), followed by\n",
    "2. a tab (the whitespace character), and then\n",
    "3. the review itself."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import string\n",
    "remove_punct=str.maketrans('','',string.punctuation)\n",
    "\n",
    "# This function converts a line of our data file into\n",
    "# a tuple (x, y), where x is 300-dimensional representation\n",
    "# of the words in a review, and y is its label.\n",
    "def convert_line_to_example(line):\n",
    "    # Pull out the first character: that's our label (0 or 1)\n",
    "    y = int(line[0])\n",
    "    \n",
    "    # Split the line into words using Python's split() function\n",
    "    words = line[2:].translate(remove_punct).lower().split()\n",
    "    \n",
    "    # Look up the embeddings of each word, ignoring words not\n",
    "    # in our pretrained vocabulary.\n",
    "    embeddings = [normalized_embeddings[index[w]] for w in words\n",
    "                  if w in index]\n",
    "    \n",
    "    # Take the mean of the embeddings\n",
    "    x = np.mean(np.vstack(embeddings), axis=0)\n",
    "    return {'x': x, 'y': y}\n",
    "\n",
    "# Apply the function to each line in the file.\n",
    "with open(\"movie-simple.txt\", \"r\", encoding='utf-8', errors='ignore') as f:\n",
    "    dataset = [convert_line_to_example(l) for l in f.readlines()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1411"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have a dataset, let's shuffle it and do a train/test split. We use a quarter of the dataset for testing, 3/4 for training (but also ensure that we have a whole number of batches in our training set, to make the code nicer later)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "random.shuffle(dataset)\n",
    "\n",
    "batch_size = 100\n",
    "total_batches = len(dataset) // batch_size\n",
    "train_batches = 3*total_batches // 4 \n",
    "train, test = dataset[:train_batches*batch_size], dataset[train_batches*batch_size:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Time to build our MLP in Tensorflow. We'll use placeholders for `X` and `y` as usual."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "# Placeholders for input\n",
    "X = tf.placeholder(tf.float32, [None, 300])\n",
    "y = tf.placeholder(tf.float32, [None, 1])\n",
    "\n",
    "# Three-layer MLP\n",
    "h1 = tf.layers.dense(X, 100, tf.nn.relu)\n",
    "h2 = tf.layers.dense(h1, 20, tf.nn.relu)\n",
    "logits = tf.layers.dense(h2, 1)\n",
    "probabilities = tf.sigmoid(logits)\n",
    "\n",
    "# Loss and metrics\n",
    "loss = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(logits=logits, labels=y))\n",
    "accuracy = tf.reduce_mean(tf.cast(tf.equal(tf.round(probabilities), y), tf.float32))\n",
    "\n",
    "# Training\n",
    "train_step = tf.train.GradientDescentOptimizer(0.05).minimize(loss)\n",
    "\n",
    "# Initialization of variables\n",
    "init_op = tf.global_variables_initializer()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now begin a session and train our model. We'll train for 250 epochs. When we're finished, we'll evaluate our accuracy on all the test data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 0 \t Loss: 0.685912013053894 \t Acc: 0.5799999833106995\n",
      "Epoch: 10 \t Loss: 0.6756182909011841 \t Acc: 0.5299999713897705\n",
      "Epoch: 20 \t Loss: 0.656575083732605 \t Acc: 0.5799999833106995\n",
      "Epoch: 30 \t Loss: 0.6393313407897949 \t Acc: 0.6399999856948853\n",
      "Epoch: 40 \t Loss: 0.6222336888313293 \t Acc: 0.6299999952316284\n",
      "Epoch: 50 \t Loss: 0.5882865786552429 \t Acc: 0.75\n",
      "Epoch: 60 \t Loss: 0.5510318279266357 \t Acc: 0.8500000238418579\n",
      "Epoch: 70 \t Loss: 0.43811488151550293 \t Acc: 0.8999999761581421\n",
      "Epoch: 80 \t Loss: 0.3752514123916626 \t Acc: 0.9100000262260437\n",
      "Epoch: 90 \t Loss: 0.30302929878234863 \t Acc: 0.9700000286102295\n",
      "Epoch: 100 \t Loss: 0.2602400481700897 \t Acc: 0.9200000166893005\n",
      "Epoch: 110 \t Loss: 0.20389726758003235 \t Acc: 0.9300000071525574\n",
      "Epoch: 120 \t Loss: 0.16896232962608337 \t Acc: 0.949999988079071\n",
      "Epoch: 130 \t Loss: 0.16247272491455078 \t Acc: 0.9800000190734863\n",
      "Epoch: 140 \t Loss: 0.1991017758846283 \t Acc: 0.9100000262260437\n",
      "Epoch: 150 \t Loss: 0.17763258516788483 \t Acc: 0.9399999976158142\n",
      "Epoch: 160 \t Loss: 0.1250958889722824 \t Acc: 0.9800000190734863\n",
      "Epoch: 170 \t Loss: 0.12842102348804474 \t Acc: 0.9800000190734863\n",
      "Epoch: 180 \t Loss: 0.24138306081295013 \t Acc: 0.8899999856948853\n",
      "Epoch: 190 \t Loss: 0.10790630429983139 \t Acc: 0.9700000286102295\n",
      "Epoch: 200 \t Loss: 0.09043605625629425 \t Acc: 0.9700000286102295\n",
      "Epoch: 210 \t Loss: 0.10588172823190689 \t Acc: 0.9800000190734863\n",
      "Epoch: 220 \t Loss: 0.12751196324825287 \t Acc: 0.9599999785423279\n",
      "Epoch: 230 \t Loss: 0.15823742747306824 \t Acc: 0.9200000166893005\n",
      "Epoch: 240 \t Loss: 0.0918678492307663 \t Acc: 0.9700000286102295\n",
      "Final accuracy: 0.9367396831512451\n"
     ]
    }
   ],
   "source": [
    "# Train\n",
    "sess = tf.Session()\n",
    "sess.run(init_op)\n",
    "\n",
    "for epoch in range(250):\n",
    "    for batch in range(train_batches):\n",
    "        data = train[batch*batch_size:(batch+1)*batch_size]\n",
    "        reviews = [sample['x'] for sample in data]\n",
    "        labels  = [sample['y'] for sample in data]\n",
    "        labels = np.array(labels).reshape([-1,1])\n",
    "        \n",
    "        _, l, acc = sess.run([train_step, loss, accuracy], feed_dict={X: reviews, y: labels})\n",
    "        \n",
    "    if epoch % 10 == 0:\n",
    "        print(\"Epoch: {0} \\t Loss: {1} \\t Acc: {2}\".format(epoch, l, acc))\n",
    "    \n",
    "    random.shuffle(train)\n",
    "        \n",
    "# Evaluate on test set\n",
    "test_reviews = [sample['x'] for sample in test]\n",
    "test_labels  = [sample['y'] for sample in test]\n",
    "test_labels  = np.array(test_labels).reshape([-1, 1])\n",
    "\n",
    "acc = sess.run(accuracy, feed_dict={X: test_reviews, y: test_labels})\n",
    "print(\"Final accuracy: {0}\".format(acc))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now examine what our model has learned, seeing how it responds to word vectors for different words:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "exciting [[0.9999993]]\n",
      "hated [[2.7804783e-06]]\n",
      "boring [[3.4574852e-05]]\n",
      "loved [[1.]]\n"
     ]
    }
   ],
   "source": [
    "# Check some words\n",
    "words_to_test = [\"exciting\", \"hated\", \"boring\", \"loved\"]\n",
    "\n",
    "for word in words_to_test:\n",
    "    print(word, sess.run(probabilities, feed_dict={X: normalized_embeddings[index[word]].reshape(1, 300)}))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Try some words of your own!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "sess.close()\n",
    "tf.reset_default_graph()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This model works great for such a simple dataset, but does a little less well on something more complex. `movie-pang02.txt`, for instance, has 2000 longer, more complex movie reviews. It's in the same format as our simple dataset. On those longer reviews, this model achieves only 60-80% accuracy. (Increasing the number of epochs to, say, 1000, does help.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Recurrent Neural Networks (RNNs)\n",
    "\n",
    "In the context of deep learning, natural language is commonly modeled with Recurrent Neural Networks (RNNs).\n",
    "RNNs pass the output of a neuron back to the input of the next time step of the same neuron.\n",
    "These directed cycles in the RNN architecture gives them the ability to model temporal dynamics, making them particularly suited for modeling sequences (e.g. text).\n",
    "We can visualize an RNN layer as follows:\n",
    "\n",
    "<img src=\"Figures/basic_RNN.PNG\" alt=\"basic_RNN\" style=\"width: 80px;\"/>\n",
    "<center>Figure from *Understanding LSTMs*. https://colah.github.io/posts/2015-08-Understanding-LSTMs/</center>\n",
    "\n",
    "We can unroll an RNN through time, making the sequence aspect of them more obvious:\n",
    "\n",
    "<img src=\"Figures/unrolled_RNN.PNG\" alt=\"basic_RNN\" style=\"width: 400px;\"/>\n",
    "<center>Figure from *Understanding LSTMs*. https://colah.github.io/posts/2015-08-Understanding-LSTMs/</center>\n",
    "\n",
    "#### RNNs in TensorFlow\n",
    "How would we implement an RNN in TensorFlow? Given the different forms of RNNs, there are quite a few ways, but we'll stick to a simple one. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# As always, import TensorFlow first\n",
    "import tensorflow as tf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's assume we have our inputs in word embedding form already, say of dimensionality 100. We'll use a minibatch size of 16."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "mb = 16\n",
    "x_dim = 100 \n",
    "\n",
    "# Inputs\n",
    "x1 = tf.placeholder(tf.float32, [mb, x_dim])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define weight matrices for projecting the input, the previous state, and the output. Rather arbitrarily, let's pick a hidden layer size of 64."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "h_dim = 64\n",
    "\n",
    "# For projecting the input\n",
    "U = tf.Variable(tf.truncated_normal([x_dim, h_dim], stddev=0.1))\n",
    "\n",
    "# For projecting the previous state\n",
    "W = tf.Variable(tf.truncated_normal([h_dim, h_dim], stddev=0.1))\n",
    "\n",
    "# For projecting the output\n",
    "V = tf.Variable(tf.truncated_normal([h_dim, x_dim], stddev=0.1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, a function for one time step of the RNN."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "def RNN_step(x, h):\n",
    "    h_next = tf.tanh(tf.matmul(x, U) + tf.matmul(h, W))\n",
    "    \n",
    "    output = tf.matmul(h_next, V)\n",
    "    return output, h_next"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Output y1 dimensions: (16, 100)\n",
      "Hidden state h1 dimensions: (16, 64)\n"
     ]
    }
   ],
   "source": [
    "# Initialize hidden state to 0\n",
    "h0 = tf.zeros([mb, h_dim])\n",
    "\n",
    "# Forward pass of one RNN step for time step t=1\n",
    "y1, h1 = RNN_step(x1, h0)\n",
    "\n",
    "print(\"Output y1 dimensions: {0}\".format(y1.shape))\n",
    "print(\"Hidden state h1 dimensions: {0}\".format(h1.shape))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can repeat using the `RNN_step` function to continue unrolling the RNN as far as we need to. For each step, we feed in the next input (a new placeholder) and get a new output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Output y2 dimensions: (16, 100)\n",
      "Hidden state h2 dimensions: (16, 64)\n"
     ]
    }
   ],
   "source": [
    "x2 = tf.placeholder(tf.float32, [mb, x_dim])\n",
    "\n",
    "# Forward pass of one RNN step for time step t=2\n",
    "y2, h2 = RNN_step(x2, h1)\n",
    "\n",
    "print(\"Output y2 dimensions: {0}\".format(y2.shape))\n",
    "print(\"Hidden state h2 dimensions: {0}\".format(h2.shape))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Of course, in practice, you'd want to do this unrolling with a `for` loop, and the RNN functionality is more cleanly wrapped up in a class. \n",
    "We're not going to implement the class version here though, as TensorFlow already has these implemented: https://www.tensorflow.org/api_guides/python/contrib.rnn#Base_interface_for_all_RNN_Cells."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x dimensions:\n",
      "[TensorShape([Dimension(16), Dimension(100)]), TensorShape([Dimension(16), Dimension(100)]), TensorShape([Dimension(16), Dimension(100)]), TensorShape([Dimension(16), Dimension(100)]), TensorShape([Dimension(16), Dimension(100)]), TensorShape([Dimension(16), Dimension(100)]), TensorShape([Dimension(16), Dimension(100)]), TensorShape([Dimension(16), Dimension(100)]), TensorShape([Dimension(16), Dimension(100)]), TensorShape([Dimension(16), Dimension(100)])]\n",
      "\n",
      "h dimensions:\n",
      "[TensorShape([Dimension(16), Dimension(64)]), TensorShape([Dimension(16), Dimension(64)]), TensorShape([Dimension(16), Dimension(64)]), TensorShape([Dimension(16), Dimension(64)]), TensorShape([Dimension(16), Dimension(64)]), TensorShape([Dimension(16), Dimension(64)]), TensorShape([Dimension(16), Dimension(64)]), TensorShape([Dimension(16), Dimension(64)]), TensorShape([Dimension(16), Dimension(64)]), TensorShape([Dimension(16), Dimension(64)])]\n"
     ]
    }
   ],
   "source": [
    "# Number of steps to unroll\n",
    "num_steps = 10\n",
    "\n",
    "# List of inputs and hidden states\n",
    "xs = []\n",
    "hs = []\n",
    "\n",
    "# Build RNN\n",
    "rnn = tf.contrib.rnn.BasicRNNCell(h_dim)\n",
    "\n",
    "# Initialize hidden state to zero\n",
    "h_t  = tf.zeros([mb, h_dim])\n",
    "\n",
    "for t in range(num_steps):\n",
    "    x_t = tf.placeholder(tf.float32, [mb, x_dim])\n",
    "    h_t, _ = rnn(x_t, h_t)\n",
    "    \n",
    "    xs.append(x_t)\n",
    "    hs.append(h_t)\n",
    "    \n",
    "print(\"x dimensions:\")\n",
    "print([x_t.shape for x_t in xs])\n",
    "print(\"\\nh dimensions:\")\n",
    "print([h_t.shape for h_t in hs])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Long Short-Term Memory (LSTM)\n",
    "One popular type of RNNs are Long Short-Term Memory (LSTM) networks.\n",
    "We're not going to go into detail here about what structural differences they have from vanilla RNNs, but LSTMs are also sequence modeling neural networks, with much better long range model capabilities.\n",
    "If you're curious, [this](https://colah.github.io/posts/2015-08-Understanding-LSTMs/) does a fantastic job describing them."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Other materials:\n",
    "Like Reinforcement Learning, Natural Language Processing can also easily be several full courses on its own at most universities, both with or without neural networks.\n",
    "In fact, Prof Mohit Bansal has [taught](http://www.cs.unc.edu/~mbansal/teaching/nlp-course-fall17.html) [several](http://www.cs.unc.edu/~mbansal/teaching/nlp-seminar-spring18.html).\n",
    "\n",
    "- [Fantastic introduction to LSTMs](https://colah.github.io/posts/2015-08-Understanding-LSTMs/)\n",
    "- [Popular blog post on RNNs](http://karpathy.github.io/2015/05/21/rnn-effectiveness/)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}