1. LSTM Structure.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# LSTM Structure and Hidden State\n",
    "\n",
    "We know that RNNs are used to maintain a kind of memory by linking the output of one node to the input of the next. In the case of an LSTM, for each piece of data in a sequence (say, for a word in a given sentence),\n",
    "there is a corresponding *hidden state* $h_t$. This hidden state is a function of the pieces of data that an LSTM has seen over time; it contains some weights and, represents both the short term and long term memory components for the data that the LSTM has already seen. \n",
    "\n",
    "So, for an LSTM that is looking at words in a sentence, **the hidden state of the LSTM will change based on each new word it sees. And, we can use the hidden state to predict the next word in a sequence** or help identify the type of word in a language model, and lots of other things!\n",
    "\n",
    "\n",
    "## LSTMs in Pytorch\n",
    "\n",
    "To create and train an LSTM, you have to know how to structure the inputs, and hidden state of an LSTM. In PyTorch an LSTM can be defined as: `lstm = nn.LSTM(input_size=input_dim, hidden_size=hidden_dim, num_layers=n_layers)`.\n",
    "\n",
    "In PyTorch, an LSTM expects all of its inputs to be 3D tensors, with dimensions defined as follows:\n",
    ">* `input_dim` = the number of inputs (a dimension of 20 could represent 20 inputs)\n",
    ">* `hidden_dim` = the size of the hidden state; this will be the number of outputs that each LSTM cell produces at each time step.\n",
    ">* `n_layers ` = the number of hidden LSTM layers to use; this is typically  a value between 1 and 3; a value of 1 means that each LSTM cell has one hidden state. This has a default value of 1.\n",
    "\n",
    "<img src='images/lstm_simple_ex.png' height=5 >\n",
    "    \n",
    "### Hidden State\n",
    "\n",
    "Once an LSTM has been defined with input and hidden dimensions, we can call it and retrieve the output and hidden state at every time step.\n",
    " `out, hidden = lstm(input.view(1, 1, -1), (h0, c0))` \n",
    "\n",
    "The inputs to an LSTM are **`(input, (h0, c0))`**.\n",
    ">* `input` = a Tensor containing the values in an input sequence; this has values: (seq_len, batch, input_size)\n",
    ">* `h0` = a Tensor containing the initial hidden state for each element in a batch\n",
    ">* `c0` = a Tensor containing the initial cell memory for each element in the batch\n",
    "\n",
    "`h0` nd `c0` will default to 0, if they are not specified. Their dimensions are: (n_layers, batch, hidden_dim).\n",
    "\n",
    "These will become clearer in the example in this notebook. This and the following notebook are modified versions of [this PyTorch LSTM tutorial](https://pytorch.org/tutorials/beginner/nlp/sequence_models_tutorial.html#lstm-s-in-pytorch).\n",
    "\n",
    "Let's take a simple example and say we want to process a single sentence through an LSTM. If we want to run the sequence model over one sentence \"Giraffes in a field\", our input should look like this `1x4` row vector of individual words:\n",
    "\n",
    "\\begin{align}\\begin{bmatrix}\n",
    "   \\text{Giraffes  } \n",
    "   \\text{in  } \n",
    "   \\text{a  } \n",
    "   \\text{field} \n",
    "   \\end{bmatrix}\\end{align}\n",
    "\n",
    "In this case, we know that we have **4 inputs words** and we decide how many outputs to generate at each time step, say we want each LSTM cell to generate **3 hidden state values**. We'll keep the number of layers in our LSTM at the default size of 1.\n",
    "\n",
    "The hidden state and cell memory will have dimensions (n_layers, batch, hidden_dim), and in this case that will be (1, 1, 3) for a 1 layer model with one batch/sequence of words to process (this one sentence) and 3 genereated, hidden state values.\n",
    "\n",
    "\n",
    "### Example Code\n",
    "\n",
    "Next, let's see an example of one LSTM that is designed to look at a sequence of 4 values (numerical values since those are easiest to create and track) and generate 3 values as output. This is what the sentence processing network from above will look like, and you are encouraged to change these input/hidden-state sizes to see the effect on the structure of the LSTM!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<torch._C.Generator at 0x193b1443270>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "%matplotlib inline\n",
    "\n",
    "torch.manual_seed(2) # so that random variables will be consistent and repeatable for testing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define a simple LSTM\n",
    "\n",
    "\n",
    "**A note on hidden and output dimensions**\n",
    "\n",
    "The `hidden_dim` and size of the output will be the same unless you define your own LSTM and change the number of outputs by adding a linear layer at the end of the network, ex. fc = nn.Linear(hidden_dim, output_dim)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "inputs: \n",
      " [tensor([[1.4934, 0.4987, 0.2319, 1.1746]]), tensor([[-1.3967,  0.8998,  1.0956, -0.5231]]), tensor([[-0.8462, -0.9946,  0.6311,  0.5327]]), tensor([[-0.8454,  0.9406, -2.1224,  0.0233]]), tensor([[ 0.4836,  1.2895,  0.8957, -0.2465]])]\n",
      "\n",
      "\n",
      "out: \n",
      " tensor([[[-0.4372,  0.2583,  0.2947]]], grad_fn=<StackBackward>)\n",
      "hidden: \n",
      " (tensor([[[-0.4372,  0.2583,  0.2947]]], grad_fn=<StackBackward>), tensor([[[-0.7344,  0.6209,  0.4191]]], grad_fn=<StackBackward>))\n",
      "out: \n",
      " tensor([[[-0.2836,  0.1314,  0.4133]]], grad_fn=<StackBackward>)\n",
      "hidden: \n",
      " (tensor([[[-0.2836,  0.1314,  0.4133]]], grad_fn=<StackBackward>), tensor([[[-0.5041,  0.2672,  0.6370]]], grad_fn=<StackBackward>))\n",
      "out: \n",
      " tensor([[[-0.3404,  0.4880,  0.1949]]], grad_fn=<StackBackward>)\n",
      "hidden: \n",
      " (tensor([[[-0.3404,  0.4880,  0.1949]]], grad_fn=<StackBackward>), tensor([[[-0.5552,  0.7909,  0.3300]]], grad_fn=<StackBackward>))\n",
      "out: \n",
      " tensor([[[-0.3544,  0.2405,  0.3150]]], grad_fn=<StackBackward>)\n",
      "hidden: \n",
      " (tensor([[[-0.3544,  0.2405,  0.3150]]], grad_fn=<StackBackward>), tensor([[[-0.5645,  1.0073,  0.6101]]], grad_fn=<StackBackward>))\n",
      "out: \n",
      " tensor([[[-0.3328,  0.0437,  0.3817]]], grad_fn=<StackBackward>)\n",
      "hidden: \n",
      " (tensor([[[-0.3328,  0.0437,  0.3817]]], grad_fn=<StackBackward>), tensor([[[-0.5311,  0.1181,  0.5304]]], grad_fn=<StackBackward>))\n"
     ]
    }
   ],
   "source": [
    "from torch.autograd import Variable\n",
    "\n",
    "# define an LSTM with an input dim of 4 and hidden dim of 3\n",
    "# this expects to see 4 values as input and generates 3 values as output\n",
    "input_dim = 4\n",
    "hidden_dim = 3\n",
    "lstm = nn.LSTM(input_size=input_dim, hidden_size=hidden_dim)  \n",
    "\n",
    "# make 5 input sequences of 4 random values each\n",
    "inputs_list = [torch.randn(1, input_dim) for _ in range(5)]\n",
    "print('inputs: \\n', inputs_list)\n",
    "print('\\n')\n",
    "\n",
    "# initialize the hidden state\n",
    "# (1 layer, 1 batch_size, 3 outputs)\n",
    "# first tensor is the hidden state, h0\n",
    "# second tensor initializes the cell memory, c0\n",
    "h0 = torch.randn(1, 1, hidden_dim)\n",
    "c0 = torch.randn(1, 1, hidden_dim)\n",
    "\n",
    "\n",
    "h0 = Variable(h0)\n",
    "c0 = Variable(c0)\n",
    "# step through the sequence one element at a time.\n",
    "for i in inputs_list:\n",
    "    # wrap in Variable \n",
    "    i = Variable(i)\n",
    "    \n",
    "    # after each step, hidden contains the hidden state\n",
    "    out, hidden = lstm(i.view(1, 1, -1), (h0, c0))\n",
    "    print('out: \\n', out)\n",
    "    print('hidden: \\n', hidden)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should see that the output and hidden Tensors are always of length 3, which we specified when we defined the LSTM with `hidden_dim`. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### All at once\n",
    "\n",
    "A for loop is not very efficient for large sequences of data, so we can also, **process all of these inputs at once.**\n",
    "\n",
    "1. concatenate all our input sequences into one big tensor, with a defined batch_size\n",
    "2. define the shape of our hidden state\n",
    "3. get the outputs and the *most recent* hidden state (created after the last word in the sequence has been seen)\n",
    "\n",
    "\n",
    "The outputs may look slightly different due to our differently initialized hidden state."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "inputs size: \n",
      " torch.Size([5, 1, 4])\n",
      "\n",
      "\n",
      "inputs: \n",
      " tensor([[[ 1.4934,  0.4987,  0.2319,  1.1746]],\n",
      "\n",
      "        [[-1.3967,  0.8998,  1.0956, -0.5231]],\n",
      "\n",
      "        [[-0.8462, -0.9946,  0.6311,  0.5327]],\n",
      "\n",
      "        [[-0.8454,  0.9406, -2.1224,  0.0233]],\n",
      "\n",
      "        [[ 0.4836,  1.2895,  0.8957, -0.2465]]])\n",
      "\n",
      "\n",
      "out: \n",
      " tensor([[[ 0.1611,  0.2200,  0.2213]],\n",
      "\n",
      "        [[ 0.0364, -0.0390,  0.2638]],\n",
      "\n",
      "        [[-0.1425, -0.0174,  0.1504]],\n",
      "\n",
      "        [[-0.1583,  0.1264,  0.1709]],\n",
      "\n",
      "        [[-0.2007, -0.1559,  0.2489]]], grad_fn=<StackBackward>)\n",
      "hidden: \n",
      " (tensor([[[-0.2007, -0.1559,  0.2489]]], grad_fn=<StackBackward>), tensor([[[-0.4429, -0.2975,  0.3252]]], grad_fn=<StackBackward>))\n"
     ]
    }
   ],
   "source": [
    "# turn inputs into a tensor with 5 rows of data\n",
    "# add the extra 2nd dimension (1) for batch_size\n",
    "inputs = torch.cat(inputs_list).view(len(inputs_list), 1, -1)\n",
    "\n",
    "# print out our inputs and their shape\n",
    "# you should see (number of sequences, batch size, input_dim)\n",
    "print('inputs size: \\n', inputs.size())\n",
    "print('\\n')\n",
    "\n",
    "print('inputs: \\n', inputs)\n",
    "print('\\n')\n",
    "\n",
    "# initialize the hidden state\n",
    "h0 = torch.randn(1, 1, hidden_dim)\n",
    "c0 = torch.randn(1, 1, hidden_dim)\n",
    "\n",
    "# wrap everything in Variable\n",
    "inputs = Variable(inputs)\n",
    "h0 = Variable(h0)\n",
    "c0 = Variable(c0)\n",
    "# get the outputs and hidden state\n",
    "out, hidden = lstm(inputs, (h0, c0))\n",
    "\n",
    "print('out: \\n', out)\n",
    "print('hidden: \\n', hidden)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Next: Hidden State and Gates\n",
    "\n",
    "This notebooks shows you the structure of the input and output of an LSTM in PyTorch. Next, you'll learn more about how exactly an LSTM represents long-term and short-term memory in it's hidden state, and you'll reach the next notebook exercise.\n",
    "\n",
    "#### Part of Speech\n",
    "\n",
    "In the notebook that comes later in this lesson, you'll see how to define a model to tag parts of speech (nouns, verbs, determinants), include an LSTM and a Linear layer to define a desired output size, *and* finally train our model to create a distribution of class scores that associates each input word with a part of speech."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}