lstm.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# LSTM (Long Short Term Memory)\n",
    "- These are designed to address the vanishing/exploding gradient problem in RNNs\n",
    "- LSTMs use two seperate paths to generate their predictions. They don't use the same paths that are used one step at a time.\n",
    "- LSTM is much more complicate dthen RNNs becasue of everything inside of a single unit. \n",
    "- LSTMs tend to not use ReLU, they tend to use Sigmoid and Tanh activation functions.\n",
    "- The long term memory (cell state) has no weights and biases.\n",
    "- The lack of weights allows the long-term memories to flow through a series of unrolled units without causing the gradient to explode or vanish.\n",
    "- The short term memories (hidden state), are directly connected to the weights which can modify them.\n",
    "- The first stage in a long short-term memory unit determines what percentage of the long-term memory is remembered. (forget gate)\n",
    "- The second stage of \"blocks\", one figures out the % potential memory to remember. The other figures the the potential long term memory. One gets the memory the other figures out how long should we remember it in the long term memory. (Input gate)\n",
    "- In the last stage of blocks, one calculates the % of potential memory to remember just like the previous step but the other takes the long term memory multiples by the result we got from the % of potential memory to remeber and we get the new short term memory. (Output gate)\n",
    "- You can unroll LSTMs more times than RNNs and accomidate them to longer sequences of data.\n",
    "- The amazing thing about long term memory is that steps from a lot earlier time steps can be useful a lot later.\n",
    "- As the lookback gap grows, this is where LSTMs are more useful than standard RNNs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torchvision.transforms as transforms\n",
    "import torchvision.datasets as datasets\n",
    "from torch import Tensor\n",
    "import torch.nn.functional as F\n",
    "import math"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Cell(nn.Module):\n",
    "    def __init__(self, input_size, hidden_size, bias=True):\n",
    "        super(Cell, self).__init__()\n",
    "        self.input_size = input_size\n",
    "        self.hidden_size = hidden_size\n",
    "        self.bias = bias\n",
    "        self.input = nn.Linear(input_size, 4*hidden_size, bias=bias)\n",
    "        self.hidden = nn.Linear(hidden_size, 4*hidden_size, bias=bias)\n",
    "        self.tensor = Tensor(hidden_size*3)\n",
    "        self.reset_parameters()\n",
    "\n",
    "    def reset_parameters(self):\n",
    "        std = 1.0 / math.sqrt(self.hidden_size)\n",
    "        for w in self.parameters():\n",
    "            w.data.uniform_(-std,std)\n",
    "    \n",
    "    def forward(self, x, hidden):\n",
    "        hx, cx = hidden\n",
    "\n",
    "        x = x.view(-1, x.size(1))\n",
    "        gates = self.input(x) + self.hidden(hx)\n",
    "        gates = gates.squeeze()\n",
    "\n",
    "        c2c = self.tensor.unsqueeze(0)\n",
    "        ci,cf,co = c2c.chunk(3,1)\n",
    "        ingate, forgetgate,cellgate, outgate = gates.chunk(4,1)\n",
    "\n",
    "        ingate = torch.sigmoid(ingate+ci*cx)\n",
    "        forgetgate = torch.sigmoid(forgetgate + cf *cx)\n",
    "        cellgate = forgetgate*cx + ingate* torch.tanh(cellgate)\n",
    "        outgate = torch.sigmoid(outgate+ co*cellgate)\n",
    "        \n",
    "        hm = outgate * F.tanh(cellgate)\n",
    "        return (hm, cellgate)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class LSTM(nn.Module):\n",
    "    def __init__(self, input_dim, hidden_dim, layer_dim, output_dim, bias=True):\n",
    "        super(LSTM, self).__init__()\n",
    "        self.hidden_dim = hidden_dim\n",
    "        self.layer_dim = layer_dim\n",
    "        self.lstm = Cell(input_dim, hidden_dim, layer_dim)\n",
    "        self.fc = nn.Linear(hidden_dim, output_dim)\n",
    "\n",
    "    def forward(self, x):\n",
    "        hidden = Tensor(torch.zeros(self.layer_dim, x.size(0), self.hidden_dim).to(\"mps\"))\n",
    "        cell = Tensor(torch.zeros(self.layer_dim, x.size(0), self.hidden_dim).to(\"mps\"))\n",
    "\n",
    "        outs = []\n",
    "\n",
    "        celln = cell[0,:,:]\n",
    "        hiddenn = hidden[0,:,:]\n",
    "\n",
    "        for seq in range(x.size(1)):\n",
    "            celln, hiddenn = self.lstm(x[:, seq, :], (hiddenn, celln))\n",
    "            outs.append(hiddenn)\n",
    "        \n",
    "        out = outs[-1].sequeeze()\n",
    "        out = self.fc(out)\n",
    "        return out\n"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}