notebooks_instructor/week_3_context.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "# Week 3: Vectors in Context\n",
    "This notebook accompanies the week 3 lecture"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# setup\n",
    "import sys\n",
    "import subprocess\n",
    "import pkg_resources\n",
    "from collections import Counter\n",
    "import re\n",
    "\n",
    "\n",
    "required = {'spacy', 'scikit-learn', 'numpy', 'pandas', 'torch'}\n",
    "installed = {pkg.key for pkg in pkg_resources.working_set}\n",
    "missing = required - installed\n",
    "\n",
    "if missing:\n",
    "    python = sys.executable\n",
    "    subprocess.check_call([python, '-m', 'pip', 'install', *missing], stdout=subprocess.DEVNULL)\n",
    "\n",
    "import spacy\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer\n",
    "from sklearn.decomposition import NMF, LatentDirichletAllocation\n",
    "from sklearn.metrics.pairwise import cosine_similarity\n",
    "from sklearn.preprocessing import normalize\n",
    "from sklearn.metrics import accuracy_score\n",
    "from sklearn.svm import LinearSVC\n",
    "import pickle\n",
    "\n",
    "from spacy.lang.en import English\n",
    "en = English()\n",
    "\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "from torch.utils.data import TensorDataset, DataLoader\n",
    "# this will set the device on which to train\n",
    "device = torch.device(\"cpu\")\n",
    "# if using collab, set your runtime to use GPU and use the line below\n",
    "#device = torch.device(\"cuda:0\")\n",
    "\n",
    "def simple_tokenizer(doc, model=en):\n",
    "    # a simple tokenizer for individual documents (different from above)\n",
    "    tokenized_docs = []\n",
    "    parsed = model(doc)\n",
    "    return([t.lower_ for t in parsed if (t.is_alpha)&(not t.like_url)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise: The limits of bag-of-words\n",
    "Look at the corpus below.  In previous implementations, we've counted up the number of \"good\" and \"bad\" instances.  What would be the problem with doing that in this case? How might you address it?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "docs = ['The movie was good',\n",
    "        'The movie was not bad, it was good',\n",
    "        'The movie was bad']\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Moving beyond unigrams\n",
    "Our work up to this point has mainly revolved around single-word tokens.  One way to include a bit more context is to move to bigrams, trigrams and maybe beyond (N-grams).  \n",
    "\n",
    "Let's see how this may help us get better measures of similarity."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using 1-grams\n",
      "[[1.         0.79056942 0.75      ]\n",
      " [0.79056942 1.         0.79056942]\n",
      " [0.75       0.79056942 1.        ]]\n",
      "Using 2-grams\n",
      "[[1.         0.7333588  0.71428571]\n",
      " [0.7333588  1.         0.64168895]\n",
      " [0.71428571 0.64168895 1.        ]]\n",
      "Using 3-grams\n",
      "[[1.         0.62554324 0.66666667]\n",
      " [0.62554324 1.         0.55603844]\n",
      " [0.66666667 0.55603844 1.        ]]\n"
     ]
    }
   ],
   "source": [
    "for i in range(1,4):\n",
    "    cv = CountVectorizer(ngram_range=(1, i))\n",
    "    counts = cv.fit_transform(docs)\n",
    "    print('Using %s-grams' % i)\n",
    "    print(cosine_similarity(counts))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can see here that with only unigrams, the second review, which has a negation of the word \"bad\", is marked as just as similar to the \"good review\" as the \"bad review\".  But once you get to the bigrams and trigrams, the second review is closer to the good review, which actually makes more sense if you read it.\n",
    "\n",
    "But let's take a look at what this does to the vocabulary size:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using 1-grams\n",
      "{'the': 5, 'movie': 3, 'was': 6, 'good': 1, 'not': 4, 'bad': 0, 'it': 2}\n",
      "7\n",
      "Using 2-grams\n",
      "{'the': 9, 'movie': 5, 'was': 11, 'good': 2, 'the movie': 10, 'movie was': 6, 'was good': 13, 'not': 7, 'bad': 0, 'it': 3, 'was not': 14, 'not bad': 8, 'bad it': 1, 'it was': 4, 'was bad': 12}\n",
      "15\n",
      "Using 3-grams\n",
      "{'the': 15, 'movie': 7, 'was': 18, 'good': 3, 'the movie': 16, 'movie was': 8, 'was good': 20, 'the movie was': 17, 'movie was good': 10, 'not': 12, 'bad': 0, 'it': 4, 'was not': 21, 'not bad': 13, 'bad it': 1, 'it was': 5, 'movie was not': 11, 'was not bad': 22, 'not bad it': 14, 'bad it was': 2, 'it was good': 6, 'was bad': 19, 'movie was bad': 9}\n",
      "23\n"
     ]
    }
   ],
   "source": [
    "for i in range(1,4):\n",
    "    cv = CountVectorizer(ngram_range=(1, i))\n",
    "    counts = cv.fit_transform(docs)\n",
    "    print('Using %s-grams' % i)\n",
    "    print(cv.vocabulary_)\n",
    "    print(len(cv.vocabulary_))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And this is just a simple corpus! Imagine if we had a realistic set of reviews, we could imagine many possible combinations of bigrams and trigrams.\n",
    "\n",
    "This is one of the reasons why it makes sense to move into sequence-based models, where we have some information being shared throughout the full parsing of the document.  This leads us to:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Recurrent Neural Networks\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "def doc_to_index(docs, vocab):\n",
    "    # transform docs into series of indices\n",
    "    docs_idxs = []\n",
    "    for d in docs:\n",
    "        w_idxs = []\n",
    "        for w in d:\n",
    "            if w in vocab:\n",
    "                w_idxs.append(vocab[w])\n",
    "            else:\n",
    "                # unknown token = 1\n",
    "                w_idxs.append(1)\n",
    "        docs_idxs.append(w_idxs)\n",
    "    return(docs_idxs)\n",
    "\n",
    "def pad_sequence(seqs, seq_len=200):\n",
    "    # function for adding padding to ensure all seq same length\n",
    "    features = np.zeros((len(seqs), seq_len),dtype=int)\n",
    "    for i, seq in enumerate(seqs):\n",
    "        if len(seq) != 0:\n",
    "            features[i, -len(seq):] = np.array(seq)[:seq_len]\n",
    "    return features\n",
    "\n",
    "def onehot_encode(data, vocab, seq_len=200):\n",
    "    # given dataset, turn each observation into a set of one-hot encoded vector\n",
    "    onehot_data = np.zeros((len(data), seq_len, len(vocab)),\n",
    "                          dtype='float32')\n",
    "    for i, d in enumerate(data):\n",
    "        for ii, w in enumerate(d):\n",
    "            onehot_data[i, ii, w] = 1\n",
    "    return(onehot_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# you will need to change this to where ever the file is stored\n",
    "data_location = '../data/assignment_1_reviews.pkl'\n",
    "with open(data_location, 'rb') as f:\n",
    "    all_text = pickle.load(f)\n",
    "neg, pos = all_text.values()\n",
    "# join all reviews\n",
    "all_reviews = np.array(neg+pos)\n",
    "# create binary indicator for positive review\n",
    "is_positive = np.array([0]*len(neg)+[1]*len(pos))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I also want to introduce the idea of a \"validation set\" here.  In training a Neural Net, often you want to leave testing the performance against the test set for last, when you have your final model.  Otherwise, you risk the model being fit to the test set, which would mean you wouldn't have an accurate read on the performance of your model on \"new\" data.  \n",
    "\n",
    "There's a lot of guidance on how to split this up, but we're just going to take 30% of our training data and make that our validation data.\n",
    "\n",
    "### Exercise: Create a training, validation and test dataset and fit a baseline model\n",
    "1) Create a training, validation, test split.  I recommend 50(train)/20(val)/30(test)\n",
    "\n",
    "2) Fit a simple SVM model on word counts, either raw or TF-IDF \n",
    "\n",
    "3) Calculate accuracy "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1233 535 731\n"
     ]
    }
   ],
   "source": [
    "# set the seed for numpy\n",
    "np.random.seed(seed=42)\n",
    "# shuffle, just for safety\n",
    "shuffled_idxs = np.random.choice(range(len(all_reviews)), size=len(all_reviews),replace=False)\n",
    "all_reviews = all_reviews[shuffled_idxs]\n",
    "is_positive = is_positive[shuffled_idxs]\n",
    "# sample random 70% for fitting model (training)\n",
    "# we'll also add a validation set, for checking the progress of the model during training\n",
    "# 30% will be simulating \"new observations\" (testing)\n",
    "pct_train = 0.7\n",
    "train_bool = np.random.random(len(all_reviews))<=pct_train\n",
    "reviews_train = all_reviews[train_bool]\n",
    "reviews_test = all_reviews[~train_bool]\n",
    "is_positive_train = is_positive[train_bool]\n",
    "is_positive_test = is_positive[~train_bool]\n",
    "# making a validation set\n",
    "pct_val = 0.3\n",
    "val_idxs = np.random.random(size=len(reviews_train))<=pct_val\n",
    "is_positive_val = is_positive_train[val_idxs]\n",
    "is_positive_val.shape\n",
    "reviews_val = reviews_train[val_idxs]\n",
    "# reconfigure train so that it doesn't include validation\n",
    "reviews_train = reviews_train[~val_idxs]\n",
    "is_positive_train = is_positive_train[~val_idxs]\n",
    "print(len(reviews_train), len(reviews_val), len(reviews_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Size of vocab: 1777\n"
     ]
    }
   ],
   "source": [
    "# transform all data to work with model\n",
    "# tokenizing ahead of time for easier match with word idx\n",
    "parsed_train = [simple_tokenizer(str(d)) for d in reviews_train]\n",
    "parsed_val = [simple_tokenizer(str(d)) for d in reviews_val]\n",
    "parsed_test = [simple_tokenizer(str(d)) for d in reviews_test]\n",
    "# this formulation works if you have previously tokenized\n",
    "cv = CountVectorizer(tokenizer=lambda doc: doc, lowercase=False, min_df=0.01)\n",
    "# **important** just fit on trained: prevents information from test in training \n",
    "cv.fit(parsed_train)\n",
    "# get out the vocab\n",
    "vocab = cv.vocabulary_\n",
    "print(\"Size of vocab:\", len(vocab))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.79890560875513\n"
     ]
    }
   ],
   "source": [
    "svc = LinearSVC(random_state=92)\n",
    "svc.fit(cv.transform(parsed_train), is_positive_train)\n",
    "base_accuracy = accuracy_score(is_positive_test,\n",
    "                               svc.predict(cv.transform(parsed_test)))\n",
    "print(base_accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A note here: We'll be using this vocabulary to transform each token to an index (numeric).  However there are two \"special\" tokens that we'll need to add:\n",
    "\n",
    "\\_PAD: The model expects all inputs to be of the same length.  So we've specified a sequence length.  If a document is longer than that, it gets truncated.  If it's shorted than that, it gets padded.  This token indicates that a particular element of the input document is padding.  This is useful information for the model\n",
    "\n",
    "\\_UNK: Depending on the vocab design, we may have certain tokens that are not included (i.e. do not have an index).  Any of these tokens are labelled as \"unknown\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "# need to adapt vocab, leave space for padding\n",
    "vocab = dict([(v, vocab[v]+2) for v in vocab])\n",
    "vocab['_UNK'] = 1\n",
    "vocab['_PAD'] = 0\n",
    "parsed_train = doc_to_index(parsed_train, vocab)\n",
    "padded_train = pad_sequence(parsed_train)\n",
    "parsed_val = doc_to_index(parsed_val, vocab)\n",
    "padded_val = pad_sequence(parsed_val)\n",
    "parsed_test = doc_to_index(parsed_test, vocab)\n",
    "padded_test = pad_sequence(parsed_test)\n",
    "# onehot encoding\n",
    "# create a \"weight matrix\" for using Embedding layer in PyTorch\n",
    "onehot_matrix = np.zeros(shape=(len(vocab), len(vocab)))\n",
    "np.fill_diagonal(onehot_matrix, 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "# construct datasets for loading by PyTorch\n",
    "train_data = TensorDataset(torch.from_numpy(padded_train), torch.from_numpy(is_positive_train))\n",
    "val_data = TensorDataset(torch.from_numpy(padded_val), torch.from_numpy(is_positive_val))\n",
    "test_data = TensorDataset(torch.from_numpy(padded_test), torch.from_numpy(is_positive_test))\n",
    "\n",
    "batch_size = 100\n",
    "\n",
    "train_loader = DataLoader(train_data, shuffle=True, batch_size=batch_size,\n",
    "                         drop_last=True) # this is to keep the size consistent\n",
    "val_loader = DataLoader(val_data, shuffle=True, batch_size=batch_size,\n",
    "                       drop_last=True)\n",
    "test_loader = DataLoader(test_data, shuffle=True, batch_size=batch_size,\n",
    "                        drop_last=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "class SentimentNet(nn.Module):\n",
    "    # sentiment classifier with single LSTM layer + Fully-connected layer, sigmoid activation and dropout\n",
    "    # adapted from https://blog.floydhub.com/long-short-term-memory-from-zero-to-hero-with-pytorch/\n",
    "    def __init__(self,\n",
    "                 weight_matrix=None,\n",
    "                 vocab_size=1000, \n",
    "                 output_size=1,  \n",
    "                 hidden_dim=512,\n",
    "                 embedding_dim=400, \n",
    "                 n_layers=2, \n",
    "                 dropout_prob=0.5):\n",
    "        super(SentimentNet, self).__init__()\n",
    "        # size of the output, in this case it's one input to one output\n",
    "        self.output_size = output_size\n",
    "        # number of layers (default 2) one LSTM layer, one fully-connected layer\n",
    "        self.n_layers = n_layers\n",
    "        # dimensions of our hidden state, what is passed from one time point to the next\n",
    "        self.hidden_dim = hidden_dim\n",
    "        # initialize the representation to pass to the LSTM\n",
    "        self.embedding, embedding_dim = self.init_embedding(\n",
    "            vocab_size, \n",
    "            embedding_dim, \n",
    "            weight_matrix)\n",
    "        # LSTM layer, where the magic happens\n",
    "        self.lstm = nn.LSTM(embedding_dim, hidden_dim, n_layers, \n",
    "                            dropout=dropout_prob, batch_first=True)\n",
    "        # dropout, similar to regularization\n",
    "        self.dropout = nn.Dropout(dropout_prob)\n",
    "        # fully connected layer\n",
    "        self.fc = nn.Linear(hidden_dim, output_size)\n",
    "        # sigmoid activiation\n",
    "        self.sigmoid = nn.Sigmoid()\n",
    "        \n",
    "    def forward(self, x, hidden):\n",
    "        # forward pass of the network\n",
    "        batch_size = x.size(0)\n",
    "        # transform input\n",
    "        embeds = self.embedding(x)\n",
    "        # run input embedding + hidden state through model\n",
    "        lstm_out, hidden = self.lstm(embeds, hidden)\n",
    "        # reshape\n",
    "        lstm_out = lstm_out.contiguous().view(-1, self.hidden_dim)\n",
    "        # dropout certain pct of connections\n",
    "        out = self.dropout(lstm_out)\n",
    "        # fully connected layer\n",
    "        out = self.fc(out)\n",
    "        # activation function\n",
    "        out = self.sigmoid(out)\n",
    "        # reshape\n",
    "        out = out.view(batch_size, -1)\n",
    "        out = out[:,-1]\n",
    "        # return the output and the hidden state\n",
    "        return out, hidden\n",
    "    \n",
    "    def init_embedding(self, vocab_size, embedding_dim, weight_matrix):\n",
    "        # initializes the embedding\n",
    "        if weight_matrix is None:\n",
    "            if vocab_size is None:\n",
    "                raise ValueError('If no weight matrix, need a vocab size')\n",
    "            # if embedding is a size, initialize trainable\n",
    "            return(nn.Embedding(vocab_size, embedding_dim),\n",
    "                   embedding_dim)\n",
    "        else:\n",
    "            # otherwise use matrix as pretrained\n",
    "            weights = torch.FloatTensor(weight_matrix)\n",
    "            return(nn.Embedding.from_pretrained(weights),\n",
    "                  weights.shape[1])\n",
    "    \n",
    "    def init_hidden(self, batch_size):\n",
    "        # initializes the hidden state\n",
    "        hidden = (torch.zeros(self.n_layers, batch_size, self.hidden_dim).to(device),\n",
    "                  torch.zeros(self.n_layers, batch_size, self.hidden_dim).to(device))\n",
    "        return hidden"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SentimentNet(\n",
       "  (embedding): Embedding(1767, 1767)\n",
       "  (lstm): LSTM(1767, 512, num_layers=2, batch_first=True, dropout=0.2)\n",
       "  (dropout): Dropout(p=0.2, inplace=False)\n",
       "  (fc): Linear(in_features=512, out_features=1, bias=True)\n",
       "  (sigmoid): Sigmoid()\n",
       ")"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_params = {'weight_matrix': onehot_matrix,\n",
    "               'output_size': 1,\n",
    "               'hidden_dim': 512,\n",
    "               'n_layers': 2,\n",
    "               'embedding_dim': 400,\n",
    "               'dropout_prob': 0.2}\n",
    "\n",
    "model = SentimentNet(**model_params)\n",
    "model.to(device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 1/1... Step: 5... Loss: 0.691114... Val Loss: 0.693912\n",
      "Validation loss decreased (inf --> 0.693912).  Saving model ...\n",
      "Epoch: 1/1... Step: 10... Loss: 0.692276... Val Loss: 0.695976\n"
     ]
    }
   ],
   "source": [
    "lr=0.005\n",
    "criterion = nn.BCELoss()\n",
    "optimizer = torch.optim.Adam(model.parameters(), lr=lr)\n",
    "# increasing this will make the training take a while on CPU\n",
    "# decrease to 5 if it's taking too long\n",
    "epochs = 1\n",
    "counter = 0\n",
    "print_every = 5\n",
    "clip = 5\n",
    "valid_loss_min = np.Inf\n",
    "\n",
    "model.train()\n",
    "for i in range(epochs):\n",
    "    h = model.init_hidden(batch_size)\n",
    "    for inputs, labels in train_loader:\n",
    "        counter += 1\n",
    "        h = tuple([e.data for e in h])\n",
    "        inputs, labels = inputs.to(device), labels.to(device)\n",
    "        model.zero_grad()\n",
    "        output, h = model(inputs, h)\n",
    "        loss = criterion(output.squeeze(), labels.float())\n",
    "        loss.backward()\n",
    "        nn.utils.clip_grad_norm_(model.parameters(), clip)\n",
    "        optimizer.step()\n",
    "        \n",
    "        if counter%print_every == 0:\n",
    "            val_h = model.init_hidden(batch_size)\n",
    "            val_losses = []\n",
    "            model.eval()\n",
    "            for inp, lab in val_loader:\n",
    "                val_h = tuple([each.data for each in val_h])\n",
    "                inp, lab = inp.to(device), lab.to(device)\n",
    "                out, val_h = model(inp, val_h)\n",
    "                val_loss = criterion(out.squeeze(), lab.float())\n",
    "                val_losses.append(val_loss.item())\n",
    "                \n",
    "            model.train()\n",
    "            print(\"Epoch: {}/{}...\".format(i+1, epochs),\n",
    "                  \"Step: {}...\".format(counter),\n",
    "                  \"Loss: {:.6f}...\".format(loss.item()),\n",
    "                  \"Val Loss: {:.6f}\".format(np.mean(val_losses)))\n",
    "            if np.mean(val_losses) <= valid_loss_min:\n",
    "                torch.save(model.state_dict(), './state_dict.pt')\n",
    "                print('Validation loss decreased ({:.6f} --> {:.6f}).  Saving model ...'.format(valid_loss_min,np.mean(val_losses)))\n",
    "                valid_loss_min = np.mean(val_losses)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now compare the LSTM performance to our SVM performance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'model' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-52-21b59eb60e75>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;31m# pytorch LSTM model\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mmodel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mload_state_dict\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mload\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'./state_dict.pt'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      3\u001b[0m \u001b[0mh\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmodel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0minit_hidden\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mbatch_size\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0mnum_correct\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      5\u001b[0m \u001b[0mmodel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0meval\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mNameError\u001b[0m: name 'model' is not defined"
     ]
    }
   ],
   "source": [
    "# pytorch LSTM model\n",
    "model.load_state_dict(torch.load('./state_dict.pt'))\n",
    "h = model.init_hidden(batch_size)\n",
    "num_correct = 0\n",
    "model.eval()\n",
    "for inputs, labels in test_loader:\n",
    "    h = tuple([each.data for each in h])\n",
    "    inputs, labels = inputs.to(device), labels.to(device)\n",
    "    output, h = model(inputs, h)\n",
    "    # takes output, rounds to 0/1\n",
    "    pred = torch.round(output.squeeze())\n",
    "    # take the correct labels, check against preds\n",
    "    correct_tensor = pred.eq(labels.float().view_as(pred))\n",
    "    correct = np.squeeze(correct_tensor.cpu().numpy())\n",
    "    # sum the number of correct\n",
    "    num_correct += np.sum(correct)\n",
    "# calc accuracy\n",
    "test_acc = num_correct/len(test_loader.dataset)\n",
    "print('LSTM accuracy:', test_acc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Yikes.  All that work and we have a model that doesn't perform as well as the simple model.\n",
    "\n",
    "There's several issues to be elaborated in the slides.  But the main one we're going to focus on is word embeddings.  Currently each element in an observation is a one-hot encoded vector for word index.  That's a pretty huge vector, and it's mostly zero.  What if we had a more dense, informative representation of an individual word?\n",
    "\n",
    "### Word-level representations\n",
    "Remember from Week 2: Our document-level representations cam also be used to create word-level representations.  For count vectors and tfidf vectors, we can just invert the matrix from document-word to word-document.  For matrix factorization, part of the estimation involves creating a word-component matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "cv = CountVectorizer(tokenizer=simple_tokenizer)\n",
    "tfidf = TfidfVectorizer(tokenizer=simple_tokenizer)\n",
    "# get vectors\n",
    "count_vecs = cv.fit_transform(all_reviews)\n",
    "tfidf_vecs = tfidf.fit_transform(all_reviews)\n",
    "n_components = 10\n",
    "nmf = NMF(n_components=n_components)\n",
    "nmf_vecs = nmf.fit_transform(tfidf_vecs)\n",
    "lda = LatentDirichletAllocation(n_components=n_components)\n",
    "lda_vecs = lda.fit_transform(count_vecs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(27053, 2499)\n",
      "(27053, 2499)\n",
      "(27053, 10)\n",
      "(27053, 10)\n"
     ]
    }
   ],
   "source": [
    "# create word-level representations\n",
    "count_words = count_vecs.T\n",
    "tfidf_words = tfidf_vecs.T\n",
    "nmf_words = nmf.components_.T\n",
    "lda_words = lda.components_.T\n",
    "for rep in [count_words, tfidf_words, nmf_words, lda_words]:\n",
    "    print(rep.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ideally, these word-level representations encode some amount of the word's meaning in them.  So let's test it with a few words.  Intuitively, we know that the words \"good\" and \"bad\" should be pretty different from eachother (semantically, at least).  We know that \"good\" and \"great\" should be pretty similar.  Let's see how our representations capture that."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[1.         0.32265411 0.32591993]\n",
      " [0.32265411 1.         0.17146286]\n",
      " [0.32591993 0.17146286 1.        ]]\n",
      "[[1.         0.2376901  0.26042242]\n",
      " [0.2376901  1.         0.10767276]\n",
      " [0.26042242 0.10767276 1.        ]]\n",
      "[[1.         0.71931288 0.77801908]\n",
      " [0.71931288 1.         0.55809816]\n",
      " [0.77801908 0.55809816 1.        ]]\n",
      "[[1.         0.99812414 0.95825282]\n",
      " [0.99812414 1.         0.94664778]\n",
      " [0.95825282 0.94664778 1.        ]]\n"
     ]
    }
   ],
   "source": [
    "seed_words = ['good', 'great', 'bad']\n",
    "# get index of seed words\n",
    "seed_idxs = [cv.vocabulary_[w] for w in seed_words]\n",
    "for rep in [count_words, tfidf_words, nmf_words, lda_words]:\n",
    "    print(cosine_similarity(rep[seed_idxs]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "None of them seem to do great with the good to bad similarity; they're often more similar than good to great.  Bad to great, however, seems more promising.\n",
    "\n",
    "But generally: These representations are based on a very small corpus and a very specific context.  What word representations (or embeddings) like Word2vec or GloVe try to do is make more general representations of the word based on its context in a large corpus of non-specific context.  Let's see how SpaCy's GloVe-based representations do on this task."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "# only the md and lg models contain GloVe vectors\n",
    "# you may need to run this on colab\n",
    "#!python -m spacy download en_core_web_md\n",
    "import en_core_web_md\n",
    "nlp = en_core_web_md.load()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[-0.42625   0.4431   -0.34517  -0.1326   -0.05816   0.052598  0.21575\n",
      " -0.36721  -0.04519   2.2444  ]\n",
      "(300,)\n"
     ]
    }
   ],
   "source": [
    "# we need to parse it with the model, then we can use the vector attribute\n",
    "glove_words = [nlp(w).vector for w in seed_words]\n",
    "# each vector is 300-dimensional dense representation\n",
    "print(glove_words[0][:10])\n",
    "print(glove_words[0].shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1.0000004, 0.8416708, 0.7355092],\n",
       "       [0.8416708, 1.       , 0.5404425],\n",
       "       [0.7355092, 0.5404425, 1.       ]], dtype=float32)"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cosine_similarity(glove_words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wow! This works really well.  We can see that good is pretty close to great, less close to bad.  Bad is far from good, but farther from great.  This is what we'd intuitively expect.\n",
    "\n",
    "This isn't to say that GloVe is always preferred.  Depending on the context, other word representations might be more useful.  But let's go with GloVe and run our RNN model with this instead of the sparse representation.\n",
    "\n",
    "### Exercise: RNN with GloVe vectors\n",
    "Instead of one-hot vectors for each word, we need the 300-dimensional word vector.  Think of a method for incorporating that into the model.  Think about the weight matrix we're using and how you might reformulate that from one-hot encoding to word vectors.\n",
    "\n",
    "Then, once you've created that weight matrix, re-run the LSTM and compare the results to the other methods."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [],
   "source": [
    "# collect vectors in matrix\n",
    "vocab = cv.vocabulary_\n",
    "vocab = dict([(v, vocab[v]+2) for v in vocab])\n",
    "vocab['_UNK'] = 1\n",
    "vocab['_PAD'] = 0\n",
    "glove_vecs = np.zeros(shape=(len(vocab), 300))\n",
    "for k, v in vocab.items():\n",
    "    glove_vecs[v] = nlp(k).vector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [],
   "source": [
    "# construct datasets for loading by PyTorch\n",
    "train_data = TensorDataset(torch.from_numpy(padded_train), torch.from_numpy(is_positive_train))\n",
    "val_data = TensorDataset(torch.from_numpy(padded_val), torch.from_numpy(is_positive_val))\n",
    "test_data = TensorDataset(torch.from_numpy(padded_test), torch.from_numpy(is_positive_test))\n",
    "\n",
    "batch_size = 100\n",
    "\n",
    "train_loader = DataLoader(train_data, shuffle=True, batch_size=batch_size,\n",
    "                         drop_last=True) # this is to keep the size consistent\n",
    "val_loader = DataLoader(val_data, shuffle=True, batch_size=batch_size,\n",
    "                       drop_last=True)\n",
    "test_loader = DataLoader(test_data, shuffle=True, batch_size=batch_size,\n",
    "                        drop_last=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SentimentNet(\n",
       "  (embedding): Embedding(27055, 300)\n",
       "  (lstm): LSTM(300, 512, num_layers=2, batch_first=True, dropout=0.5)\n",
       "  (dropout): Dropout(p=0.5, inplace=False)\n",
       "  (fc): Linear(in_features=512, out_features=1, bias=True)\n",
       "  (sigmoid): Sigmoid()\n",
       ")"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_params = {'weight_matrix': glove_vecs,\n",
    "               'output_size': 1,\n",
    "               'hidden_dim': 512,\n",
    "               'n_layers': 2,\n",
    "               'embedding_dim': 400,\n",
    "               'dropout_prob': 0.2}\n",
    "\n",
    "model = SentimentNet(**model_params)\n",
    "model.to(device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 1/2... Step: 5... Loss: 0.713173... Val Loss: 0.740239\n",
      "Validation loss decreased (inf --> 0.740239).  Saving model ...\n",
      "Epoch: 1/2... Step: 10... Loss: 0.704543... Val Loss: 0.702813\n",
      "Validation loss decreased (0.740239 --> 0.702813).  Saving model ...\n",
      "Epoch: 2/2... Step: 15... Loss: 0.697206... Val Loss: 0.699178\n",
      "Validation loss decreased (0.702813 --> 0.699178).  Saving model ...\n",
      "Epoch: 2/2... Step: 20... Loss: 0.707497... Val Loss: 0.696238\n",
      "Validation loss decreased (0.699178 --> 0.696238).  Saving model ...\n"
     ]
    }
   ],
   "source": [
    "lr=0.005\n",
    "criterion = nn.BCELoss()\n",
    "optimizer = torch.optim.Adam(model.parameters(), lr=lr)\n",
    "epochs = 2\n",
    "counter = 0\n",
    "print_every = 5\n",
    "clip = 5\n",
    "valid_loss_min = np.Inf\n",
    "\n",
    "model.train()\n",
    "for i in range(epochs):\n",
    "    h = model.init_hidden(batch_size)\n",
    "    for inputs, labels in train_loader:\n",
    "        counter += 1\n",
    "        h = tuple([e.data for e in h])\n",
    "        inputs, labels = inputs.to(device), labels.to(device)\n",
    "        model.zero_grad()\n",
    "        output, h = model(inputs, h)\n",
    "        loss = criterion(output.squeeze(), labels.float())\n",
    "        loss.backward()\n",
    "        nn.utils.clip_grad_norm_(model.parameters(), clip)\n",
    "        optimizer.step()\n",
    "        \n",
    "        if counter%print_every == 0:\n",
    "            val_h = model.init_hidden(batch_size)\n",
    "            val_losses = []\n",
    "            model.eval()\n",
    "            for inp, lab in val_loader:\n",
    "                val_h = tuple([each.data for each in val_h])\n",
    "                inp, lab = inp.to(device), lab.to(device)\n",
    "                out, val_h = model(inp, val_h)\n",
    "                val_loss = criterion(out.squeeze(), lab.float())\n",
    "                val_losses.append(val_loss.item())\n",
    "                \n",
    "            model.train()\n",
    "            print(\"Epoch: {}/{}...\".format(i+1, epochs),\n",
    "                  \"Step: {}...\".format(counter),\n",
    "                  \"Loss: {:.6f}...\".format(loss.item()),\n",
    "                  \"Val Loss: {:.6f}\".format(np.mean(val_losses)))\n",
    "            if np.mean(val_losses) <= valid_loss_min:\n",
    "                torch.save(model.state_dict(), './state_dict.pt')\n",
    "                print('Validation loss decreased ({:.6f} --> {:.6f}).  Saving model ...'.format(valid_loss_min,np.mean(val_losses)))\n",
    "                valid_loss_min = np.mean(val_losses)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LSTM accuracy: 0.4911080711354309\n"
     ]
    }
   ],
   "source": [
    "# pytorch LSTM model\n",
    "model.load_state_dict(torch.load('./state_dict.pt'))\n",
    "h = model.init_hidden(batch_size)\n",
    "num_correct = 0\n",
    "model.eval()\n",
    "for inputs, labels in test_loader:\n",
    "    h = tuple([each.data for each in h])\n",
    "    inputs, labels = inputs.to(device), labels.to(device)\n",
    "    output, h = model(inputs, h)\n",
    "    # takes output, rounds to 0/1\n",
    "    pred = torch.round(output.squeeze())\n",
    "    # take the correct labels, check against preds\n",
    "    correct_tensor = pred.eq(labels.float().view_as(pred))\n",
    "    correct = np.squeeze(correct_tensor.cpu().numpy())\n",
    "    # sum the number of correct\n",
    "    num_correct += np.sum(correct)\n",
    "# calc accuracy\n",
    "test_acc = num_correct/len(test_loader.dataset)\n",
    "print('LSTM accuracy:', test_acc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Depending on your initialization, you may or may not see an increase in accuracy here.  There are a couple of reasons:\n",
    "\n",
    "- We are using a small dataset, considering some of the state-of-the-art models are fit on many millions of observations\n",
    "- We have only trained for a short time here (small number of epochs).  Again, more state-of-the-art models train for hundreds of epochs.\n",
    "\n",
    "We'll experiment with tuning parameters in Assignment \\#3"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:mainpy3] *",
   "language": "python",
   "name": "conda-env-mainpy3-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}