News_Title_Classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# News Title Classification\n",
    "---\n",
    "\n",
    "<img src=\"assets/news.jpg\" width=70%>\n",
    "\n",
    "In this notebook, we will classify news into categories based on their headline. Our goal is to identify whether a headline is belong to one of four classes: \n",
    "- Entertainment\n",
    "- Business\n",
    "- Technology\n",
    "- Medical\n",
    "\n",
    "To achieve that, we will use a recurrent neural network (RNN). Using an RNN rather than a strictly feedforward network is more accurate since we can include information about the *sequence* of words. We will use multiple bidirectional GRU/LSTM layers in the network. The bidirectional LSTM/GRU keeps the contextual information in both directions which is pretty useful in text classification task. We also use attention model to build our model architecture. \n",
    "\n",
    "In the past conventional methods like TFIDF/CountVectorizer etc., we used to find features from text by doing a keyword extraction. Some word are more helpful in determining the category of a text than others. But in this method we sort of lost the sequential structure of text. With LSTM and deep learning methods while we are able to take case of the sequence structure we lose the ability to give higher weightage to more important words. \n",
    "\n",
    "Additionally, attention mechanism is introduced to extract such words that are important to the meaning of the sentence and aggregate the representation of those informative words to form a sentence vector.\n",
    "\n",
    "The following workflow is performed:\n",
    "1. **Data Import**: Data is the lifeblood of predictive analytics. We have to know which data to use, where to gather them, and how to make them useful to solve our problem. \n",
    "2. **Data Cleaning**: Raw data are generally incomplete, inconsistent, and contain many errors. Thus, we need to prepare the data for further processing. \n",
    "3. **Data Pre-Processing**: We structure and enrich data into another format with the intent of making it more appropriate and valuable for a variety of downstream purposes, such as analytics. \n",
    "4. **Data Modelling**: We will use a recurrent neural network (RNN).\n",
    "5. **Prediction**: Once we have developed the best predictive model, we can deploy it to make predictions. \n",
    "\n",
    "Before moving to the next section, we need to import all packages required to do the analysis by calling the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Data analysis packages\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# Deep learning packages\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.utils.data\n",
    "from torch.utils.data import TensorDataset, DataLoader\n",
    "\n",
    "# Miscellaneous\n",
    "import bcolz\n",
    "import pickle\n",
    "import re"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Table of Contents\n",
    "- [**1.0. Import the Data**](#section 1.0.)\n",
    "\n",
    "\n",
    "- [**2.0. Clean the Data**](#section 2.0.)\n",
    "  - [2.1. Lowercase All the Words](#section 2.1.)\n",
    "  - [2.2. Remove Short Forms](#section 2.2.)\n",
    "  - [2.3. Remove Punctuations and Symbols](#section 2.3.)\n",
    "\n",
    "\n",
    "- [**3.0. Data Pre-Processing**](#section 3.0.)\n",
    "  - [3.1. Tokenize the Titles](#section 3.1.)\n",
    "  - [3.2. Track Vocabulary](#section 3.2.)\n",
    "  - [3.3. Encode the Data](#section 3.3.)\n",
    "  - [3.4. Pad the Titles](#section 3.4.)\n",
    "  \n",
    "  \n",
    "- [**4.0. Define Training, Validation, and Test Set**](#section 4.0.)\n",
    "\n",
    "\n",
    "- [**5.0. Build Network Architecture**](#section 5.0.)\n",
    "  - [5.1. Import Pre-Trained Word Embeddings](#section 5.1.)\n",
    "  - [5.2. Define RNN Architecture](#section 5.2.)\n",
    "  - [5.3. Instantiate the network](#section 5.3.)\n",
    "  - [5.4. Training](#section 5.4.)\n",
    "\n",
    "\n",
    "- [**6.0. Prediction**](#section 6.0.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "<a id='section 1.0.'></a>\n",
    "## 1.0. Import the Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>No</th>\n",
       "      <th>News Title</th>\n",
       "      <th>Category</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>Google+ rolls out 'Stories' for tricked out ph...</td>\n",
       "      <td>Technology</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>Dov Charney's Redeeming Quality</td>\n",
       "      <td>Business</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>White God adds Un Certain Regard to the Palm Dog</td>\n",
       "      <td>Entertainment</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>Google shows off Androids for wearables, cars,...</td>\n",
       "      <td>Technology</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>China May new bank loans at 870.8 bln yuan</td>\n",
       "      <td>Business</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   No                                         News Title       Category\n",
       "0   1  Google+ rolls out 'Stories' for tricked out ph...     Technology\n",
       "1   2                    Dov Charney's Redeeming Quality       Business\n",
       "2   3   White God adds Un Certain Regard to the Palm Dog  Entertainment\n",
       "3   4  Google shows off Androids for wearables, cars,...     Technology\n",
       "4   5         China May new bank loans at 870.8 bln yuan       Business"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Import dataset\n",
    "df = pd.read_excel('data/News Title.xls')\n",
    "\n",
    "# Show the first 5 rows the dataset\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We only need data from `News Title` and `Category` column. So, we put them in the `titles` and `labels` variable, respectively. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "titles = df['News Title']\n",
    "labels = df['Category']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see the distribution of labels in the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Entertainment    23961\n",
       "Business         17707\n",
       "Technology       16776\n",
       "Medical           7091\n",
       "Name: Category, dtype: int64"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "labels.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To enable analytics, we transform the label into integers by calling the following. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Replace labels with integers\n",
    "dict_labels = {'Entertainment':0, 'Business':1, 'Technology': 2, 'Medical':3}\n",
    "labels = pd.Series(labels).replace(dict_labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "<a id='section 2.0.'></a>\n",
    "## 2.0. Clean the Data\n",
    "To improve the predictive performance of the model, the following data cleaning are performed:\n",
    "- Lowercase all the words in the data\n",
    "- Remove short forms\n",
    "- Remove symbols and punctuations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='section 2.1.'></a>\n",
    "### 2.1. Lowercase All the Words\n",
    "Sentences can contain a mixture of uppercase and lower case letters. Multiple sentences make up a text document. To reduce the problem space, the most common approach is to reduce everything to lower case."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Lowercase all words\n",
    "titles = titles.apply(lambda x: x.lower())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='section 2.2.'></a>\n",
    "### 2.2. Remove Short Forms\n",
    "There are some short form words in the dataset. We will fix this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Dictionary of short form words and mispellings\n",
    "short_forms_dict = {\"ain't\": \"is not\", \"aren't\": \"are not\",\"can't\": \"cannot\", \n",
    "                    \"'cause\": \"because\", \"could've\": \"could have\", \"couldn't\": \"could not\", \n",
    "                    \"didn't\": \"did not\",  \"doesn't\": \"does not\", \"don't\": \"do not\", \"hadn't\": \"had not\", \n",
    "                    \"hasn't\": \"has not\", \"haven't\": \"have not\", \"he'd\": \"he would\",\"he'll\": \"he will\", \n",
    "                    \"he's\": \"he is\", \"how'd\": \"how did\", \"how'd'y\": \"how do you\", \"how'll\": \"how will\", \n",
    "                    \"how's\": \"how is\",  \"I'd\": \"I would\", \"I'd've\": \"I would have\", \"I'll\": \"I will\", \n",
    "                    \"I'll've\": \"I will have\",\"I'm\": \"I am\", \"I've\": \"I have\", \"i'd\": \"i would\", \n",
    "                    \"i'd've\": \"i would have\", \"i'll\": \"i will\",  \"i'll've\": \"i will have\",\"i'm\": \"i am\", \n",
    "                    \"i've\": \"i have\", \"isn't\": \"is not\", \"it'd\": \"it would\", \"it'd've\": \"it would have\", \n",
    "                    \"it'll\": \"it will\", \"it'll've\": \"it will have\",\"it's\": \"it is\", \"let's\": \"let us\", \n",
    "                    \"ma'am\": \"madam\", \"mayn't\": \"may not\", \"might've\": \"might have\",\"mightn't\": \"might not\",\n",
    "                    \"mightn't've\": \"might not have\", \"must've\": \"must have\", \"mustn't\": \"must not\", \n",
    "                    \"mustn't've\": \"must not have\", \"needn't\": \"need not\", \"needn't've\": \"need not have\",\n",
    "                    \"o'clock\": \"of the clock\", \"oughtn't\": \"ought not\", \"oughtn't've\": \"ought not have\", \n",
    "                    \"shan't\": \"shall not\", \"sha'n't\": \"shall not\", \"shan't've\": \"shall not have\", \n",
    "                    \"she'd\": \"she would\", \"she'd've\": \"she would have\", \"she'll\": \"she will\", \n",
    "                    \"she'll've\": \"she will have\", \"she's\": \"she is\", \"should've\": \"should have\", \n",
    "                    \"shouldn't\": \"should not\", \"shouldn't've\": \"should not have\", \"so've\": \"so have\",\n",
    "                    \"so's\": \"so as\", \"this's\": \"this is\",\"that'd\": \"that would\", \"that'd've\": \"that would have\", \n",
    "                    \"that's\": \"that is\", \"there'd\": \"there would\", \"there'd've\": \"there would have\", \n",
    "                    \"there's\": \"there is\", \"here's\": \"here is\",\"they'd\": \"they would\", \n",
    "                    \"they'd've\": \"they would have\", \"they'll\": \"they will\", \"they'll've\": \"they will have\", \n",
    "                    \"they're\": \"they are\", \"they've\": \"they have\", \"to've\": \"to have\", \"wasn't\": \"was not\", \n",
    "                    \"we'd\": \"we would\", \"we'd've\": \"we would have\", \"we'll\": \"we will\", \n",
    "                    \"we'll've\": \"we will have\", \"we're\": \"we are\", \"we've\": \"we have\", \n",
    "                    \"weren't\": \"were not\", \"what'll\": \"what will\", \"what'll've\": \"what will have\", \n",
    "                    \"what're\": \"what are\",  \"what's\": \"what is\", \"what've\": \"what have\", \n",
    "                    \"when's\": \"when is\", \"when've\": \"when have\", \"where'd\": \"where did\", \n",
    "                    \"where's\": \"where is\", \"where've\": \"where have\", \"who'll\": \"who will\", \n",
    "                    \"who'll've\": \"who will have\", \"who's\": \"who is\", \"who've\": \"who have\", \"why's\": \"why is\", \n",
    "                    \"why've\": \"why have\", \"will've\": \"will have\", \"won't\": \"will not\", \"won't've\": \"will not have\", \n",
    "                    \"would've\": \"would have\", \"wouldn't\": \"would not\", \"wouldn't've\": \"would not have\", \n",
    "                    \"y'all\": \"you all\", \"y'all'd\": \"you all would\",\"y'all'd've\": \"you all would have\",\n",
    "                    \"y'all're\": \"you all are\",\"y'all've\": \"you all have\",\"you'd\": \"you would\", \n",
    "                    \"you'd've\": \"you would have\", \"you'll\": \"you will\", \"you'll've\": \"you will have\", \n",
    "                    \"you're\": \"you are\", \"you've\": \"you have\"}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We define `clean_shortforms` function to fix short forms."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def clean_shortforms(text):\n",
    "    clean_text = text\n",
    "    for shortform in short_forms_dict.keys():\n",
    "        if re.search(shortform, text):\n",
    "            clean_text = re.sub(shortform, short_forms_dict[shortform], text)\n",
    "    return clean_text\n",
    "\n",
    "# fix short forms\n",
    "titles = titles.apply(lambda x: clean_shortforms(x))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='section 2.3.'></a>\n",
    "### 2.3. Remove Punctuations and Symbols\n",
    "We will remove all punctuations and symbols from the dataset using `clean_symbol` function. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "symbols = [',', '.', '\"', ':', ')', '(', '-', '!', '?', '|', \n",
    "           ';', \"'\", '$', '&', '/', '[', ']', '>', '%', '=', \n",
    "           '#', '*', '+', '\\\\', '•',  '~', '@', '£', '·', '_', \n",
    "           '{', '}', '©', '^', '®', '`',  '<', '→', '°', '€', \n",
    "           '™', '›',  '♥', '←', '×', '§', '″', '′', 'Â', '█', \n",
    "           '½', 'à', '…', '“', '★', '”', '–', '●', 'â', '►', \n",
    "           '−', '¢', '²', '¬', '░', '¶', '↑', '±', '¿', '▾', \n",
    "           '═', '¦', '║', '―', '¥', '▓', '—', '‹', '─', '▒', '：', \n",
    "           '¼', '⊕', '▼', '▪', '†', '■', '’', '▀', '¨', '▄', '♫', \n",
    "           '☆', 'é', '¯', '♦', '¤', '▲', 'è', '¸', '¾', 'Ã', '⋅', \n",
    "           '‘', '∞', '∙', '）', '↓', '、', '│', '（', '»', '，', '♪', \n",
    "           '╩', '╚', '³', '・', '╦', '╣', '╔', '╗', '▬', '❤', 'ï', 'Ø', \n",
    "           '¹', '≤', '‡', '√', ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def clean_symbol(text):\n",
    "    text = str(text)\n",
    "    for symbol in symbols:\n",
    "        text = text.replace(symbol, '')\n",
    "    return text\n",
    "\n",
    "# remove symbols and punctuations \n",
    "titles = titles.apply(lambda x: clean_symbol(x))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "<a id='section 3.0.'></a>\n",
    "## 3.0. Data Pre-Processing\n",
    "We will implement the following pre-processing functions:\n",
    "- Tokenize the titles\n",
    "- Track vocabulary\n",
    "- Encode the data\n",
    "- Pad the titles"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='section 3.1.'></a>\n",
    "### 3.1. Tokenize the Titles\n",
    "We will be splitting the titles into a word array using spaces as delimiters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# tokenize all titles in the data\n",
    "titles_token = titles.apply(lambda x: x.split())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average word length of titles is 9.\n",
      "Max word length of titles is 1819.\n",
      "Min word length of titles is 1.\n"
     ]
    }
   ],
   "source": [
    "print('Average word length of titles is {0:.0f}.'.format(np.mean(titles_token.apply(lambda x: len(x)))))\n",
    "print('Max word length of titles is {0:.0f}.'.format(np.max(titles_token.apply(lambda x: len(x)))))\n",
    "print('Min word length of titles is {0:.0f}.'.format(np.min(titles_token.apply(lambda x: len(x)))))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "9       9331\n",
       "8       9075\n",
       "10      8807\n",
       "11      7757\n",
       "7       7654\n",
       "12      5782\n",
       "6       5481\n",
       "13      3466\n",
       "5       3229\n",
       "14      1753\n",
       "4       1400\n",
       "15       694\n",
       "3        558\n",
       "2        255\n",
       "16       201\n",
       "17        60\n",
       "1         22\n",
       "18         6\n",
       "71         1\n",
       "1784       1\n",
       "19         1\n",
       "1819       1\n",
       "dtype: int64"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Distribution of word length of titles\n",
    "count = [len(title) for title in titles_token]\n",
    "pd.Series(count).value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, there are a few titles that have more than 100 words. This is quite odd for a title. We consider these titles to be outliers. Thus, we will exclude all titles that have more than 100 words. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Exclude all titles that have more than 100 words\n",
    "titles_token = [title for title in titles_token if len(title) < 100]\n",
    "labels = np.array([label for title, label in zip(titles_token, labels) if len(title) < 100])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='section 3.2.'></a>\n",
    "### 3.2. Track Vocabulary\n",
    "We define `track_vocab` to track our training vocabulary, which goes through all our text and counts the occurence of the contained words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def track_vocab(sentences, verbose =  True):\n",
    "    \n",
    "    vocab = {}\n",
    "    for sentence in sentences:\n",
    "        for word in sentence:\n",
    "            try:\n",
    "                vocab[word] += 1\n",
    "            except KeyError:\n",
    "                vocab[word] = 1\n",
    "                \n",
    "    return vocab"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'google': 1683, 'rolls': 62, 'out': 1029, 'stories': 67, 'for': 8133}\n"
     ]
    }
   ],
   "source": [
    "# count the occurrence of all words in the data\n",
    "vocab_count = track_vocab(titles_token)\n",
    "print({k: vocab_count[k] for k in list(vocab_count)[:5]})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='section 3.3.'></a>\n",
    "### 3.3. Encode the Data\n",
    "Since we will use embedding layers, we'll need to encode each word with an integer. To create a word embedding, we first need to transform the words to ids.  In this function, we create two dictionaries:\n",
    "- Dictionary to go from the words to an id, we'll call `vocab_to_int`\n",
    "- Dictionary to go from the id to word, we'll call `int_to_vocab`\n",
    "\n",
    "We return these dictionaries in the **tuple** `(vocab_to_int, int_to_vocab)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def create_lookup_tables(vocab_count):\n",
    "    \n",
    "    # sorting the words from most to least frequent in text occurrence\n",
    "    sorted_vocab = sorted(vocab_count, key=vocab_count.get, reverse=True)\n",
    "    # create vocab_to_int dictionary\n",
    "    int_to_vocab = {ii: word for ii, word in enumerate(sorted_vocab)}\n",
    "    vocab_to_int = {word: ii for ii, word in int_to_vocab.items()}\n",
    "    \n",
    "    # return tuple\n",
    "    return (vocab_to_int, int_to_vocab)\n",
    "\n",
    "vocab_to_int, int_to_vocab = create_lookup_tables(vocab_count)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, we can convert our data into integers, so they can be passed into the network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# encode the data\n",
    "title_ints = []\n",
    "for title in titles_token:\n",
    "    title_ints.append([vocab_to_int[word] for word in title])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='section 3.4.'></a>\n",
    "### 3.4. Pad the Titles\n",
    "To deal with both short and very long title, we'll pad or truncate all our titles to a specific length. For titles shorter than some `seq_length`, we'll pad with 0s. For titles longer than `seq_length`, we can truncate them to the first `seq_length` words. A good `seq_length`, in this case, is 31, because the maximum title length from the data is 31.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def pad_features(sentences_token, seq_length):\n",
    "    # getting the correct rows x cols shape\n",
    "    features = np.zeros((len(sentences_token), seq_length), dtype=int)\n",
    "\n",
    "    # for each title, I grab that title and \n",
    "    for i, row in enumerate(sentences_token):\n",
    "        features[i, -len(row):] = np.array(row)[:seq_length]\n",
    "    \n",
    "    return features\n",
    "\n",
    "# pad the titles\n",
    "seq_length = 31\n",
    "features = pad_features(title_ints, seq_length)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "<a id='section 4.0.'></a>\n",
    "## 4.0. Define Training, Validation, and Test Set\n",
    "With our data in nice shape, we'll split it into training, validation, and test sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "train_X, val_test_X, train_y, val_test_y = train_test_split(features, labels, \n",
    "                                                            test_size=0.2, \n",
    "                                                            random_state=42, shuffle=True,\n",
    "                                                            stratify=labels)\n",
    "\n",
    "val_X, test_X, val_y, test_y = train_test_split(val_test_X, val_test_y, \n",
    "                                                test_size=0.5, \n",
    "                                                random_state=42, shuffle=True,\n",
    "                                                stratify=val_test_y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### DataLoaders and Batching\n",
    "After creating training, test, and validation data, we can create DataLoaders for this data by following two steps:\n",
    "1. Create a known format for accessing our data, using [TensorDataset](https://pytorch.org/docs/stable/data.html#) which takes in an input set of data and a target set of data with the same first dimension, and creates a dataset.\n",
    "2. Create DataLoaders and batch our training, validation, and test Tensor datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# create Tensor datasets\n",
    "train_data = TensorDataset(torch.from_numpy(train_X), torch.from_numpy(train_y))\n",
    "valid_data = TensorDataset(torch.from_numpy(val_X), torch.from_numpy(val_y))\n",
    "test_data = TensorDataset(torch.from_numpy(test_X), torch.from_numpy(test_y))\n",
    "\n",
    "# dataloaders\n",
    "batch_size = 50\n",
    "num_workers = 8\n",
    "\n",
    "# make sure to SHUFFLE the training data\n",
    "train_loader = DataLoader(train_data, shuffle=True, batch_size=batch_size, num_workers=num_workers)\n",
    "valid_loader = DataLoader(valid_data, shuffle=True, batch_size=batch_size, num_workers=num_workers)\n",
    "test_loader = DataLoader(test_data, shuffle=True, batch_size=batch_size, num_workers=num_workers)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sample input size:  torch.Size([50, 31])\n",
      "Sample input: \n",
      " tensor([[    0,     0,     0,  ..., 25034,   900,  3322],\n",
      "        [    0,     0,     0,  ...,     5,   425,  3243],\n",
      "        [    0,     0,     0,  ...,  4634,     0,   136],\n",
      "        ...,\n",
      "        [    0,     0,     0,  ...,   143,   278,  4303],\n",
      "        [    0,     0,     0,  ...,    78,   413,  1204],\n",
      "        [    0,     0,     0,  ...,  1182,     6,  6458]])\n",
      "\n",
      "Sample label size:  torch.Size([50])\n",
      "Sample label: \n",
      " tensor([0, 1, 1, 3, 3, 1, 2, 0, 0, 0, 1, 0, 0, 2, 3, 1, 0, 0, 1, 2, 1, 1, 3, 3,\n",
      "        0, 3, 0, 1, 1, 1, 0, 0, 2, 0, 1, 0, 2, 3, 2, 2, 1, 3, 0, 3, 2, 1, 1, 1,\n",
      "        0, 3])\n"
     ]
    }
   ],
   "source": [
    "# obtain one batch of training data\n",
    "dataiter = iter(train_loader)\n",
    "sample_x, sample_y = dataiter.next()\n",
    "\n",
    "print('Sample input size: ', sample_x.size()) # batch_size, seq_length\n",
    "print('Sample input: \\n', sample_x)\n",
    "print()\n",
    "print('Sample label size: ', sample_y.size()) # batch_size\n",
    "print('Sample label: \\n', sample_y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='section 5.0.'></a>\n",
    "## 5.0. Build Network Architecture\n",
    "<a id='section 5.1.'></a>\n",
    "### 5.1. Import Pre-Trained Word Embeddings\n",
    "**Note**: This part is taken from a Medium article [How to use Pre-trained Word Embeddings in PyTorch](https://medium.com/@martinpella/how-to-use-pre-trained-word-embeddings-in-pytorch-71ca59249f76). \n",
    "\n",
    "Rather than training our own word vectors from scratch for word embedding, we will leverage on GloVe. Its authors have released four text files with word vectors trained on different massive web datasets. They are available for download [here](https://nlp.stanford.edu/projects/glove/). We will use “Wikipedia 2014 + Gigaword 5” which is the smallest file (“ [glove.6B.zip](http://nlp.stanford.edu/data/wordvecs/glove.6B.zip)”) with 822 MB. It was trained on a corpus of 6 billion tokens and contains a vocabulary of 400 thousand tokens.\n",
    "\n",
    "We need to parse the file to get as output: list of words, dictionary mapping each word to their id (position) and array of vectors. Given that the vocabulary have 400k tokens, we will use [bcolz](https://github.com/Blosc/bcolz) to store the array of vectors. It provides columnar, chunked data containers that can be compressed either in-memory and on-disk. It is based on NumPy, and uses it as the standard data container to communicate with bcolz objects. We then save the outputs to disk for future uses."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "words = []\n",
    "idx = 0\n",
    "word2idx = {}\n",
    "vectors = bcolz.carray(np.zeros(1), rootdir=f'embedding/glove/6B.300.dat', mode='w')\n",
    "\n",
    "with open(f'embedding/glove/glove.6B.300d.txt', 'rb') as f:\n",
    "    for l in f:\n",
    "        line = l.decode().split()\n",
    "        word = line[0]\n",
    "        words.append(word)\n",
    "        word2idx[word] = idx\n",
    "        idx += 1\n",
    "        vect = np.array(line[1:]).astype(np.float)\n",
    "        vectors.append(vect)\n",
    "    \n",
    "vectors = bcolz.carray(vectors[1:].reshape((400000, 300)), rootdir=f'embedding/glove/6B.300.dat', mode='w')\n",
    "vectors.flush()\n",
    "pickle.dump(words, open(f'embedding/glove/6B.300_words.pkl', 'wb'))\n",
    "pickle.dump(word2idx, open(f'embedding/glove/6B.300_idx.pkl', 'wb'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using those objects we can now create a dictionary that given a word returns its vector."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "vectors = bcolz.open(f'embedding/glove/6B.300.dat')[:]\n",
    "words = pickle.load(open(f'embedding/glove/6B.300_words.pkl', 'rb'))\n",
    "word2idx = pickle.load(open(f'embedding/glove/6B.300_idx.pkl', 'rb'))\n",
    "\n",
    "glove = {w: vectors[word2idx[w]] for w in words}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What we need to do at this point is to create an embedding layer, that is a dictionary mapping integer indices (that represent words) to dense vectors. It takes as input integers, it looks up these integers into an internal dictionary, and it returns the associated vectors.\n",
    "\n",
    "We have already built a Python dictionary with similar characteristics, but it does not support auto differentiation so can not be used as a neural network layer and was also built based on GloVe’s vocabulary, likely different from our dataset’s vocabulary. In PyTorch an embedding layer is available through torch.nn.Embedding class.\n",
    "\n",
    "We must build a matrix of weights that will be loaded into the PyTorch embedding layer. Its shape will be equal to (dataset’s vocabulary length, word vectors dimension).\n",
    "\n",
    "For each word in dataset’s vocabulary, we check if it is on GloVe’s vocabulary. If it do it, we load its pre-trained word vector. Otherwise, we initialize a random vector."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sorted_vocab = sorted(vocab_count, key=vocab_count.get, reverse=True)\n",
    "target_vocab = sorted_vocab\n",
    "emb_dim = 300\n",
    "\n",
    "matrix_len = len(target_vocab)\n",
    "weights_matrix = np.zeros((matrix_len, emb_dim))\n",
    "words_found = 0\n",
    "\n",
    "for i, word in enumerate(target_vocab):\n",
    "    try: \n",
    "        weights_matrix[i] = glove[word]\n",
    "        words_found += 1\n",
    "    except KeyError:\n",
    "        weights_matrix[i] = np.random.normal(scale=0.6, size=(emb_dim, ))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def create_emb_layer(weights_matrix, non_trainable=False):\n",
    "    num_embeddings, embedding_dim = weights_matrix.shape\n",
    "    emb_layer = nn.Embedding(num_embeddings, embedding_dim)\n",
    "    emb_layer.load_state_dict({'weight': torch.from_numpy(weights_matrix)})\n",
    "    if non_trainable:\n",
    "        emb_layer.weight.requires_grad = False\n",
    "\n",
    "    return emb_layer, num_embeddings, embedding_dim"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='section 5.2.'></a>\n",
    "### 5.2. Define RNN Architecture\n",
    "\n",
    "In this model, we use multiple bidirectional GRU/LSTM layers in the network. The bidirectional LSTM/GRU keeps the contextual information in both directions which is pretty useful in text classification task. \n",
    "\n",
    "We also use attention model to build our model architecture. In the past conventional methods like TFIDF/CountVectorizer etc., we used to find features from text by doing a keyword extraction. Some word are more helpful in determining the category of a text than others. But in this method we sort of lost the sequential structure of text. With LSTM and deep learning methods while we are able to take case of the sequence structure we lose the ability to give higher weightage to more important words. Attention mechanism is introduced to extract such words that are important to the meaning of the sentence and aggregate the representation of those informative words to form a sentence vector."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "No GPU available, training on CPU.\n"
     ]
    }
   ],
   "source": [
    "# First checking if GPU is available\n",
    "train_on_gpu=torch.cuda.is_available()\n",
    "\n",
    "if(train_on_gpu):\n",
    "    print('Training on GPU.')\n",
    "else:\n",
    "    print('No GPU available, training on CPU.')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# implementation of attention layer\n",
    "class Attention(nn.Module):\n",
    "    def __init__(self, feature_dim, step_dim, bias=True, **kwargs):\n",
    "        super(Attention, self).__init__(**kwargs)\n",
    "        \n",
    "        self.supports_masking = True\n",
    "\n",
    "        self.bias = bias\n",
    "        self.feature_dim = feature_dim\n",
    "        self.step_dim = step_dim\n",
    "        self.features_dim = 0\n",
    "        \n",
    "        weight = torch.zeros(feature_dim, 1)\n",
    "        nn.init.kaiming_uniform_(weight)\n",
    "        self.weight = nn.Parameter(weight)\n",
    "        \n",
    "        if bias:\n",
    "            self.b = nn.Parameter(torch.zeros(step_dim))\n",
    "        \n",
    "    def forward(self, x, mask=None):\n",
    "        feature_dim = self.feature_dim \n",
    "        step_dim = self.step_dim\n",
    "\n",
    "        eij = torch.mm(\n",
    "            x.contiguous().view(-1, feature_dim), \n",
    "            self.weight\n",
    "        ).view(-1, step_dim)\n",
    "        \n",
    "        if self.bias:\n",
    "            eij = eij + self.b\n",
    "            \n",
    "        eij = torch.tanh(eij)\n",
    "        a = torch.exp(eij)\n",
    "        \n",
    "        if mask is not None:\n",
    "            a = a * mask\n",
    "\n",
    "        a = a / (torch.sum(a, 1, keepdim=True) + 1e-10)\n",
    "\n",
    "        weighted_input = x * torch.unsqueeze(a, -1)\n",
    "        return torch.sum(weighted_input, 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">**First, we'll pass in words to an embedding layer.** We need an embedding layer because we have tens to hundreds of thousands of words, so we will need a more efficient representation for our input data than one-hot encoded vectors. \n",
    "\n",
    ">**After input words are passed to an embedding layer, the new embeddings will be passed to bidirectional LSTM/GRU layers.** These layers will add *recurrent* connections to the network and give us the ability to include information about the *sequence* of words in our data. The bidirectional LSTM/GRU layers keep the contextual information in both directions which is pretty useful in text classification task. \n",
    "\n",
    ">**The outputs of the bidirectional LSTM/GRU layers will be passed to the attention layer.** Attention mechanism is introduced to extract such words that are important to the meaning of the sentence and aggregate the representation of those informative words to form a sentence vector.\n",
    "\n",
    ">**The outputs of the attention layer will be passed to the second bidirectional LSTM/GRU layers.** \n",
    "\n",
    ">**Finally, the outputs will go to a output layer.** We are using a fully-connected neural network layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import torch.nn as nn\n",
    "\n",
    "class RNN(nn.Module):\n",
    "    \"\"\"\n",
    "    The RNN model that will be used to perform classification.\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(self, weights_matrix, output_size, hidden_dim, drop_prob=0.1):\n",
    "        \"\"\"\n",
    "        Initialize the model by setting up the layers.\n",
    "        \"\"\"\n",
    "        super(RNN, self).__init__()\n",
    "\n",
    "        self.output_size = output_size\n",
    "        self.hidden_dim = hidden_dim\n",
    "        \n",
    "        # embedding layers\n",
    "        self.embedding, self.num_embeddings, self.embedding_dim = create_emb_layer(weights_matrix, True)\n",
    "        \n",
    "        # embedding dropout\n",
    "        self.dropout = nn.Dropout2d(drop_prob)\n",
    "        \n",
    "        # First lstm and GRU layers\n",
    "        self.lstm1 = nn.LSTM(self.embedding_dim, hidden_dim, batch_first=True, bidirectional=True)\n",
    "        self.gru1 = nn.GRU(hidden_dim * 2, hidden_dim, bidirectional=True, batch_first=True)\n",
    "        \n",
    "        # attention layer\n",
    "        self.attention = Attention(hidden_dim*2, seq_length)\n",
    "        \n",
    "        # Second lstm and GRU layers\n",
    "        self.lstm2 = nn.LSTM(hidden_dim * 2, hidden_dim, batch_first=True, bidirectional=True)\n",
    "        self.gru2 = nn.GRU(hidden_dim * 2, hidden_dim, bidirectional=True, batch_first=True)\n",
    "        \n",
    "        # linear\n",
    "        self.fc = nn.Linear(hidden_dim * 2, 64)\n",
    "        self.out = nn.Linear(64, self.output_size)\n",
    "        \n",
    "        self.relu = nn.ReLU()\n",
    "        \n",
    "\n",
    "    def forward(self, x):\n",
    "        \"\"\"\n",
    "        Perform a forward pass of our model on some inputs.\n",
    "        \"\"\"\n",
    "        batch_size = x.size(0)\n",
    "\n",
    "        # embedding output\n",
    "        x = x.long()\n",
    "        embeds = self.embedding(x)\n",
    "        embeds = torch.squeeze(torch.unsqueeze(embeds, 0))\n",
    "        \n",
    "        # lstm, gru, and attention outputs\n",
    "        lstm_out, _ = self.lstm1(embeds)\n",
    "        gru_out, _ = self.gru1(lstm_out)\n",
    "        attention_out = self.attention(gru_out, 256)\n",
    "        attention_out = attention_out.view(batch_size, -1, self.hidden_dim * 2)\n",
    "        lstm_out, _ = self.lstm2(attention_out)\n",
    "        gru_out, _ = self.gru2(lstm_out)\n",
    "        \n",
    "        # linear outputs\n",
    "        out = gru_out.view(-1, gru_out.shape[2])\n",
    "        fc_out = self.relu(self.fc(out))\n",
    "        final_out = self.out(fc_out)\n",
    "    \n",
    "        return final_out"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='section 5.3.'></a>\n",
    "### 5.3. Instantiate the network\n",
    "Here, we'll instantiate the network. First up, defining the hyperparameters.\n",
    "\n",
    "* `weights_matrix`: The pre-trained word vector.\n",
    "* `output_size`: Size of our desired output.\n",
    "* `hidden_dim`: Number of units in the hidden layers of our LSTM/GRU cells. Usually larger is better performance wise. Common values are 128, 256, 512, etc.\n",
    "* `n_layers`: Number of LSTM/GRU layers in the network. Typically between 1-3. We use the value of 1 as our `n_layers`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RNN(\n",
      "  (embedding): Embedding(32779, 300)\n",
      "  (dropout): Dropout2d(p=0.1)\n",
      "  (lstm1): LSTM(300, 256, batch_first=True, bidirectional=True)\n",
      "  (gru1): GRU(512, 256, batch_first=True, bidirectional=True)\n",
      "  (attention): Attention()\n",
      "  (lstm2): LSTM(512, 256, batch_first=True, bidirectional=True)\n",
      "  (gru2): GRU(512, 256, batch_first=True, bidirectional=True)\n",
      "  (fc): Linear(in_features=512, out_features=64, bias=True)\n",
      "  (out): Linear(in_features=64, out_features=4, bias=True)\n",
      "  (relu): ReLU()\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "# Instantiate the model w/ hyperparams\n",
    "weights_matrix = weights_matrix\n",
    "output_size = 4\n",
    "hidden_dim = 256\n",
    "\n",
    "net = RNN(weights_matrix, output_size, hidden_dim)\n",
    "\n",
    "print(net)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='section 5.4.'></a>\n",
    "### 5.4. Training\n",
    "We use `CrossEntropyLoss` as our loss function and `SGD` as our optimizer. We also have some data and training hyparameters:\n",
    "\n",
    "* `lr`: Learning rate for our optimizer.\n",
    "* `epochs`: Number of times to iterate through the training dataset.\n",
    "* `clip`: The maximum gradient value to clip at (to prevent exploding gradients)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# loss and optimization functions\n",
    "lr = 0.0001\n",
    "criterion = nn.CrossEntropyLoss()\n",
    "optimizer = torch.optim.Adam(filter(lambda p: p.requires_grad, net.parameters()), lr=lr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 1/1... Step: 100... Loss: 0.510917... Val Loss: 0.723591\n",
      "Epoch: 1/1... Step: 200... Loss: 0.633727... Val Loss: 0.718747\n",
      "Epoch: 1/1... Step: 300... Loss: 0.679442... Val Loss: 0.721072\n",
      "Epoch: 1/1... Step: 400... Loss: 0.530738... Val Loss: 0.708940\n",
      "Epoch: 1/1... Step: 500... Loss: 0.563486... Val Loss: 0.709165\n",
      "Epoch: 1/1... Step: 600... Loss: 0.548085... Val Loss: 0.706518\n",
      "Epoch: 1/1... Step: 700... Loss: 0.503572... Val Loss: 0.709962\n",
      "Epoch: 1/1... Step: 800... Loss: 0.409615... Val Loss: 0.711098\n",
      "Epoch: 1/1... Step: 900... Loss: 0.793967... Val Loss: 0.709682\n",
      "Epoch: 1/1... Step: 1000... Loss: 0.969674... Val Loss: 0.702137\n"
     ]
    }
   ],
   "source": [
    "# training params\n",
    "\n",
    "epochs = 1\n",
    "\n",
    "counter = 0\n",
    "print_every = 100\n",
    "clip = 5 # gradient clipping\n",
    "\n",
    "# move model to GPU, if available\n",
    "if(train_on_gpu):\n",
    "    net.cuda()\n",
    "\n",
    "net.train()\n",
    "# train for some number of epochs\n",
    "for e in range(epochs):\n",
    "\n",
    "    # batch loop\n",
    "    for inputs, labels in train_loader:\n",
    "        counter += 1\n",
    "\n",
    "        if(train_on_gpu):\n",
    "            inputs, labels = inputs.cuda(), labels.cuda()\n",
    "\n",
    "        # zero accumulated gradients\n",
    "        net.zero_grad()\n",
    "\n",
    "        # get the output from the model\n",
    "        output = net(inputs)\n",
    "\n",
    "        # calculate the loss and perform backprop\n",
    "        loss = criterion(output, labels)\n",
    "        loss.backward()\n",
    "        \n",
    "        # `clip_grad_norm` helps prevent the exploding gradient problem in RNNs / LSTMs.\n",
    "        nn.utils.clip_grad_norm_(net.parameters(), clip)\n",
    "        optimizer.step()\n",
    "\n",
    "        # loss stats\n",
    "        if counter % print_every == 0:\n",
    "            # Get validation loss\n",
    "            val_losses = []\n",
    "            net.eval()\n",
    "            for inputs, labels in valid_loader:\n",
    "\n",
    "                if(train_on_gpu):\n",
    "                    inputs, labels = inputs.cuda(), labels.cuda()\n",
    "\n",
    "                output = net(inputs)\n",
    "                val_loss = criterion(output, labels)\n",
    "\n",
    "                val_losses.append(val_loss.item())\n",
    "\n",
    "            net.train()\n",
    "            print(\"Epoch: {}/{}...\".format(e+1, epochs),\n",
    "                  \"Step: {}...\".format(counter),\n",
    "                  \"Loss: {:.6f}...\".format(loss.item()),\n",
    "                  \"Val Loss: {:.6f}\".format(np.mean(val_losses)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='section 6.0.'></a>\n",
    "## 6.0. Predictions\n",
    "We'll see how our trained model performs on all of our defined test_data, above. We'll calculate the average loss and accuracy over the test data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test loss: 0.684\n",
      "Test accuracy: 0.788\n"
     ]
    }
   ],
   "source": [
    "# Get test data loss and accuracy\n",
    "\n",
    "test_losses = [] # track loss\n",
    "num_correct = 0\n",
    "\n",
    "net.eval()\n",
    "# iterate over test data\n",
    "for inputs, labels in test_loader:\n",
    "\n",
    "    if(train_on_gpu):\n",
    "        inputs, labels = inputs.cuda(), labels.cuda()\n",
    "    \n",
    "    # get predicted outputs\n",
    "    output = net(inputs)\n",
    "    \n",
    "    # calculate loss\n",
    "    test_loss = criterion(output, labels)\n",
    "    test_losses.append(test_loss.item())\n",
    "    \n",
    "    # convert output probabilities to predicted class (0, 1, 2, or 3)\n",
    "    pred = output.data.max(1, keepdim=True)[1]  \n",
    "    \n",
    "    # compare predictions to true label\n",
    "    correct_tensor = pred.eq(labels.view_as(pred))\n",
    "    correct = np.squeeze(correct_tensor.numpy()) if not train_on_gpu else np.squeeze(correct_tensor.cpu().numpy())\n",
    "    num_correct += np.sum(correct)\n",
    "\n",
    "\n",
    "# avg test loss\n",
    "print(\"Test loss: {:.3f}\".format(np.mean(test_losses)))\n",
    "\n",
    "# accuracy over all test data\n",
    "test_acc = num_correct/len(test_loader.dataset)\n",
    "print(\"Test accuracy: {:.3f}\".format(test_acc))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our mode is able to achieve **78.8% accuracy**. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Save the Model\n",
    "After running the above training cell, the trained model will be saved by name, `trained_rnn`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "def save_model(filename, decoder):\n",
    "    save_filename = os.path.splitext(os.path.basename(filename))[0] + '.pt'\n",
    "    torch.save(decoder, save_filename)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model Trained and Saved\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/anaconda3/lib/python3.6/site-packages/torch/serialization.py:256: UserWarning: Couldn't retrieve source code for container of type RNN. It won't be checked for correctness upon loading.\n",
      "  \"type \" + obj.__name__ + \". It won't be checked \"\n",
      "/anaconda3/lib/python3.6/site-packages/torch/serialization.py:256: UserWarning: Couldn't retrieve source code for container of type Attention. It won't be checked for correctness upon loading.\n",
      "  \"type \" + obj.__name__ + \". It won't be checked \"\n"
     ]
    }
   ],
   "source": [
    "# saving the trained model\n",
    "save_model('./save/trained_rnn', net)\n",
    "print('Model Trained and Saved')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}