lstm_sentiment.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Parts of Speech Sentiment Analysis Mini-Study\n",
    "To further my interest in natural language processing using non-text/stylometric data, I created this mini-study with the purpose of showing that sentiment can be observed using only parts-of-speech tags for input. While there is a sizable amount of research dealing with POS and other stylometric data in authorship-attribution tasks, there is very little - if any - research into using solely POS/non-text stylometric data for other common NLP tasks, such as gauging user sentiment. I can't say why this is; maybe this whole thing is just so well-known that nobody is talking about it, or maybe it's the fact that POS and its related data types tend to be slightly less performant than traditional text input which fosters a lack of interest. Regardless, I have a hunch that POS tags generalized nature could make them useful as a strong base for transfer learning. This is an initial proof of concept to what will hopefully be part of a much larger experiment.\n",
    "\n",
    "I've chosen to use sentiment140 as is somewhat of a NLP \"Hello World!\" dataset, similar to how MINST is to computer vision. It consists of 1.6 million tweets which have been labeled as either positive or negative. I've built of off Arun Pandian's \"NLP Beginner - Text Classification using LSTM\" on Kaggle because their kernel does most of what I want to study already and because of how thorough their comments are. I've modified the preprocessing to convert all text to POS tags using nltk. I swapped out the author’s Glove vectorizations for Word to Vec vectorizations. This is  because I won't be using transfer learning and because I've had better luck with W2V than Glove when starting from scratch in the past. The model itself has been modified significantly in order to better fit the new POS data. The key takeaways from the changes to the model are dropout, dropout, and a little more dropout, which the generalized POS embeddings appear to thrive under.\n",
    "\n",
    "In all we scored 64% in overall accuracy, which isn't great considering random guessing would yield us 50%. The good news is that we've accomplished our goal of showing that POS tags alone do in fact convey sentiment. A slight drop in performance with POS is to be expected, and considering that the original model only got 78% using full text input with transfer learning, our results look a little bit better. I plan to graduate to a dataset with larger entries as twitter’s 140 character limit is likely contributing to the low scores across the board for this dataset on Kaggle. \n",
    "\n",
    "All comments below this line belong to the original author"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Natural Language Processing\n",
    "**Natural Language Processing or NLP** is a branch of Artificial Intelligence which deal with bridging the machines understanding humans in their Natural Language. Natural Language can be in form of text or sound, which are used for humans to communicate each other. NLP can enable humans to communicate to machines in a natural way. \n",
    "\n",
    "\n",
    "**Text Classification** is a process involved in Sentiment Analysis. It is classification of peoples opinion or expressions into different sentiments. Sentiments include *Positive, Neutral*, and *Negative*, *Review Ratings* and *Happy, Sad*. Sentiment Analysis can be done on different consumer centered industries to analyse people's opinion on a particular product or subject. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  Importing Dependencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "_cell_guid": "79c7e3d0-c299-4dcb-8224-4455121ee9b0",
    "_uuid": "d629ff2d2480ee46fbb7e2d37f6b5fab8052498a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tensorflow Version 2.8.0\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "import nltk\n",
    "\n",
    "from keras import Sequential\n",
    "from keras.preprocessing.text import Tokenizer\n",
    "from keras.preprocessing.sequence import pad_sequences\n",
    "from tensorflow.keras.layers import Conv1D, Bidirectional, LSTM, Dense, Input, Dropout, SpatialDropout1D, GlobalMaxPool1D\n",
    "from tensorflow.keras.layers import SpatialDropout1D\n",
    "from tensorflow.keras.callbacks import ModelCheckpoint\n",
    "from tensorflow.keras.optimizers import Adam\n",
    "from tensorflow.keras.callbacks import ReduceLROnPlateau, EarlyStopping\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import LabelEncoder\n",
    "from sklearn.metrics import confusion_matrix, classification_report, accuracy_score\n",
    "\n",
    "from gensim.models import Word2Vec\n",
    "\n",
    "import itertools\n",
    "import random\n",
    "import re\n",
    "import os\n",
    "\n",
    "print(\"Tensorflow Version\",tf.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#  Dataset Preprocessing\n",
    "In this notebook, I am using **Sentiment-140** from [Kaggle](https://www.kaggle.com/kazanova/sentiment140). It contains a labels data of 1.6 Million Tweets and I find it a good amount of data to train our model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>1467810369</td>\n",
       "      <td>Mon Apr 06 22:19:45 PDT 2009</td>\n",
       "      <td>NO_QUERY</td>\n",
       "      <td>_TheSpecialOne_</td>\n",
       "      <td>@switchfoot http://twitpic.com/2y1zl - Awww, t...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>1467810672</td>\n",
       "      <td>Mon Apr 06 22:19:49 PDT 2009</td>\n",
       "      <td>NO_QUERY</td>\n",
       "      <td>scotthamilton</td>\n",
       "      <td>is upset that he can't update his Facebook by ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>1467810917</td>\n",
       "      <td>Mon Apr 06 22:19:53 PDT 2009</td>\n",
       "      <td>NO_QUERY</td>\n",
       "      <td>mattycus</td>\n",
       "      <td>@Kenichan I dived many times for the ball. Man...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>1467811184</td>\n",
       "      <td>Mon Apr 06 22:19:57 PDT 2009</td>\n",
       "      <td>NO_QUERY</td>\n",
       "      <td>ElleCTF</td>\n",
       "      <td>my whole body feels itchy and like its on fire</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>1467811193</td>\n",
       "      <td>Mon Apr 06 22:19:57 PDT 2009</td>\n",
       "      <td>NO_QUERY</td>\n",
       "      <td>Karoli</td>\n",
       "      <td>@nationwideclass no, it's not behaving at all....</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   0           1                             2         3                4  \\\n",
       "0  0  1467810369  Mon Apr 06 22:19:45 PDT 2009  NO_QUERY  _TheSpecialOne_   \n",
       "1  0  1467810672  Mon Apr 06 22:19:49 PDT 2009  NO_QUERY    scotthamilton   \n",
       "2  0  1467810917  Mon Apr 06 22:19:53 PDT 2009  NO_QUERY         mattycus   \n",
       "3  0  1467811184  Mon Apr 06 22:19:57 PDT 2009  NO_QUERY          ElleCTF   \n",
       "4  0  1467811193  Mon Apr 06 22:19:57 PDT 2009  NO_QUERY           Karoli   \n",
       "\n",
       "                                                   5  \n",
       "0  @switchfoot http://twitpic.com/2y1zl - Awww, t...  \n",
       "1  is upset that he can't update his Facebook by ...  \n",
       "2  @Kenichan I dived many times for the ball. Man...  \n",
       "3    my whole body feels itchy and like its on fire   \n",
       "4  @nationwideclass no, it's not behaving at all....  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv('tweets.csv', encoding = 'latin',header=None)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can see the columns are without any proper names. Lets rename them for our reference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sentiment</th>\n",
       "      <th>id</th>\n",
       "      <th>date</th>\n",
       "      <th>query</th>\n",
       "      <th>user_id</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>1467810369</td>\n",
       "      <td>Mon Apr 06 22:19:45 PDT 2009</td>\n",
       "      <td>NO_QUERY</td>\n",
       "      <td>_TheSpecialOne_</td>\n",
       "      <td>@switchfoot http://twitpic.com/2y1zl - Awww, t...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>1467810672</td>\n",
       "      <td>Mon Apr 06 22:19:49 PDT 2009</td>\n",
       "      <td>NO_QUERY</td>\n",
       "      <td>scotthamilton</td>\n",
       "      <td>is upset that he can't update his Facebook by ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>1467810917</td>\n",
       "      <td>Mon Apr 06 22:19:53 PDT 2009</td>\n",
       "      <td>NO_QUERY</td>\n",
       "      <td>mattycus</td>\n",
       "      <td>@Kenichan I dived many times for the ball. Man...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>1467811184</td>\n",
       "      <td>Mon Apr 06 22:19:57 PDT 2009</td>\n",
       "      <td>NO_QUERY</td>\n",
       "      <td>ElleCTF</td>\n",
       "      <td>my whole body feels itchy and like its on fire</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>1467811193</td>\n",
       "      <td>Mon Apr 06 22:19:57 PDT 2009</td>\n",
       "      <td>NO_QUERY</td>\n",
       "      <td>Karoli</td>\n",
       "      <td>@nationwideclass no, it's not behaving at all....</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   sentiment          id                          date     query  \\\n",
       "0          0  1467810369  Mon Apr 06 22:19:45 PDT 2009  NO_QUERY   \n",
       "1          0  1467810672  Mon Apr 06 22:19:49 PDT 2009  NO_QUERY   \n",
       "2          0  1467810917  Mon Apr 06 22:19:53 PDT 2009  NO_QUERY   \n",
       "3          0  1467811184  Mon Apr 06 22:19:57 PDT 2009  NO_QUERY   \n",
       "4          0  1467811193  Mon Apr 06 22:19:57 PDT 2009  NO_QUERY   \n",
       "\n",
       "           user_id                                               text  \n",
       "0  _TheSpecialOne_  @switchfoot http://twitpic.com/2y1zl - Awww, t...  \n",
       "1    scotthamilton  is upset that he can't update his Facebook by ...  \n",
       "2         mattycus  @Kenichan I dived many times for the ball. Man...  \n",
       "3          ElleCTF    my whole body feels itchy and like its on fire   \n",
       "4           Karoli  @nationwideclass no, it's not behaving at all....  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.columns = ['sentiment', 'id', 'date', 'query', 'user_id', 'text']\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are going to train only on text to classify its sentiment. So we can ditch the rest of the useless columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = df.drop(['id', 'date', 'query', 'user_id'], axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sentiment</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Negative</td>\n",
       "      <td>@switchfoot http://twitpic.com/2y1zl - Awww, t...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Negative</td>\n",
       "      <td>is upset that he can't update his Facebook by ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Negative</td>\n",
       "      <td>@Kenichan I dived many times for the ball. Man...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Negative</td>\n",
       "      <td>my whole body feels itchy and like its on fire</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Negative</td>\n",
       "      <td>@nationwideclass no, it's not behaving at all....</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  sentiment                                               text\n",
       "0  Negative  @switchfoot http://twitpic.com/2y1zl - Awww, t...\n",
       "1  Negative  is upset that he can't update his Facebook by ...\n",
       "2  Negative  @Kenichan I dived many times for the ball. Man...\n",
       "3  Negative    my whole body feels itchy and like its on fire \n",
       "4  Negative  @nationwideclass no, it's not behaving at all...."
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lab_to_sentiment = {0:\"Negative\", 4:\"Positive\"}\n",
    "def label_decoder(label):\n",
    "    return lab_to_sentiment[label]\n",
    "df.sentiment = df.sentiment.apply(lambda x: label_decoder(x))\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here are decoding the labels. We map **0 -> Negative and 1 -> Positive** as directed by the datset desciption. Now that we decoded we shall now analyse the dataset by its distribution. Because it's important that we have almost small amount of examples for given classes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0.5, 1.0, 'Sentiment Data Distribution')"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfkAAAEICAYAAABceI1YAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAe70lEQVR4nO3de7xdZX3n8c9XIoIX7hnUBA2tqRYZoRIhVq2OWAhoG9pRxKElUkq0ora1nYq2FUVpcWaqwlRxGKGE1gpotaSKxBRwqrYgUREEaklRTCKXmHARAbn0N3+s5+j2uM/ZOyEhyeLzfr32a6/1W896nmefZJ/vXpdzTqoKSZLUP4/Z0hOQJEmbhyEvSVJPGfKSJPWUIS9JUk8Z8pIk9ZQhL0lSTxny0iMgyYeT/OmWnse2LMlnkyzaRH29KMk3B9a/neRlm6Lv1t+1SV6yqfqTNpYhr0etJC9M8s9J7kyyPsmXkjxvE/T72iRfHKxV1eur6t0Pt++NmMs7k/zNiDbfTnJvku8nuaN9TV6fZKzvD0nmJKkkMx7GPCvJD5LcnWRdkkuSvHqwTVUdVlVLxuzrGdO1qaovVNUzN3a+k8Y7J8l7JvX/7Kr6/KboX3o4DHk9KiXZCfg08L+B3YBZwLuAH27JeW1Bv1JVTwKeDpwKvBU46xGew35V9UTgmcA5wF8mOWlTD/JwPoxI25yq8uHjUfcA5gF3jGjzW8D1wO3AMuDpA9sKeD1wA3AH8EEgwM8D9wEPAXdPjEEXWu9pyy8BVgN/BNwG3AwcARwO/BuwHnj7wFiPAU4E/h1YB1wA7Na2zWlzWQR8B/ge8Mdt2wLgfuCBNpevT/E6vw28bFLtQOA/gH3b+suBrwF3AauAdw60/U6bw93t8XzgZ4FL23y/B3wU2GWar3UBz5hUe2X7Wu7e1j8P/HZbfgbw/4A7W//nt/o/tb5+0Oby6oGv91uBW4C/nqhN+hq8Dbiu/Xv/FbBD2/Za4IvD5gssbl/f+9t4/zD5awo8DvgA8N32+ADwuEn/F/5g4P/CsVv6/eGjPw+P5PVo9W/AQ0mWJDksya6DG5MsBN4O/DowE/gC8LFJfbwCeB7wHOBI4NCqup4u/P+lqp5YVbtMMf6TgR3oziC8A/i/wG8ABwAvAv40yd6t7ZvoPgS8GHgqXQh9cFJ/L6Q7Aj4YeEeSn6+qi4E/owvAJ1bVfuN8YQCq6st04fOiVvoBcAywC13g/06SI9q2X2rPu7Rx/oXuA8+ft/n+PLAX8M5xx28uBGbQfeCY7N3A54Bdgdl0Z2Soqom57Nfmcn5bfzLdGZun0wXzMEcDh9J9QPk54E9GTbCqzqT7APM/2ni/MqTZHwPzgf2B/drrGez7ycDOdP8XjgM+OPn/o7SxDHk9KlXVXXTBWHQBuzbJ0iR7tiavB/68qq6vqgfpwnL/JE8f6ObUqrqjqr4DXEb3TXxcDwCnVNUDwHnAHsBpVfX9qrqW7ohyIpRfT3d0vrqqfkgXlq+cdNr5XVV1b1V9Hfj6wL4Px3fpgpGq+nxVXVNV/1FVV9N94HnxVDtW1cqqWl5VP6yqtcD7pms/RR8P0B2l7zZk8wN0gf3Uqrqvqr44pM2g/wBOavO5d4o2f1lVq6pqPXAK8JoNme80jgZOrqrb2tfiXcBvDmx/oG1/oKouojsjsEnuF5AMeT1qtQB/bVXNBvalO+r8QNv8dOC0diPaHXSn0EN3tDXhloHle4AnbsDw66rqobY8ETq3Dmy/d6C/pwOfGpjL9XSXA/YcaP9w5jKVWXSvmyQHJbksydokd9J98Nhjqh2T7JnkvCRrktwF/M107afo47F0Z1HWD9n8R3T/Hl9ud7L/1oju1lbVfSParBpYvonu/8Om8NTW31R9r2sfJCdsqn8/yZCXAKrqX+mum+/bSquA11XVLgOPHavqn8fpbhNPbxVw2KS57FBVazbXXNpPGcwCJo6Q/xZYCuxVVTsDH6YL2anG+LNW/89VtRPdpYgMaTedhcCDwJcnb6iqW6rq+Kp6KvA64EMj7qgf5+uw18Dy0+jOZEB3qeLxExuSPHkD+/4u3Qe1YX1Lm5Uhr0elJM9K8gdJZrf1vehOz17emnwYeFuSZ7ftOyd51Zjd3wrMTrL9Jpruh4FTJi4VJJnZ7hkYdy5zNuDH4XZK8gq6Swh/U1XXtE1PAtZX1X1JDgT+28Bua+lOh//MQO1JdKed70wyC/jvY86XJLslOZruvoP3VtW6IW1eNfFvR3ePQrU5QPeaf2byPmM4IcnsJLvRXUefuJ7/deDZSfZPsgM/fW/BqPE+BvxJ+3fbg+4ejGl/rFHaVAx5PVp9HzgIuCLJD+jC/Rt0dzlTVZ8C3guc1043fwM4bMy+LwWuBW5J8r1NMNfT6I6iP5fk+22uB42578fb87okX52m3T+0vlfRBdz7gGMHtr8BOLm1eQfdHf4AVNU9dNewv9QuKcynu+78XLq73z8DfHKMuX49yd3ASuC3gd+vqndM0fZ5dP92d9N9bX63qm5s294JLGlzOXKMcSf8Ld3NfDfS/STDe9rr+zfgZOAf6X6aYvL1/7OAfdp4fz+k3/cAK4CrgWuAr070LW1uqdrUZxYlSdLWwCN5SZJ6ypCXJKmnDHlJknrKkJckqad694ca9thjj5ozZ86WnoYkSY+Ir3zlK9+rqpnDtvUu5OfMmcOKFSu29DQkSXpEJLlpqm2erpckqacMeUmSesqQlySppwx5SZJ6ypCXJKmnDHlJknpq3D8/+ftJrk3yjSQfS7JDkr2TXJFkZZLzJ/6sZpLHtfWVbfucgX7e1urfTHLoQH1Bq61McuJAfegYkiRptJEh3/4W9JuBeVW1L7AdcBTdn+F8f1U9g+7vOR/XdjkOuL3V39/akWSftt+zgQXAh5Jsl2Q7ur8bfRiwD/Ca1pZpxpAkSSOMe7p+BrBjkhnA44GbgZcCn2jblwBHtOWFbZ22/eAkafXzquqHVfUtur8ZfWB7rKyqG6vqfuA8YGHbZ6oxJEnSCCN/411VrUnyv4DvAPcCnwO+AtxRVQ+2ZquBWW15FrCq7ftgkjuB3Vv98oGuB/dZNal+UNtnqjF+QpLFwGKApz3taaNe0gaZc+JnNml/0pb07VNfvqWnsFF8H6pPHsn34Tin63elOwrfG3gq8AS60+1bjao6s6rmVdW8mTOH/vpeSZIedcY5Xf8y4FtVtbaqHgA+CbwA2KWdvgeYDaxpy2uAvQDa9p2BdYP1SftMVV83zRiSJGmEcUL+O8D8JI9v18kPBq4DLgNe2dosAi5sy0vbOm37pVVVrX5Uu/t+b2Au8GXgSmBuu5N+e7qb85a2faYaQ5IkjTAy5KvqCrqb374KXNP2ORN4K/CWJCvprp+f1XY5C9i91d8CnNj6uRa4gO4DwsXACVX1ULvm/kZgGXA9cEFryzRjSJKkEcb6U7NVdRJw0qTyjXR3xk9uex/wqin6OQU4ZUj9IuCiIfWhY0iSpNH8jXeSJPWUIS9JUk8Z8pIk9ZQhL0lSTxnykiT1lCEvSVJPGfKSJPWUIS9JUk8Z8pIk9ZQhL0lSTxnykiT1lCEvSVJPGfKSJPWUIS9JUk8Z8pIk9ZQhL0lST40M+STPTHLVwOOuJL+XZLcky5Pc0J53be2T5PQkK5NcneS5A30tau1vSLJooH5AkmvaPqcnSasPHUOSJI02MuSr6ptVtX9V7Q8cANwDfAo4EbikquYCl7R1gMOAue2xGDgDusAGTgIOAg4EThoI7TOA4wf2W9DqU40hSZJG2NDT9QcD/15VNwELgSWtvgQ4oi0vBM6tzuXALkmeAhwKLK+q9VV1O7AcWNC27VRVl1dVAedO6mvYGJIkaYQNDfmjgI+15T2r6ua2fAuwZ1ueBawa2Gd1q01XXz2kPt0YkiRphLFDPsn2wK8CH5+8rR2B1yac10+Zbowki5OsSLJi7dq1m3MakiRtMzbkSP4w4KtVdWtbv7Wdaqc939bqa4C9Bvab3WrT1WcPqU83xk+oqjOral5VzZs5c+YGvCRJkvprQ0L+Nfz4VD3AUmDiDvlFwIUD9WPaXfbzgTvbKfdlwCFJdm033B0CLGvb7koyv91Vf8ykvoaNIUmSRpgxTqMkTwB+GXjdQPlU4IIkxwE3AUe2+kXA4cBKujvxjwWoqvVJ3g1c2dqdXFXr2/IbgHOAHYHPtsd0Y0iSpBHGCvmq+gGw+6TaOrq77Se3LeCEKfo5Gzh7SH0FsO+Q+tAxJEnSaP7GO0mSesqQlySppwx5SZJ6ypCXJKmnDHlJknrKkJckqacMeUmSesqQlySppwx5SZJ6ypCXJKmnDHlJknrKkJckqacMeUmSesqQlySppwx5SZJ6ypCXJKmnDHlJknpqrJBPskuSTyT51yTXJ3l+kt2SLE9yQ3vetbVNktOTrExydZLnDvSzqLW/IcmigfoBSa5p+5yeJK0+dAxJkjTauEfypwEXV9WzgP2A64ETgUuqai5wSVsHOAyY2x6LgTOgC2zgJOAg4EDgpIHQPgM4fmC/Ba0+1RiSJGmEkSGfZGfgl4CzAKrq/qq6A1gILGnNlgBHtOWFwLnVuRzYJclTgEOB5VW1vqpuB5YDC9q2narq8qoq4NxJfQ0bQ5IkjTDOkfzewFrgr5J8LclHkjwB2LOqbm5tbgH2bMuzgFUD+69utenqq4fUmWaMn5BkcZIVSVasXbt2jJckSVL/jRPyM4DnAmdU1S8AP2DSafN2BF6bfnrjjVFVZ1bVvKqaN3PmzM05DUmSthnjhPxqYHVVXdHWP0EX+re2U+2059va9jXAXgP7z2616eqzh9SZZgxJkjTCyJCvqluAVUme2UoHA9cBS4GJO+QXARe25aXAMe0u+/nAne2U+zLgkCS7thvuDgGWtW13JZnf7qo/ZlJfw8aQJEkjzBiz3ZuAjybZHrgROJbuA8IFSY4DbgKObG0vAg4HVgL3tLZU1fok7waubO1Orqr1bfkNwDnAjsBn2wPg1CnGkCRJI4wV8lV1FTBvyKaDh7Qt4IQp+jkbOHtIfQWw75D6umFjSJKk0fyNd5Ik9ZQhL0lSTxnykiT1lCEvSVJPGfKSJPWUIS9JUk8Z8pIk9ZQhL0lSTxnykiT1lCEvSVJPGfKSJPWUIS9JUk8Z8pIk9ZQhL0lSTxnykiT1lCEvSVJPGfKSJPXUWCGf5NtJrklyVZIVrbZbkuVJbmjPu7Z6kpyeZGWSq5M8d6CfRa39DUkWDdQPaP2vbPtmujEkSdJoG3Ik/1+qav+qmtfWTwQuqaq5wCVtHeAwYG57LAbOgC6wgZOAg4ADgZMGQvsM4PiB/RaMGEOSJI3wcE7XLwSWtOUlwBED9XOrczmwS5KnAIcCy6tqfVXdDiwHFrRtO1XV5VVVwLmT+ho2hiRJGmHckC/gc0m+kmRxq+1ZVTe35VuAPdvyLGDVwL6rW226+uoh9enG+AlJFidZkWTF2rVrx3xJkiT124wx272wqtYk+U/A8iT/OrixqipJbfrpjTdGVZ0JnAkwb968zToPSZK2FWMdyVfVmvZ8G/Apumvqt7ZT7bTn21rzNcBeA7vPbrXp6rOH1JlmDEmSNMLIkE/yhCRPmlgGDgG+ASwFJu6QXwRc2JaXAse0u+znA3e2U+7LgEOS7NpuuDsEWNa23ZVkfrur/phJfQ0bQ5IkjTDO6fo9gU+1n2qbAfxtVV2c5ErggiTHATcBR7b2FwGHAyuBe4BjAapqfZJ3A1e2didX1fq2/AbgHGBH4LPtAXDqFGNIkqQRRoZ8Vd0I7Dekvg44eEi9gBOm6Ots4Owh9RXAvuOOIUmSRvM33kmS1FOGvCRJPWXIS5LUU4a8JEk9ZchLktRThrwkST1lyEuS1FOGvCRJPWXIS5LUU4a8JEk9ZchLktRThrwkST1lyEuS1FOGvCRJPWXIS5LUU4a8JEk9ZchLktRTY4d8ku2SfC3Jp9v63kmuSLIyyflJtm/1x7X1lW37nIE+3tbq30xy6EB9QautTHLiQH3oGJIkabQNOZL/XeD6gfX3Au+vqmcAtwPHtfpxwO2t/v7WjiT7AEcBzwYWAB9qHxy2Az4IHAbsA7ymtZ1uDEmSNMJYIZ9kNvBy4CNtPcBLgU+0JkuAI9rywrZO235wa78QOK+qflhV3wJWAge2x8qqurGq7gfOAxaOGEOSJI0w7pH8B4A/Av6jre8O3FFVD7b11cCstjwLWAXQtt/Z2v+oPmmfqerTjfETkixOsiLJirVr1475kiRJ6reRIZ/kFcBtVfWVR2A+G6WqzqyqeVU1b+bMmVt6OpIkbRVmjNHmBcCvJjkc2AHYCTgN2CXJjHakPRtY09qvAfYCVieZAewMrBuoTxjcZ1h93TRjSJKkEUYeyVfV26pqdlXNobtx7tKqOhq4DHhla7YIuLAtL23rtO2XVlW1+lHt7vu9gbnAl4ErgbntTvrt2xhL2z5TjSFJkkZ4OD8n/1bgLUlW0l0/P6vVzwJ2b/W3ACcCVNW1wAXAdcDFwAlV9VA7Sn8jsIzu7v0LWtvpxpAkSSOMc7r+R6rq88Dn2/KNdHfGT25zH/CqKfY/BThlSP0i4KIh9aFjSJKk0fyNd5Ik9ZQhL0lSTxnykiT1lCEvSVJPGfKSJPWUIS9JUk8Z8pIk9ZQhL0lSTxnykiT1lCEvSVJPGfKSJPWUIS9JUk8Z8pIk9ZQhL0lSTxnykiT1lCEvSVJPGfKSJPXUyJBPskOSLyf5epJrk7yr1fdOckWSlUnOT7J9qz+ura9s2+cM9PW2Vv9mkkMH6gtabWWSEwfqQ8eQJEmjjXMk/0PgpVW1H7A/sCDJfOC9wPur6hnA7cBxrf1xwO2t/v7WjiT7AEcBzwYWAB9Ksl2S7YAPAocB+wCvaW2ZZgxJkjTCyJCvzt1t9bHtUcBLgU+0+hLgiLa8sK3Tth+cJK1+XlX9sKq+BawEDmyPlVV1Y1XdD5wHLGz7TDWGJEkaYaxr8u2I+yrgNmA58O/AHVX1YGuyGpjVlmcBqwDa9juB3Qfrk/aZqr77NGNMnt/iJCuSrFi7du04L0mSpN4bK+Sr6qGq2h+YTXfk/azNOakNVVVnVtW8qpo3c+bMLT0dSZK2Cht0d31V3QFcBjwf2CXJjLZpNrCmLa8B9gJo23cG1g3WJ+0zVX3dNGNIkqQRxrm7fmaSXdryjsAvA9fThf0rW7NFwIVteWlbp22/tKqq1Y9qd9/vDcwFvgxcCcxtd9JvT3dz3tK2z1RjSJKkEWaMbsJTgCXtLvjHABdU1aeTXAecl+Q9wNeAs1r7s4C/TrISWE8X2lTVtUkuAK4DHgROqKqHAJK8EVgGbAecXVXXtr7eOsUYkiRphJEhX1VXA78wpH4j3fX5yfX7gFdN0dcpwClD6hcBF407hiRJGs3feCdJUk8Z8pIk9ZQhL0lSTxnykiT1lCEvSVJPGfKSJPWUIS9JUk8Z8pIk9ZQhL0lSTxnykiT1lCEvSVJPGfKSJPWUIS9JUk8Z8pIk9ZQhL0lSTxnykiT1lCEvSVJPjQz5JHsluSzJdUmuTfK7rb5bkuVJbmjPu7Z6kpyeZGWSq5M8d6CvRa39DUkWDdQPSHJN2+f0JJluDEmSNNo4R/IPAn9QVfsA84ETkuwDnAhcUlVzgUvaOsBhwNz2WAycAV1gAycBBwEHAicNhPYZwPED+y1o9anGkCRJI4wM+aq6uaq+2pa/D1wPzAIWAktasyXAEW15IXBudS4HdknyFOBQYHlVra+q24HlwIK2baequryqCjh3Ul/DxpAkSSNs0DX5JHOAXwCuAPasqpvbpluAPdvyLGDVwG6rW226+uohdaYZY/K8FidZkWTF2rVrN+QlSZLUW2OHfJInAn8H/F5V3TW4rR2B1yae20+YboyqOrOq5lXVvJkzZ27OaUiStM0YK+STPJYu4D9aVZ9s5VvbqXba822tvgbYa2D32a02XX32kPp0Y0iSpBHGubs+wFnA9VX1voFNS4GJO+QXARcO1I9pd9nPB+5sp9yXAYck2bXdcHcIsKxtuyvJ/DbWMZP6GjaGJEkaYcYYbV4A/CZwTZKrWu3twKnABUmOA24CjmzbLgIOB1YC9wDHAlTV+iTvBq5s7U6uqvVt+Q3AOcCOwGfbg2nGkCRJI4wM+ar6IpApNh88pH0BJ0zR19nA2UPqK4B9h9TXDRtDkiSN5m+8kySppwx5SZJ6ypCXJKmnDHlJknrKkJckqacMeUmSesqQlySppwx5SZJ6ypCXJKmnDHlJknrKkJckqacMeUmSesqQlySppwx5SZJ6ypCXJKmnDHlJknrKkJckqadGhnySs5PcluQbA7XdkixPckN73rXVk+T0JCuTXJ3kuQP7LGrtb0iyaKB+QJJr2j6nJ8l0Y0iSpPGMcyR/DrBgUu1E4JKqmgtc0tYBDgPmtsdi4AzoAhs4CTgIOBA4aSC0zwCOH9hvwYgxJEnSGEaGfFX9E7B+UnkhsKQtLwGOGKifW53LgV2SPAU4FFheVeur6nZgObCgbdupqi6vqgLOndTXsDEkSdIYNvaa/J5VdXNbvgXYsy3PAlYNtFvdatPVVw+pTzfGT0myOMmKJCvWrl27ES9HkqT+edg33rUj8NoEc9noMarqzKqaV1XzZs6cuTmnIknSNmNjQ/7Wdqqd9nxbq68B9hpoN7vVpqvPHlKfbgxJkjSGjQ35pcDEHfKLgAsH6se0u+znA3e2U+7LgEOS7NpuuDsEWNa23ZVkfrur/phJfQ0bQ5IkjWHGqAZJPga8BNgjyWq6u+RPBS5IchxwE3Bka34RcDiwErgHOBagqtYneTdwZWt3clVN3Mz3Bro7+HcEPtseTDOGJEkaw8iQr6rXTLHp4CFtCzhhin7OBs4eUl8B7Dukvm7YGJIkaTz+xjtJknrKkJckqacMeUmSesqQlySppwx5SZJ6ypCXJKmnDHlJknrKkJckqacMeUmSesqQlySppwx5SZJ6ypCXJKmnDHlJknrKkJckqacMeUmSesqQlySppwx5SZJ6aqsP+SQLknwzycokJ27p+UiStK3YqkM+yXbAB4HDgH2A1yTZZ8vOSpKkbcNWHfLAgcDKqrqxqu4HzgMWbuE5SZK0TZixpScwwixg1cD6auCgyY2SLAYWt9W7k3zzEZibNq09gO9t6Un0Xd67pWegrZzvw0fAZngfPn2qDVt7yI+lqs4EztzS89DGS7KiquZt6XlIj2a+D/tnaz9dvwbYa2B9dqtJkqQRtvaQvxKYm2TvJNsDRwFLt/CcJEnaJmzVp+ur6sEkbwSWAdsBZ1fVtVt4Wto8vNwibXm+D3smVbWl5yBJkjaDrf10vSRJ2kiGvCRJPWXIa4MlqSR/MbD+h0neuRnGefuk9X/e1GNIfZDkoSRXJflGko8nefwG7v/UJJ9oy/snOXxg26/6K8W3XYa8NsYPgV9PssdmHucnQr6qfnEzjydtq+6tqv2ral/gfuD1G7JzVX23ql7ZVvcHDh/YtrSqTt1kM9UjypDXxniQ7i7c35+8IcnMJH+X5Mr2eMFAfXmSa5N8JMlNEx8Skvx9kq+0bYtb7VRgx3Z08tFWu7s9n5fk5QNjnpPklUm2S/I/27hXJ3ndZv9KSFufLwDPSLJbe29dneTyJM8BSPLi9r66KsnXkjwpyZx2FmB74GTg1W37q5O8NslfJtm5vW8f0/p5QpJVSR6b5GeTXNzex19I8qwt+Po1wJDXxvogcHSSnSfVTwPeX1XPA/4r8JFWPwm4tKqeDXwCeNrAPr9VVQcA84A3J9m9qk7kx0cnR08a43zgSID2Telg4DPAccCdbeznAccn2XsTvV5pq5dkBt0f9LoGeBfwtap6Dt1ZsXNbsz8ETqiq/YEXAfdO7N/+Rsg7gPPbe+/8gW13AlcBL26lVwDLquoBug/9b2rv4z8EPrS5XqM2zFb9c/LaelXVXUnOBd7MwDcJ4GXAPkkm1ndK8kTghcCvtX0vTnL7wD5vTvJrbXkvYC6wbprhPwucluRxwALgn6rq3iSHAM9JMnHacefW17c29nVK24gdk1zVlr8AnAVcQfdBm6q6NMnuSXYCvgS8r50h+2RVrR54v45yPvBq4DK6X072ofb+/kXg4wP9PO7hvyRtCoa8Ho4PAF8F/mqg9hhgflXdN9hwqm8iSV5C98Hg+VV1T5LPAztMN2hV3dfaHUr3Dee8ie7ojiaWbdjLkLZ597Yj8x+Z6j1XVacm+QzddfcvJTkUuG9o45+2FPizJLsBBwCXAk8A7pg8vrYOnq7XRquq9cAFdKfJJ3wOeNPESpL92+KX+PEp9kOAXVt9Z+D2FvDPAuYP9PVAksdOMfz5wLF0pxsvbrVlwO9M7JPk55I8YeNenbTN+wJwNPzow/T32hm4n62qa6rqvXS/Onzy9fPvA08a1mFV3d32OQ34dFU9VFV3Ad9K8qo2VpLstzlekDacIa+H6y/o/jzlhDcD89rNPtfx47t83wUckuQbwKuAW+i+mVwMzEhyPXAqcPlAX2cCV0/ceDfJ5+iuDf5ju44I3fX/64CvtnH+D56t0qPXO4EDklxN995a1Oq/126yuxp4gO7y16DL6C65XZXk1UP6PR/4jfY84WjguCRfB64FFm66l6GHw19rq0dEu37+UPt7BM8HzvD0niRtXh7l6JHyNOCC9uM39wPHb+H5SFLveSQvSVJPeU1ekqSeMuQlSeopQ16SpJ4y5CVJ6ilDXpKknvr/igDZgG3vDVAAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 576x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "val_count = df.sentiment.value_counts()\n",
    "\n",
    "plt.figure(figsize=(8,4))\n",
    "plt.bar(val_count.index, val_count.values)\n",
    "plt.title(\"Sentiment Data Distribution\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's a very good dataset without any skewness.\n",
    "\n",
    "Now let us explore the data we having here... "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sentiment</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>84260</th>\n",
       "      <td>Negative</td>\n",
       "      <td>Doesn't want to go to work right now</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1402618</th>\n",
       "      <td>Positive</td>\n",
       "      <td>@Traphik this: http://bit.ly/8DhRM , is HILARI...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1599027</th>\n",
       "      <td>Positive</td>\n",
       "      <td>@twistedsunshine I see. But I don't want to sy...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>670648</th>\n",
       "      <td>Negative</td>\n",
       "      <td>@anticlimatic I CAN'T CONNECT TO IT FOR SOME R...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1363208</th>\n",
       "      <td>Positive</td>\n",
       "      <td>@scottsayare yes sir  in fact, it's summer break!</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1147124</th>\n",
       "      <td>Positive</td>\n",
       "      <td>@awildberry how are u feeling?i dont feel so h...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1589797</th>\n",
       "      <td>Positive</td>\n",
       "      <td>@FunkyChunks Yes, it's a good thing  How are y...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>417501</th>\n",
       "      <td>Negative</td>\n",
       "      <td>I saw a guy that reminds me of a guy that I re...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>504185</th>\n",
       "      <td>Negative</td>\n",
       "      <td>@Chriscorrell I am so jealous! I haven't had R...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>707348</th>\n",
       "      <td>Negative</td>\n",
       "      <td>rt: Looks like Chicago was overlooked in the T...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        sentiment                                               text\n",
       "84260    Negative              Doesn't want to go to work right now \n",
       "1402618  Positive  @Traphik this: http://bit.ly/8DhRM , is HILARI...\n",
       "1599027  Positive  @twistedsunshine I see. But I don't want to sy...\n",
       "670648   Negative  @anticlimatic I CAN'T CONNECT TO IT FOR SOME R...\n",
       "1363208  Positive  @scottsayare yes sir  in fact, it's summer break!\n",
       "1147124  Positive  @awildberry how are u feeling?i dont feel so h...\n",
       "1589797  Positive  @FunkyChunks Yes, it's a good thing  How are y...\n",
       "417501   Negative  I saw a guy that reminds me of a guy that I re...\n",
       "504185   Negative  @Chriscorrell I am so jealous! I haven't had R...\n",
       "707348   Negative  rt: Looks like Chicago was overlooked in the T..."
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "random_idx_list = [random.randint(1,len(df.text)) for i in range(10)] # creates random indexes to choose from dataframe\n",
    "df.loc[random_idx_list,:].head(10) # Returns the rows with the index and display it"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looks like we have a nasty data in text. Because in general we use lot of punctuations and other words without any contextual meaning. It have no value as feature to the model we are training. So we need to get rid of them.\n",
    "\n",
    "# Text Preprocessing\n",
    "Tweet texts often consists of other user mentions, hyperlink texts, emoticons and punctuations. In order to use them for learning using a Language Model. We cannot permit those texts for training a model. So we have to clean the text data using various preprocessing and cleansing methods. Let's continue\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Hyperlinks and Mentions\n",
    "Twitter is a social media platform where people can tag and mentions other people's ID and share videos and blogs from internet. So the tweets often contain lots of Hyperlinks and twitter mentions.\n",
    "\n",
    "- Twitter User Mentions - Eg. @arunrk7, @andrewng\n",
    "- Hyperlinks - Eg. https://keras.io, https://tensorflow.org\n",
    "\n",
    "**NLTK** is a python library which got functions to perform text processing task for NLP."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "text_cleaning_re = \"@\\S+|https?:\\S+|http?:\\S|[^A-Za-z0-9]+\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def preprocess(text):\n",
    "    text = re.sub(text_cleaning_re, ' ', str(text).lower()).strip()\n",
    "    words = nltk.tokenize.word_tokenize(text)\n",
    "    pos = nltk.pos_tag(words)        \n",
    "    pos = [tag[1] for tag in pos]\n",
    "    pos = ' '.join(pos)\n",
    "    return pos"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "if not os.path.exists('pos.csv'):\n",
    "    df.text = df.text.apply(lambda x: preprocess(x))\n",
    "    df.to_csv('pos.csv')\n",
    "df = pd.read_csv('pos.csv')\n",
    "df.text = df.text.astype(str)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train and Test Split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "TRAIN_SIZE = 0.85\n",
    "MAX_SEQUENCE_LENGTH = 30"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train Data size: 1359999\n",
      "Test Data size 240001\n"
     ]
    }
   ],
   "source": [
    "train_data, test_data = train_test_split(df, test_size=1-TRAIN_SIZE,random_state=2022)\n",
    "print(\"Train Data size:\", len(train_data))\n",
    "print(\"Test Data size\", len(test_data))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`train_test_split` will shuffle the dataset and split it to gives training and testing dataset. It's important to shuffle our dataset before training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>sentiment</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>975197</th>\n",
       "      <td>975197</td>\n",
       "      <td>Positive</td>\n",
       "      <td>NN DT NN RB PRP$ NN NNS RP DT NN CC DT NNS VBG...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1093428</th>\n",
       "      <td>1093428</td>\n",
       "      <td>Positive</td>\n",
       "      <td>VBG NN IN NN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1389857</th>\n",
       "      <td>1389857</td>\n",
       "      <td>Positive</td>\n",
       "      <td>WRB JJ</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>830173</th>\n",
       "      <td>830173</td>\n",
       "      <td>Positive</td>\n",
       "      <td>NN RB RB CD NNS IN NN CD IN CC RB RB JJ NN CD ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>105232</th>\n",
       "      <td>105232</td>\n",
       "      <td>Negative</td>\n",
       "      <td>NN VBP PRP RB</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26109</th>\n",
       "      <td>26109</td>\n",
       "      <td>Negative</td>\n",
       "      <td>NN NN VBP DT NNS RB JJ WRB NN VBP DT NN NN NN ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1382867</th>\n",
       "      <td>1382867</td>\n",
       "      <td>Positive</td>\n",
       "      <td>NN CD CD VBZ VBN JJ RB DT NN IN PRP RB VB NNS ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>350153</th>\n",
       "      <td>350153</td>\n",
       "      <td>Negative</td>\n",
       "      <td>NN NN VBP JJ VBD JJ NN IN RB IN RB RB CD JJ NN...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1452274</th>\n",
       "      <td>1452274</td>\n",
       "      <td>Positive</td>\n",
       "      <td>DT NN CC NN WRB JJ VBP VBN PRP</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>366595</th>\n",
       "      <td>366595</td>\n",
       "      <td>Negative</td>\n",
       "      <td>JJ NN NN VBD JJ VB RB RB JJ RB VB VBG WP VB VB...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         Unnamed: 0 sentiment  \\\n",
       "975197       975197  Positive   \n",
       "1093428     1093428  Positive   \n",
       "1389857     1389857  Positive   \n",
       "830173       830173  Positive   \n",
       "105232       105232  Negative   \n",
       "26109         26109  Negative   \n",
       "1382867     1382867  Positive   \n",
       "350153       350153  Negative   \n",
       "1452274     1452274  Positive   \n",
       "366595       366595  Negative   \n",
       "\n",
       "                                                      text  \n",
       "975197   NN DT NN RB PRP$ NN NNS RP DT NN CC DT NNS VBG...  \n",
       "1093428                                       VBG NN IN NN  \n",
       "1389857                                             WRB JJ  \n",
       "830173   NN RB RB CD NNS IN NN CD IN CC RB RB JJ NN CD ...  \n",
       "105232                                       NN VBP PRP RB  \n",
       "26109    NN NN VBP DT NNS RB JJ WRB NN VBP DT NN NN NN ...  \n",
       "1382867  NN CD CD VBZ VBN JJ RB DT NN IN PRP RB VB NNS ...  \n",
       "350153   NN NN VBP JJ VBD JJ NN IN RB IN RB RB CD JJ NN...  \n",
       "1452274                     DT NN CC NN WRB JJ VBP VBN PRP  \n",
       "366595   JJ NN NN VBD JJ VB RB RB JJ RB VB VBG WP VB VB...  "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_data.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 1359999 entries, 975197 to 349053\n",
      "Data columns (total 3 columns):\n",
      " #   Column      Non-Null Count    Dtype \n",
      "---  ------      --------------    ----- \n",
      " 0   Unnamed: 0  1359999 non-null  int64 \n",
      " 1   sentiment   1359999 non-null  object\n",
      " 2   text        1359999 non-null  object\n",
      "dtypes: int64(1), object(2)\n",
      "memory usage: 41.5+ MB\n"
     ]
    }
   ],
   "source": [
    "train_data.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tokenization\n",
    "Given a character sequence and a defined document unit, tokenization is the task of chopping it up into pieces, called *tokens* , perhaps at the same time throwing away certain characters, such as punctuation. The process is called **Tokenization.**\n",
    "![Tokenization](https://cdn.analyticsvidhya.com/wp-content/uploads/2019/11/tokenization.png)\n",
    "\n",
    "`tokenizer` create tokens for every word in the data corpus and map them to a index using dictionary.\n",
    "\n",
    "`word_index` contains the index for each word\n",
    "\n",
    "`vocab_size` represents the total number of word in the data corpus"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Vocabulary Size : 37\n"
     ]
    }
   ],
   "source": [
    "tokenizer = Tokenizer()\n",
    "tokenizer.fit_on_texts(train_data.text)\n",
    "\n",
    "word_index = tokenizer.word_index\n",
    "vocab_size = len(tokenizer.word_index) + 1\n",
    "print(\"Vocabulary Size :\", vocab_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we got a `tokenizer` object, which can be used to covert any word into a Key in dictionary (number).\n",
    "\n",
    "Since we are going to build a sequence model. We should feed in a sequence of numbers to it. And also we should ensure there is no variance in input shapes of sequences. It all should be of same lenght. But texts in tweets have different count of words in it. To avoid this, we seek a little help from `pad_sequence` to do our job. It will make all the sequence in one constant length `MAX_SEQUENCE_LENGTH`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training X Shape: (1359999, 30)\n",
      "Testing X Shape: (240001, 30)\n"
     ]
    }
   ],
   "source": [
    "x_train = pad_sequences(tokenizer.texts_to_sequences(train_data.text), maxlen = MAX_SEQUENCE_LENGTH)\n",
    "x_test = pad_sequences(tokenizer.texts_to_sequences(test_data.text), maxlen = MAX_SEQUENCE_LENGTH)\n",
    "\n",
    "print(\"Training X Shape:\",x_train.shape)\n",
    "print(\"Testing X Shape:\",x_test.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "labels = train_data.sentiment.unique().tolist()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Label Encoding \n",
    "We are building the model to predict class in enocoded form (0 or 1 as this is a binary classification). We should encode our training labels to encodings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "y_train shape: (1359999, 1)\n",
      "y_test shape: (240001, 1)\n"
     ]
    }
   ],
   "source": [
    "encoder = LabelEncoder()\n",
    "encoder.fit(train_data.sentiment.to_list())\n",
    "\n",
    "y_train = encoder.transform(train_data.sentiment.to_list())\n",
    "y_test = encoder.transform(test_data.sentiment.to_list())\n",
    "\n",
    "y_train = y_train.reshape(-1,1)\n",
    "y_test = y_test.reshape(-1,1)\n",
    "\n",
    "print(\"y_train shape:\", y_train.shape)\n",
    "print(\"y_test shape:\", y_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Word Emdedding\n",
    "In Language Model, words are represented in a way to intend more meaning and for learning the patterns and contextual meaning behind it. \n",
    "\n",
    "**Word Embedding** is one of the popular representation of document vocabulary.It is capable of capturing context of a word in a document, semantic and syntactic similarity, relation with other words, etc.\n",
    "\n",
    "Basically, it's a feature vector representation of words which are used for other natural language processing applications."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "def word_vector(model, tokens, dim):\n",
    "    \"\"\" \n",
    "    Generate a word vector.\n",
    "    model: a completed model\n",
    "    tokens: a list of words (in this case POS)\n",
    "    dim: Number of dimensions. 100 - 300 is good for w2v\n",
    "    \"\"\"\n",
    "    \n",
    "    i = 0\n",
    "    vec = np.zeros(dim).reshape((1, dim))\n",
    "    for word in tokens:\n",
    "        vec += model[word].reshape((1, dim))\n",
    "        i += 1.\n",
    "    if i != 0:\n",
    "        vec /= i\n",
    "    return vec"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "DIM = 300\n",
    "pos = df.text\n",
    "w2v_sg_model = Word2Vec(sentences=pos, vector_size=DIM, workers=18, seed=2022)\n",
    "\n",
    "embedding_matrix = np.zeros((vocab_size, DIM)) \n",
    "for i in range(vocab_size):\n",
    "    embedding_matrix[i,:] = word_vector(w2v_sg_model.wv, pos.iloc[i], DIM)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "embedding_layer = tf.keras.layers.Embedding(vocab_size, DIM, weights=[embedding_matrix], input_length=MAX_SEQUENCE_LENGTH, trainable=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "BATCH_SIZEretrained_embedding_layer(word_to_vec_map, word_to_index, max_le# Model Training - LSTM\n",
    "We are clear to build our Deep Learning model. While developing a DL model, we should keep in mind of key things like Model Architecture, Hyperparmeter Tuning and Performance of the model.\n",
    "\n",
    "As you can see in the word cloud, the some words are predominantly feature in both Positive and Negative tweets. This could be a problem if we are using a Machine Learning model like Naive Bayes, SVD, etc.. That's why we use **Sequence Models**.\n",
    "\n",
    "### Sequence Model\n",
    "![Sequence Model](https://miro.medium.com/max/1458/1*SICYykT7ybua1gVJDNlajw.png)\n",
    "\n",
    "Reccurent Neural Networks can handle a seqence of data and learn a pattern of input seqence to give either sequence or scalar value as output. In our case, the Neural Network outputs a scalar value prediction. \n",
    "\n",
    "For model architecture, we use\n",
    "\n",
    "1) **Embedding Layer** - Generates Embedding Vector for each input sequence.\n",
    "\n",
    "2) **LSTM** - Long Short Term Memory, its a variant of RNN which has memory state cell to learn the context of words which are at further along the text to carry contextual meaning rather than just neighbouring words as in case of RNN.\n",
    "\n",
    "3) **Dense** - Fully Connected Layers for classification\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = Sequential([\n",
    "    embedding_layer,\n",
    "    Dropout(rate=0.5),\n",
    "    Bidirectional(LSTM(units=128, return_sequences=True)),\n",
    "    Dropout(rate=0.5),\n",
    "    Bidirectional(LSTM(units=128, return_sequences=False)),\n",
    "    Dense(units=1, activation='sigmoid')\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Optimization Algorithm\n",
    "This notebook uses Adam, optimization algorithm for Gradient Descent. You can learn more about Adam [here](https://www.tensorflow.org/api_docs/python/tf/keras/optimizers/Adam)\n",
    "\n",
    "### Callbacks\n",
    "Callbacks are special functions which are called at the end of an epoch. We can use any functions to perform specific operation after each epoch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
    "callbacks = [\n",
    "    ReduceLROnPlateau(monitor='val_loss', patience=5, cooldown=0),\n",
    "    EarlyStopping(monitor='val_accuracy', min_delta=1e-4, patience=5)\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start training... It takes a heck of a time if training in CPU, be sure your GPU turned on... May the CUDA Cores be with you...."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From <ipython-input-24-d4a4e34b7f9f>:1: is_gpu_available (from tensorflow.python.framework.test_util) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use `tf.config.list_physical_devices('GPU')` instead.\n",
      "Training on GPU...\n"
     ]
    }
   ],
   "source": [
    "print(\"Training on GPU...\") if tf.test.is_gpu_available() else print(\"Training on CPU...\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "10625/10625 [==============================] - 167s 15ms/step - loss: 0.6722 - accuracy: 0.5778 - val_loss: 0.6554 - val_accuracy: 0.6094 - lr: 0.0010\n",
      "Epoch 2/10\n",
      "10625/10625 [==============================] - 159s 15ms/step - loss: 0.6501 - accuracy: 0.6159 - val_loss: 0.6409 - val_accuracy: 0.6271 - lr: 0.0010\n",
      "Epoch 3/10\n",
      "10625/10625 [==============================] - 159s 15ms/step - loss: 0.6424 - accuracy: 0.6248 - val_loss: 0.6353 - val_accuracy: 0.6340 - lr: 0.0010\n",
      "Epoch 4/10\n",
      "10625/10625 [==============================] - 159s 15ms/step - loss: 0.6382 - accuracy: 0.6299 - val_loss: 0.6327 - val_accuracy: 0.6371 - lr: 0.0010\n",
      "Epoch 5/10\n",
      "10625/10625 [==============================] - 160s 15ms/step - loss: 0.6353 - accuracy: 0.6331 - val_loss: 0.6313 - val_accuracy: 0.6392 - lr: 0.0010\n",
      "Epoch 6/10\n",
      "10625/10625 [==============================] - 159s 15ms/step - loss: 0.6331 - accuracy: 0.6357 - val_loss: 0.6298 - val_accuracy: 0.6400 - lr: 0.0010\n",
      "Epoch 7/10\n",
      "10625/10625 [==============================] - 159s 15ms/step - loss: 0.6313 - accuracy: 0.6378 - val_loss: 0.6285 - val_accuracy: 0.6415 - lr: 0.0010\n",
      "Epoch 8/10\n",
      "10625/10625 [==============================] - 159s 15ms/step - loss: 0.6300 - accuracy: 0.6391 - val_loss: 0.6292 - val_accuracy: 0.6426 - lr: 0.0010\n",
      "Epoch 9/10\n",
      "10625/10625 [==============================] - 159s 15ms/step - loss: 0.6289 - accuracy: 0.6401 - val_loss: 0.6277 - val_accuracy: 0.6423 - lr: 0.0010\n",
      "Epoch 10/10\n",
      "10625/10625 [==============================] - 160s 15ms/step - loss: 0.6277 - accuracy: 0.6415 - val_loss: 0.6280 - val_accuracy: 0.6431 - lr: 0.0010\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(\n",
    "    x_train,\n",
    "    y_train,\n",
    "    batch_size=128,\n",
    "    epochs=10,\n",
    "    validation_data=(x_test, y_test),\n",
    "    callbacks=callbacks,\n",
    "    verbose=1,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Model Evaluation\n",
    "Now that we have trained the model, we can evaluate its performance. We will some evaluation metrics and techniques to test the model.\n",
    "\n",
    "Let's start with the Learning Curve of loss and accuracy of the model on each epoch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7fc8e4597910>"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEWCAYAAAB8LwAVAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAABDQElEQVR4nO3deXxU1fn48c8zk0kmC5AQCAgBQUUEQUFx135d0dpaWxdoa23RqrXVulRttV+/rbV77aJtbd1rF6sodWtrtWix/twJCLKDbBJkCYFAtkkmM8/vj3MnmYRJMiEzTJbn/Xrd19y5c5dnLuE+c8659xxRVYwxxpi2fJkOwBhjTM9kCcIYY0xCliCMMcYkZAnCGGNMQpYgjDHGJGQJwhhjTEKWIIwBRORREflBkutuEJEz0x2TMZlmCcIYY0xCliCM6UNEJCvTMZi+wxKE6TW8qp1bROR9EakVkYdFZJiI/EtEqkXkZREpilv/UyKyTESqRORVEZkQ99lUEVnobTcbCLY51idFZJG37ZsickSSMX5CRN4TkT0isklE7mjz+cne/qq8z2d5y3NF5BcislFEdovI696yU0WkPMF5ONObv0NE5ojIX0RkDzBLRI4Vkbe8Y2wRkd+KSHbc9oeLyFwR2Ski20Tk2yIyXETqRKQ4br2jRKRCRALJfHfT91iCML3NhcBZwKHAecC/gG8DQ3F/z9cBiMihwOPADd5nLwB/F5Fs72L5LPBnYDDwlLdfvG2nAo8AXwGKgfuB50UkJ4n4aoEvAoXAJ4Cvisinvf0e6MX7Gy+mKcAib7ufA0cDJ3oxfROIJnlOzgfmeMd8DIgANwJDgBOAM4CveTEMAF4GXgRGAIcAr6jqVuBVYEbcfi8FnlDVcJJxmD7GEoTpbX6jqttUdTPw/4B3VPU9VQ0BzwBTvfVmAv9U1bneBe7nQC7uAnw8EADuVtWwqs4B5scd4yrgflV9R1UjqvpHoMHbrkOq+qqqLlHVqKq+j0tS/+N9/HngZVV93DtupaouEhEfcDlwvapu9o75pqo2JHlO3lLVZ71j1qvqAlV9W1WbVHUDLsHFYvgksFVVf6GqIVWtVtV3vM/+CHwBQET8wOdwSdT0U5YgTG+zLW6+PsH7Am9+BLAx9oGqRoFNwEjvs83auqfKjXHzBwI3eVU0VSJSBYzytuuQiBwnIvO8qpndwNW4X/J4+1ibYLMhuCquRJ8lY1ObGA4VkX+IyFav2ulHScQA8BwwUUTG4kppu1X13X2MyfQBliBMX/UR7kIPgIgI7uK4GdgCjPSWxYyOm98E/FBVC+OmPFV9PInj/hV4HhilqoOA+4DYcTYBByfYZgcQauezWiAv7nv4cdVT8dp2yfx7YCUwTlUH4qrg4mM4KFHgXinsSVwp4lKs9NDvWYIwfdWTwCdE5AyvkfUmXDXRm8BbQBNwnYgEROQC4Ni4bR8ErvZKAyIi+V7j84AkjjsA2KmqIRE5FletFPMYcKaIzBCRLBEpFpEpXunmEeCXIjJCRPwicoLX5rEaCHrHDwC3A521hQwA9gA1InIY8NW4z/4BHCAiN4hIjogMEJHj4j7/EzAL+BSWIPo9SxCmT1LVVbhfwr/B/UI/DzhPVRtVtRG4AHch3Ilrr3g6btsy4Ergt8Au4ANv3WR8DbhTRKqB7+ASVWy/HwLn4pLVTlwD9ZHexzcDS3BtITuBnwI+Vd3t7fMhXOmnFmh1V1MCN+MSUzUu2c2Oi6EaV310HrAVWAOcFvf5G7jG8YWqGl/tZvohsQGDjDHxROQ/wF9V9aFMx2IyyxKEMaaZiBwDzMW1oVRnOh6TWVbFZIwBQET+iHtG4gZLDgasBGGMMaYdVoIwxhiTUJ/p2GvIkCE6ZsyYTIdhjDG9yoIFC3aoattna4A+lCDGjBlDWVlZpsMwxpheRUTavZ3ZqpiMMcYkZAnCGGNMQn2miskY00c1NUFNjZtqa91rfT2IQFYW+P1dmxJt4/O5/aWCKjQ0QCjk4gyFuj51dbsjjoB//CM18cfp0wkiHA5TXl5OKBTKdCh9WjAYpLS0lEDAxpXp16JRqKtrfSGPnxItS2bdhmR7Pe8mn6/riQX2vlinIt5gcO8pN7dlfsiQ1p+NG9f9YyaQ1gQhIucA9wB+4CFV/UmCdWYAd+B6pFysqp+P+2wgsBx4VlWv7erxy8vLGTBgAGPGjEFS9evAtKKqVFZWUl5eztixYzMdjtkXTU2wezdUVbV+bW++vQt5bW3yxxSB/HwoKGg9FRXBqFGtlyVaLxh0v9QjETc1NbXMJzulYhtofeHubEpi3Yg/m5paYc8e2LPHnfLYfHvLxuyEu1L/l5G+BOF1S3wvrmOwcmC+iDyvqsvj1hkH3AacpKq7RKSkzW6+D7y2rzGEQiFLDmkmIhQXF1NRUZHpUPqnaNRdnLtygW+7rK6u8+MMGACDBrlpwAB3kR46dO8Ld6KLeaLlubmpq9LpISIRqK7u4EK+uf0LfvzymprkjldQ4P45Bg50pzMd0lmCOBb4QFXXAYjIE7ihEZfHrXMlcK+q7gJQ1e2xD0TkaGAYbmjEafsahCWH9LNz3I5odO865fjXRMs6+ix2FYm/wO/e7X5JdyQnBwoLWy7whYVQWrr3srbzsdeBA1uqU3o5VXc6a2tbT3V1Hb9vb1ltbcs/S7IFqAED3CmNndpBg1yhaeDA1svjp7bLCgr2zz9JOhPESFqPdFUOHNdmnUMBROQNXDXUHar6ojcE4y9w3TWf2d4BROQq3PCQjB49ur3VjOmcqvtfXlEB27e3ft2zp+sX81TURceqI3Jz3RT7FT92bPsX9UQX+JxkhtLuWVRbcmJ1dcsv89h8dXVyF/BEF/6u9i4UDLoCUPyUlwclJW4+/uLd2cW9oMA1dfQWmW6kzgLGAacCpcBrIjIZlxheUNXyjn6dquoDwAMA06ZNs06lTAtVd0VJdMFvb1k4nHhfOTmt649jF+zYfFHR3svi5/dlWXZ2r6uCiUbdRbjtxXxf56PR5I6bnd36wh2bLypyBaX4ZW3X6ex9Xl7vuqCnWjoTxGbcEI8xpd6yeOW4QefDwHoRWY1LGCcAp4jI13BjDGeLSI2q3prGeNOioKCAmjaViqtWreIrX/kKVVVVNDQ0cMopp3DhhRfyrW99C4APPviAkSNHkpubyxFHHMHll1/OaaedxoMPPsgVV1wBwKJFi5g6dSp33XUXN998c8JjP/roo0yfPp0RIzodSrmV++67j7y8PL74xS/uwzdOE1VXOdv2wt7Rxb+xMfG+Cgrcz7+hQ13Z/uij3XxsWew1NgWD+/e7Zlh9fctpbDtVVLiCVqILe01Ncr/O/X73a3rAgNbVLaWley9vbz7WlJGX524uMumRzlM7HxjnDYC+GfgsrYdfBHgW+BzwBxEZgqtyWqeql8RWEJFZwLTemBzac91113HjjTdy/vnnA7BkyRImT57M2WefDcCpp57Kz3/+c6ZNc00vr776KpMmTeLJJ59sThCPP/44Rx55ZOIDeB599FEmTZqUMEFEIhH87VRiXn311fv83fZJNArbtsH69W7asMG9lpe3vuC3V2WTn99yYR8xAqZM2ftCHz+frha9HqqpCSorE1/wE03tNZLm5rrTV1joLtTFxa62q7OLedv5YLDXFY76rbQlCFVtEpFrgZdw7QuPqOoyEbkTKFPV573PpovIciAC3KKqlemI54YbYNGi1O5zyhS4++6ub7dlyxZKS0ub30+ePLnTbQ488ED27NnDtm3bKCkp4cUXX+Tcc89td/05c+ZQVlbGJZdcQm5uLm+99RYTJkxg5syZzJ07l29+85tUV1fzwAMP0NjYyCGHHMKf//xn8vLyuOOOOygoKODmm2/m1FNP5bjjjmPevHlUVVXx8MMPc8opp3TtC6u6K1TbBBD/2vbiP2yY+3U/fDhMnrz3r/v4C35eXtfi6eVizSXJXOwrKtypT/TL3u9vfToPOsi9tjfl5+//72oyK62FM1V9AXihzbLvxM0r8A1vam8fjwKPpifCzLjxxhs5/fTTOfHEE5k+fTqXXXYZhYWFnW530UUX8dRTTzF16lSOOuoocjpofLzooov47W9/26okAlBcXMzChQsBqKys5MorrwTg9ttv5+GHH+brX//6Xvtqamri3Xff5YUXXuB73/seL7/88t4HjEZdBo5d9NsmgLY/SwcPdj8/J02C885z82PGuNcDD+x3F31wF/EdO2Dz5papvNy9fvSRK2TFLvztNZcUFbVc0CdO7PiCX1jYv+vXTef6Te3dvvzST5fLLruMs88+mxdffJHnnnuO+++/n8WLF3d4wQeYMWMGM2fOZOXKlXzuc5/jzTff7PKxZ86c2Ty/dOlSbr/9dqqqqqipqWmu4mrrggsugEiEoydOZMO6de4K1dDgpsZG97ptG3z84y0bFRS4i/1BB8EZZ7Rc/GOJYODALsfemzU0uIt8/MU/PgHEkkDbZhMRV4gaOdJNU6e2f8EfMgTsYXaTSv0mQfQ0I0aM4PLLL+fyyy9n0qRJLF26lKOPPrrDbYYPH04gEGDu3Lncc889+5Qg8uPqCWbNmsWzzz7LkUceyaOPPsqrr77qWihra13F9dq1UFdHzoYNkJODv6qKplAIPvzQXblyctyUn++ugE891ZIABg/uFxXNqrBr194X/rYJYMeOvbfNy2u58J90Ust8aWnL/PDh1ghrMsf+9DLgxRdf5IwzziAQCLB161YqKysZOXJkUtveeeedbN++vd0G5ngDBgygurr9oYWrq6s5YOhQwtu28dhDDzFy8GBYtsxVcOfluZvGRVzL4siR7ld/IABHHumuWvEJoK4Ojj8+qe/QW0SjsGWLy4ftJYDNm11ObaukxJ2yUaPcaYld8OMTwKBB/SKHml7MEkSa1dXVtWqQ/sY3vkF5eTnXX389Qe/2ybvuuovhw4cntb8TTzwx6WPPmjWLq6++urmRGmi5XXTPHr5/zTUcd8wxDC0s5LjJk6kOh139f0mJu3pNnuxuXRkxAg44wCUHkT5Tj9G27bzttHHj3lU+OTnudJSWwrRpcP75rS/+I0e6U9ULn00zZi+iXX2ssIeaNm2ath1RbsWKFUyYMCFDEfUgjY2tO4CJdTIW/xhofn63fs721HNdU9N+Atiwwd2/Hy9262b8NHp0y8W/uNh+9Zu+RUQWqGrC7oysBNEXxXfgFusmAtwv/6Kiluf++0DldmOj+6XfXhJoW/efn99y4T/ttNaJoB+2nRvTod5/hTBc87Wv8cbrr7vEEIlANMr1M2dy2fnntzzRNGhQr3xCKRJxd/e0lwA2b259j38g4GrJxo6FCy7YuzQwZEivOwXGZIwliN6qqam5r4N7r74aLr/cLQ8GW/ou2F9dPqbIrl1QVgbvvgvz57v28o0bW9/zL+KqesaOhdNP3zsBjBjRq76yMT2aJYjeQtXdKdS203ifzyWEAw5wr72kdbS+Ht57ryUZvPsufPBBy+eHHuru+b/wwr3bA3rJVzSm17ME0ZM1NrYeUaSpyS3Py2tJCPn5Pf5x2KYmVxqIJYP582HJkpa28pEj4dhjXSHo2GNd33lJPFhujEkzSxA9SaxxOTYCSXzjcnyn8j34NlNV93xdrFQwfz4sXNjyVYqK4Jhj4NZbXTI45hiX64wxPY8liJ5A1fWqVl7ukoSIaz+IPU3Vg4dnbGx0NV9VVXD22S4h7NrlPsvNhaOOgq98xSWCY4+Fgw/usV/FGNOGJYg0S2o8iCOP5MKTTuJbv/sdZGXxwfr1KRkPoqtmzZrFJz/5SS666KKEnzc17T1CV6wBOda76EUXtSSDww/vE3fSGtNv2X/fDGgeD+K002D9epasXMnkU0/l7GuvBZGUjQfRHdFo62RQV+dG0YzJyXF30MZG3tq40TU6G2P6jv6TIHrQgBBbtmyhNBCA1ashGGTyeed12r11V8eDWLlyJV/84hd59913AdiwYQPnnXceS5Ys4c477+Tvf/879fX1nHjiidx///2ICJGI63pi+XLXZhB7viAQcEmguLhlGMa2JYMe3k5ujNkHSf23FpGnReQTImKXge4Khbhx5kxOnzGDj998M7966SWq2hsas43YeBBvvvlmp+NBHHbYYTQ2NrJ+/XoAZs+e3dzV97XXXsv8+fNZunQptbX1/OUv/2DFCtd2UFXlLvbDhrn2giOOcH3zHXJIy41TVm1kTP+Q7H/13wGXAb8WkaeAP6jqqvSFlQaZHhAi1jPchx9y2bnncvZnPsOLb7/txoN48MG0jAcxY8YMZs+eza233srs2bOZPXs2AK+8Mo+f/ORn1NTUsWvXToqKDueoo84jP989hXzYYSn71saYXiypEoGqvuyNE30UsAF4WUTeFJHLRKTn3nPZUzQ1tfQOl5cHhx/OiIkTufzyy3nuuefIyspi6dKlne4mfjyIM844o9P1Z86cyZNPPsnq1asREYYPH8eqVSGuvvprfO97c3jyySVccsmVDBoUYuJEd9eRlQ6MMTFJVxmJSDEwC7gCeA+4B5cw5nawzTkiskpEPhCRW9tZZ4aILBeRZSLyV2/ZFBF5y1v2vojMTLRtr7F8Oezc6W5bHT+eF//zH8Le7T/7Mh7ET3/606TGgzj44IMR8XPbbd/nYx+byapVsHVrCBGYNm0IBx1Uw9y5c5p78TbGmHhJ/V4UkWeA8cCfgfNUdYv30WwRKWtnGz9wL3AWUA7MF5HnVXV53DrjgNuAk1R1l4iUeB/VAV9U1TUiMgJYICIvqWpV179ihqjCli1uPIizzmoeYGd/jAfR1OTyUWUlnHzyTH7961t45ZX1jBkDRUWFXH31lZx00iSGDx/OMccc051vaYzpw5IaD0JETlPVeV3ascgJwB2qerb3/jYAVf1x3Do/A1ar6kOd7GsxcJGqrmlvnR41HkRjI6xb556KHjzYdSCU5rqbaNQ9i1BZ6V5VXZVRcbELITs7rYfvseNBGGM6lorxICaKyHuxX/AiUgR8TlV/18E2I4FNce/LgeParHOot783AD8uobzYJvhjgWxgbZKxZtauXa6tQdX1LldcnLZDxQaH27nTTZGIuyW1pMQdtgc/gG2M6QWSTRBXquq9sTdeddCVuLubunv8ccCpQCnwmohMjktEB+Cqtb6kqtG2G4vIVcBVAKNHj+5mKN0UicCmTW6EmtioNF4VUqqFQq6ksHMnNDTAz352DUuXvoHf39LV9fXXX89ll12WluMbY/qHZBOEX0REvfoor32hs0qLzcCouPel3rJ45cA7qhoG1ovIalzCmC8iA4F/Av+rqm8nOoCqPgA8AK6KqZ11kHT/jK6rc1VKoRAMH+4GJUjxk2PhsCucVFa6J5uhpZfvxx+/N6NjIPSVYWuNMa0lmyBexDVI3++9/4q3rCPzgXEiMhaXGD4LfL7NOs8CnwP+ICJDcFVO60QkG3gG+JOqzkkyxr0Eg0EqKyspLi5OT5JQdR0QlZe7NoZDD03pmJXttSuUlu6fdoVkqCqVlZXNDe7GmL4j2QTxLVxS+Kr3fi7QYcOyqjaJyLXAS7j2hUdUdZmI3AmUqerz3mfTRWQ5EAFuUdVKEfkC8DGgWERmebucpaqLkv9qUFpaSnl5ORUVFV3ZLDmRiKtOCoVaWoM3b3ZTN4VCLf0fRaOu2ijW55HP50oSsR5Te4JgMEhpaWmmwzDGpFhSdzH1BonuYkqbF16Ayy5zQ37+6ldw1VXdbg0OheCnP4U//tE9U5eX58ZUvvRSOOMMG0bTGJMe3b6LyXte4cfARKC5LkFVD0pJhL1FKORGurnnHtdJ0bx5MHFiSnb9ne/AXXfBmWfC974Hn/mMGxLCGGMyJdkqpj8A3wV+BZyG65epf3Xct3w5fP7zsHgxXH89/OQnKbtLacEC+MUv4Ior4MEHU7JLY4zptmQv8rmq+gquSmqjqt4BfCJ9YfUgqnD//TBtGnz0Efzzn67jvxQlh3AYvvxl13vqXXelZJfGGJMSyZYgGryuvtd4Dc+bgb5fAVJZCVdeCc88A9OnuwaCJLvESNZdd7lCyTPPQGFhSndtjDHdkmwJ4nogD7gOOBr4AvCldAXVI8yb5wZC+Mc/XP3Pv/6V8uSwahXceacbpvPTn07pro0xpts6TRDeQ3EzVbVGVctV9TJVvbC9h9d6vXAY/vd/3a1DBQXwzjvwjW+k/MG3aNS1OeTlwW9+k9JdG2NMSnRaxaSqERE5eX8Ek3Fr17qG6HffdVfvu+92Dx+kwf33w+uvwyOPpLxgYowxKZFsG8R7IvI88BRQG1uoqk+nJapM+Mtf4Gtfcw8cPPWUq/dJk02b4Fvfcre0zpqVtsMYY0y3JJsggkAlcHrcMgV6f4LYs8clhsceg499zCWKUaM6324fqcJXv+oexH7gAett1RjTcyWVIFS173YLWlMDr7wC3/8+3HZb2h9ZfuIJd6fsL3/pOnw1xpieKtknqf+AKzG0oqqXpzyi/W3ECFizZr88trxjB1x3HRx7rHs1xpieLNkqpn/EzQeBzwAfpT6cDNlPfVrceCNUVcFDD1nfSsaYni/ZKqa/xb8XkceB19MSUR/1r3+55o3vfAcmT850NMYY07l9vbl/HFCSykD6supq+MpXYMIE+Pa3Mx2NMcYkJ9k2iGpat0FsxY0RYZLw7W+7MYXeeANycjIdjTHGJCfZKqYB6Q6kr3rzTbj3Xrj2WjjhhExHY4wxyUuqiklEPiMig+LeF4rIp9MWVR/R0OAeyB41Cn70o0xHY4wxXZNsG8R3VXV37I2qVuHGhzAd+OEPYcUK162GDf5jjOltkk0QidZL9hbZfun99+HHP3ZDhp5zTqajMcaYrks2QZSJyC9F5GBv+iWwoLONROQcEVklIh+IyK3trDNDRJaLyDIR+Wvc8i+JyBpv6lVdi0cirmqpqMgNWW2MMb1RsqWArwP/B8zG3c00F7imow28bsLvBc4CyoH5IvK8qi6PW2cccBtwkqruEpESb/lgXBXWNO94C7xtd3Xly2XKPffA/Pnw+ONQXJzpaIwxZt8kexdTLZCwBNCBY4EPVHUdgIg8AZwPLI9b50rg3tiFX1W3e8vPBuaq6k5v27nAOcDjXYxhv1u3Dm6/HT75SZg5M9PRGGPMvkv2Lqa5IlIY975IRF7qZLORwKa49+XesniHAoeKyBsi8raInNOFbRGRq0SkTETKKioqkvkqaaUKV10FWVnw+99bT63GmN4t2SqmId6dSwDEVwel4PjjgFOBUuA1EUm6IwpVfQB4AGDatGl7dSa4v/3hD65j2N//HkpLMx2NMcZ0T7KN1FERGR17IyJjSNC7axubgfiBFUq9ZfHKgedVNayq64HVuISRzLY9ypYtcNNNbkiJq67KdDTGGNN9ySaI/wVeF5E/i8hfgP/iGpc7Mh8YJyJjRSQb+CzwfJt1nsWVHhCRIbgqp3XAS8B0ryqrCJjuLeuxvv51qK+HBx9M+fDVxhiTEck2Ur8oItOAq4D3cBf2+k62aRKRa3EXdj/wiKouE5E7gTJVfZ6WRLAciAC3qGolgIh8H5dkAO6MNVj3RM88A3/7m3ta+tBDMx2NMcakhqh2XnUvIlcA1+OqehYBxwNvqerpHW23P02bNk3Lysr2+3F37YKJE2H4cHj3XQgE9nsIxhizz0RkgapOS/RZspUh1wPHABtV9TRgKlCVmvB6t1tugYoKePhhSw7GmL4l2QQRUtUQgIjkqOpKYHz6wuod/vMflxhuugmOOirT0RhjTGole5trufccxLPAXBHZBWxMV1C9QV0dXHklHHII3HFHpqMxxpjUS7aR+jPe7B0iMg8YBLyYtqh6ge9+1z01PW8e5OZmOhpjjEm9LvfIqqr/TUcgvUlZGfzyl+55h1NPzXQ0xhiTHnbHfheFw/DlL7u7ln72s0xHY4wx6WNjOnTRz37mxnp49lkYNKjT1Y0xpteyEkQXrFwJd94JF18M55+f6WiMMSa9LEEkKRp1dy3l58NvfpPpaIwxJv2siilJ990Hr78Ojz4Kw4ZlOhpjjEk/K0Ek4cMP4VvfgunT4YtfzHQ0xhizf1iC6IQqfPWrrorp/vttECBjTP9hVUydePxxeOEF+NWvYMyYTEdjjDH7j5UgOlBRAddfD8cd58Z7MMaY/sQSRAduuAF273Yd8vn9mY7GGGP2L0sQ7fjnP+Gvf4VvfxsOPzzT0RhjzP5nCSKB6mrXMD1xItzW2cCqxhjTR1kjdQK33Qbl5fDGG5CTk+lojDEmM6wE0cbrr8PvfgfXXQcnnJDpaIwxJnPSmiBE5BwRWSUiH4jIrQk+nyUiFSKyyJuuiPvsZyKyTERWiMivRdL/BEIoBFdcAaNHww9+kO6jGWNMz5a2KiYR8QP3AmcB5cB8EXleVZe3WXW2ql7bZtsTgZOAI7xFrwP/A7yarnjBJYVVq+Cll6CgIJ1HMsaYni+dJYhjgQ9UdZ2qNgJPAMn2gapAEMgGcoAAsC0tUXrefx9++lPXlcb06ek8kjHG9A7pTBAjgU1x78u9ZW1dKCLvi8gcERkFoKpvAfOALd70kqquaLuhiFwlImUiUlZRUbHPgTY1uUGAiorcSHHGGGMy30j9d2CMqh4BzAX+CCAihwATgFJcUjldRE5pu7GqPqCq01R12tChQ/c5iHvuccOI/va3UFy8z7sxxpg+JZ0JYjMwKu59qbesmapWqmqD9/Yh4Ghv/jPA26pao6o1wL+AtNxTtH49/N//wac+5QYCMsYY46QzQcwHxonIWBHJBj4LPB+/gogcEPf2U0CsGulD4H9EJEtEArgG6r2qmFJh5Ei4/XZ3a6v11GqMMS3SdheTqjaJyLXAS4AfeERVl4nInUCZqj4PXCcinwKagJ3ALG/zOcDpwBJcg/WLqvr3dMSZne260zDGGNOaqGqmY0iJadOmaVlZWabDMMaYXkVEFqjqtESfZbqR2hhjTA/VZ0oQIlIBbOzGLoYAO1IUTm9n56I1Ox+t2flo0RfOxYGqmvA20D6TILpLRMraK2b1N3YuWrPz0ZqdjxZ9/VxYFZMxxpiELEEYY4xJyBJEiwcyHUAPYueiNTsfrdn5aNGnz4W1QRiTAiLyKFCuqrcnse4G4ApVfbk7+zEm3awEYYwxJiFLEMYYYxLq9wmis1Hv+hMRGSUi80RkuTea3/WZjimVRGSDiNzidS9fKyIPi8gwEfmXiFSLyMsiUhS3/qe889AkIpUiMiHus6kistDbbjZu/JL4Y33SGyWxSkTeFJEj2AcicqX3t7lTRJ4XkRHechGRX4nIdhHZIyJLRGSS99m53r9htYhsFpGb9+mE7R1Lodct/0pvpMd+PSiviNzo/X0sFZHHRSTY+Va9jKr22wnXR9Ra4CDc4ESLgYmZjiuD5+MA4ChvfgCwui+dD2AD8DYwDNeN/HZgITAVd4H/D/Bdb91DgVrgPuBxXGeRH3h/J9m4hzJvxA1mdREQBn7gbTvV2/dx3t/Yl7xj58TFcWY7MT4at5/TcQ9hHYUbOOs3wGveZ2cDC4BCQHDd4x/gfbYFOMWbL4r9m6bg/P0R13aCdw4KM/1vmsG/pZHAeiDXe/8kMCvTcaV66u8liO6MetfnqOoWVV3ozVfjLoqJBnnqzX6jqttUdTPw/4B3VPU9VQ0Bz+Au7gAzcQljHPAg7odELnAicDwuMdytqmFVnYPrvTjmKuB+VX1HVSOq+kegwduuKy7BdXK5UF23+LcBJ4jIGFxCGgAchrvZZIWqbvG2CwMTRWSgqu6K/Zt2h4gMAj4GPAygqo2qWtXd/fZyWUCuiGQBecBHGY4n5fp7gkh21Lt+x7sITQXeyXAoqRY/dG19gvex0chH4EoR3wSi3rJNuL+PEcBm9X46euK7eTkQuMmrXqoSkSrc2CgjuhjriPj9qhsbpRIYqar/AX6LG/d9u4g8ICIDvVUvBM4FNorIf1NUFTQWqAD+ICLvichDIpKfgv32St4PjJ/jhibYAuxW1X9nNqrU6+8JwiQgIgXA34AbVHVPpuPJkALAr6oL4paNwg16tQUYKdJqBJHRcfObgB+qamHclKeqj3cxho9wyQYA74Jc7MWAqv5aVY8GJuKS2S3e8vmqej5QAjyLq/7orixcVdfvVXUqrvqt37bZeW1V5+MS5wggX0S+kNmoUq+/J4hOR73rb7wBmv4GPKaqT2c6ngyqAw4Ska24qsfpwEDgTeAt3Bgm14lIQEQuwFVXxjwIXC0ix3mNyfki8gkRGdDFGB4HLhORKSKSA/wIVyW2QUSO8fYfwF2sQ0BURLJF5BIRGaSqYWAPLSWg7ijHPZ8RK1HOwSWM/upMYL2qVnjn+Wlc9WOf0t8TRKej3vUn3i/ih4EVqvrLTMeTSar6FVxVzU5caaIaON6re28ELsANcLUT117xdNy2ZcCVuCqgXbjG7Vn7EMPLwP/hEvYW4GDc3yi4ZPWgt/+NuKqnu7zPLgU2iMge4GpcW0a3qOpWYJOIjPcWnQEs7+5+e7EPgeNFJM/7f3MGaRr1MpP6/ZPUInIucDcto979MLMRZY6InIxruF1Cy6/Ob6vqC5mLKvNE5FTgZlX9ZIZDySgRmYIbOz4bWAdcpqq7MhpUBonI93A/DpqA93B3eDVkNqrU6vcJwhhjTGL9vYrJGGNMOyxBGGOMScgShDHGmISyMh1AqgwZMkTHjBmT6TCMMaZXWbBgwQ5tZ0zqPpMgxowZQ1lZWabDMMaYXkVENrb3mVUxGWOMScgSBFDxtwqaapoyHYYxxvQo/T5B1K6sZdmMZay+ejX2TIgxxrToM20QiYTDYcrLywmFQh2uV/xGMQ1VDSybvwz/AP9+ii61gsEgpaWlBAKBTIdijOkj+nSCKC8vZ8CAAYwZM4bWHW+2pqrUr6knUh0h78A8/Hm9K0moKpWVlZSXlzN27NhMh2OM6SP6dBVTKBSiuLi4w+QAICIExwaRLKF+bT0a6V1VTSJCcXFxpyUlY4zpij6dIIBOk0OML+AjeFAQbVBCG0K9rj0i2e9pjDHJ6vMJoiuyBmSRXZpN064mwhXhTIdjjDEZZQmijexh2fgH+WnY1ECkNtLt/VVVVfG73/2uy9ude+65VFVVdfv4xhizryxBtNHcHhFw7RHRpu4NxtVegmhq6vi5ixdeeIHCwsJuHdsYY7qjT9/FFG/NDWuoWVST/AYRiNRFkCzBl5s4jxZMKWDc3eM63M2tt97K2rVrmTJlCoFAgGAwSFFREStXrmT16tV8+tOfZtOmTYRCIa6//nquuuoqoKXrkJqaGj7+8Y9z8skn8+abbzJy5Eiee+45cnNzk/8uxhizD6wE0R4/+HJ8aJOijfveYP2Tn/yEgw8+mEWLFnHXXXexcOFC7rnnHlavXg3AI488woIFCygrK+PXv/41lZWVe+1jzZo1XHPNNSxbtozCwkL+9re/7XM8xhiTrH5Tgujsl34iqkpobYim3U3kjs8lq6D7p+vYY49t9azCr3/9a5555hkANm3axJo1ayguLm61zdixY5kyZQoARx99NBs2bOh2HMYY0xkrQXRARMgZk4NkC6F1oW63RwDk5+c3z7/66qu8/PLLvPXWWyxevJipU6cmfJYhJyened7v93fafmGMMalgCaITviwfuQflomEltL7rz0cMGDCA6urqhJ/t3r2boqIi8vLyWLlyJW+//XYqQjbGmJToN1VM3eHP95MzKoeGDxto3NpIzgE5nW/kKS4u5qSTTmLSpEnk5uYybNiw5s/OOecc7rvvPiZMmMD48eM5/vjj0xG+McbsE+ltTwy3Z9q0adp2wKAVK1YwYcKElOxfVQmtC9G0y2uPGNDzcmsqv68xpn8QkQWqOi3RZ1bFlCQRITgmiOR47RHh7rdHGGNMT2YJogvEL+QenIs27Vt7hDHG9CaWILrIn+cnZ3QOkT0RGrc0ZjocY4xJG0sQ+yAwJEBWcRaNHzXStMduOTXG9E2WIPaBiBAcHcQX9Ln2iEZrjzDG9D2WIPaR+IXgwUE0au0Rxpi+yRJEN/hz/QRHB4lUR2j8KDXtEQUFBSnZjzHGdFdaE4SInCMiq0TkAxG5tZ11ZojIchFZJiJ/jVs+WkT+LSIrvM/HpDPWfRUYEiBrSBaNWxpp2m3tEcaYviNtT3uJiB+4FzgLKAfmi8jzqro8bp1xwG3ASaq6S0RK4nbxJ+CHqjpXRAqAblX037BmDYtqutDddxKmFBRw97hxBEcHqauto359PfkT8/Flt+TdW2+9lVGjRnHNNdcAcMcdd5CVlcW8efPYtWsX4XCYH/zgB5x//vkpjc0YY7ornSWIY4EPVHWdqjYCTwBtr4JXAveq6i4AVd0OICITgSxVnestr1HVujTG2i3ic+0RRKF+XT0abWmPmDlzJk8++WTz+yeffJIvfelLPPPMMyxcuJB58+Zx0003WRuGMabHSWd/ESOBTXHvy4Hj2qxzKICIvAH4gTtU9UVveZWIPA2MBV4GblXVVmOAishVwFUAo0eP7jCYu8d1vbvvrvAH/QTHBAmtC9HwUQPB0iAAU6dOZfv27Xz00UdUVFRQVFTE8OHDufHGG3nttdfw+Xxs3ryZbdu2MXz48LTGaIwxXZHpDoWygHHAqUAp8JqITPaWnwJMBT4EZgOzgIfjN1bVB4AHwPXFtL+Cbk9gcIBIdYTw1jD+Aj+BwgAAF198MXPmzGHr1q3MnDmTxx57jIqKChYsWEAgEGDMmDEJu/k2xphMSmcV02ZgVNz7Um9ZvHLgeVUNq+p6YDUuYZQDi7zqqSbgWeCoNMaaMjmjcvDl+QitDxFtcM0mM2fO5IknnmDOnDlcfPHF7N69m5KSEgKBAPPmzWPjxo0ZjtoYY/aWzgQxHxgnImNFJBv4LPB8m3WexZUeEJEhuKqldd62hSIy1FvvdGA5vYD4hNyD3HjRsfaIww8/nOrqakaOHMkBBxzAJZdcQllZGZMnT+ZPf/oThx12WIajNsaYvaWtiklVm0TkWuAlXPvCI6q6TETuBMpU9Xnvs+kishyIALeoaiWAiNwMvCIiAiwAHkxXrKnmC/pce8TaEA3lDQRHB1myZEnz50OGDOGtt95KuG1Niu+0MsaYfZXWNghVfQF4oc2y78TNK/ANb2q77VzgiHTGl06BogCRkgjh7WH8A/wEigKZDskYY7rEnqROo5zSHHz5PkIbQkRD1l+TMaZ36fMJIpPPFyRqj0gXe47CGJNqfTpBBINBKisrO7x4qipbGxoIR9PzC9+X4yM4Nki0LkrDpoa0HENVqaysJBgMpmX/xpj+KdPPQaRVaWkp5eXlVFRUtLtOOBplS2MjfhFKAgECvvTkzHBjmMiKCIGKAP58f8r3HwwGKS0tTfl+jTH9V59OEIFAgLFjx3a6Xt3u3Xxq6VKaVHlu0iROKSxMeSzRcJRFpy1i1+JdHF12NHnj81J+DGOMSaU+XcWUrOMHDeLto46iJBDgzMWLeXzbtpQfwxfwMfGJifiCPpZdvIxIXaTzjYwxJoMsQXgOys3lzaOO4viBA/n8ihX8aOPGlDf8BkuDHPbnw6hdWsuar69J6b6NMSbVLEHEGRwI8O8jj+TzJSX87/r1XLlqVcobr4vPKWb0t0ez9ZGtbP3j1pTu2xhjUimpBCEi14vIQHEeFpGFIjI93cFlQo7Px18mTOD2Aw/k4a1b+cSSJexpSu1AQGPuGEPhqYWs/upqapbak9PGmJ4p2RLE5aq6B5gOFAGXAj9JW1QZJiJ8f+xYHh4/nnlVVZz83ntsSmFvq74sHxP+OgH/QD/LL15OU42NRGeM6XmSTRDivZ4L/FlVl8Ut67MuP+AA/jV5MhtDIY5buJD3qqtTtu+cA3KY+NeJ1K2qY/XVq+1BN2NMj5NsglggIv/GJYiXRGQA3RwCtLc4c/Bg3pg6lSwRTnnvPV6orEzZvotOL2LMHWPY/th2tjy0JWX7NcaYVEg2QXwZuBU4xhv6MwBclraoephJBQW8fdRRjM/L47wlS/j95rbDWuy7A//3QIrOKmLN19dQvSh1JRRjjOmuZBPECcAqVa0SkS8AtwO70xdWzzMiJ4f/TpnCxwcP5mtr1nDL2rVEU1AtJH5hwl8mECgOsPzi5dQstkZrY0zPkGyC+D1QJyJHAjcBa4E/pS2qHqogK4tnJ03iayNG8PNNm5i5fDn1ke4/8JZdks3E2RNp3NpI2ZQyFp+9mJ1zd1q7hDEmo5JNEE3e2A3nA79V1XuBAekLq+fK8vn47bhx/OLgg/lbRQWnL15MRWNjt/dbeHIhx288nrE/Gkvt+7W8P/19yqaWsfUvW4mG+0VzjzGmh0k2QVSLyG2421v/KSI+XDtEvyQifGPUKOYcfjiLamo4fuFCVtXVdXu/gcEBDrztQI7fcDzjHxmPhpWVl67knYPe4cOff0jTHrsd1hiz/ySbIGYCDbjnIbYCpcBdaYuql7hg6FDmHXkk1ZEIJyxcyP+rqkrJfn05Pg647ACOWXIMk/85mdxDcll3yzreGvUWa29ZS6g8dc9kGGNMeyTZem4RGQYc4719V1W3py2qfTBt2jQtKyvLyLHX1ddz7vvvsz4U4tHDDuNzw4al/Bh7yvZQ/otytj+1HRGh5HMljLppFAVHFqT8WMaY/kNEFqjqtESfJdvVxgzgXeBiYAbwjohclLoQe7f90dHfwGkDmfj4RI774DhGXDOCiqcrXIP2dGvQNsakR1IlCBFZDJwVKzWIyFDgZVU9Ms3xJS2TJYiYhmiUy1eu5K/bt/Pl4cP5/aGHpm8Aol1hPrr/Izbfs5nGrY3kH5HPqJtHUTKzBF+29cFojElOt0sQgK9NlVJlF7btNxJ19Lc7xR39xQSKAhx4a1yDdpOy8osrefugt12D9m5r0DbGdE+yF/kXReQlEZklIrOAfwIvpC+s3qttR3+npLijv7baNmjnHZrXukF7kzVoG2P2TVcaqS8ETvLe/j9VfSZtUe2DnlDF1NbLO3dy4bJl5Pv9/HPyZKYO2D+PjlQvqGbTzze1NGh/toRRN1uDtjFmbx1VMSWdIHq6npggAJbW1HDukiXsDId58vDDObe4eL8du35DPZvv2cxHD35EtDZK0VlFjLp5FEVnFSHS5zvjNcYkYZ/bIESkWkT2JJiqRWRPesLtW9LZ0V9ncsfkcsivDuGETScw9sdjqV1ay/tnv0/ZlDK2/nkr0UZ7QtsY0z4rQewnNU1NfHb5cv65cyc3jxrFTw86CN9+/hUfbYiy7a/b2PTzTdQtryN7ZDalN5Qy4soRZA3K2q+xGGN6hlTcxWS6KV0d/XVFc4P20mOY/ELrBu0Pbv7AGrSNMa1YgtiP0tXRX1eJCMUfL2bKf6ZwdNnRFH+ymPK7y3nnoHdYcekK9pTtQaN9o2RpjNl3VsWUIU9XVHDJihWMyM7mhSOOYHxeXkbjadugHRgSoPDUQgpPK6Tw1ELyJuRZw7YxfZDdxdRDvb17N59aupQmVZ6bNIlTCgszHRLhXWF2PLeDqnlVVM2romFTAwCBYS5hFJ1WROGpheQemmsJw5g+IGMJQkTOAe4B/MBDqvqTBOvMAO4AFFisqp+P+2wgsBx4VlWv7ehYvTFBQOuO/v5w2GF8Pg0d/e0rVSW0PkTVvCp2zdtF1bwqGj9yVWLZB2Q3ly4KTysk92BLGMb0RhlJECLiB1YDZwHlwHzgc6q6PG6dccCTwOmquktESuK79BCRe4ChwM6+miAAdobDfGbpUl7bvZsj8/OZUVLCxUOHMi7D1U5tqSr1H9Q3ly6qXq2icatLGDmlOS1VUqcVEhwTtIRhTC+QqQRxAnCHqp7tvb8NQFV/HLfOz4DVqvpQgu2PBm4BXgSm9eUEAa6jv/s/+ojZ27fz5h73iMnUggJmDB3KxSUlHJybm+EI96aq1K2qa04WVfOqCFeEAcgZnUPhaS1VUsEDgxmO1hiTSKYSxEXAOap6hff+UuC4+Au9iDyLK2WchKuGukNVX/RGrPsP8AXgTNpJECJyFXAVwOjRo4/euHFjWr7L/rYpFGJORQVPVlTwtpcsjiooaC5ZHNQDkwV4CWN5HVWvelVSr1bRVOk6DQyODbaqkgqWWsIwpifoyQniH0AYN8ZEKfAaMBmXGPJU9Wde54B9vgTRno2xZLF9O+9WVwMwbcAAV7IYOpQxPTRZAGhUqV1W21Il9d8qmna5hJF7SG6ru6RyRuRkOFpj+qeeXMV0H/COqv7Be/8KcCtwA3AKEAUKgGzgd6p6a3vH66sJIt6G+nqe8koWZV6yOHbAgOaSxehgz/5VrlGl5v2aloTxWhWR3e5hwdxDc1tVSWUPy85wtMb0D5lKEFm46qMzgM24RurPq+qyuHXOwTVcf0lEhgDvAVNUtTJunVn04xJEe9bFksX27SysqQHg+IEDmTF0KBcNHcqoHp4sADSi1Cyqaa6O2v3abiLVLmHkTchj0MmDyJ+cT/6kfPIn55M9xJKGMamWydtczwXuxrUvPKKqPxSRO4EyVX1e3G0uvwDOASLAD1X1iTb7mIUliA6tra/nqe3bebKigve8ZHFCXLIo7QXJAiDaFKVmYU1zg/eed/fQtLNl4KPs4dnNyaL5dWI+/nx/BqM2pnezB+X6kTV1dc0li8W1tQCcNHAgM0pKuGjoUEbk9J66flWlcWsjtUtq3bS0lpolNdQtqyMa8nqiFQgeFKRgckGrxJE7LhdflvUkY0xnLEH0U6vq6ppLFktqaxHg5EGDmDF0KBcOHcoBvShZxNOIUr+uvjlp1C5xiaN+Tb1rtQIkW8ibkOcSR6y0MTmfnNIcez7DmDiWIAwramubSxbL6uoQ4GODBjGjpIQLhgxheC9NFvEioQh1K+qak0YsgTSUNzSv4x/kdwljUn6rUkdgcCCDkRuTOZYgTCvLvWQxe/t2VtTV4QP+p7CQi4cO5YKhQxmW3bcag8O7wi5pLK1tVepoqopr3xiR3ZwsYqWOvIl5+HOtfcP0bZYgTLuW1dby5PbtzN6+nVX19fiAEwcNYlJ+PuNzczksL4/xeXmMDgbx96GqGVWlYXPDXqWN2uW1aIP3f8LnntfIG59HcEyQ4Fg35Y7NJTg2SNZAG2TJ9H6WIEynVJWltbU8WVHB3J07WVVfT1VTyy/sHBHG5eW5hJGby3gvcYzPy2NQVt+5UEabooTWhqhZUtOcOOo/qCe0PkSkpvUAT1lFWXsljeZEMiZopQ/TK1iCMF2mqmwPh1lVV9cy1dezsq6O9fX1xF8qh2dnt0oasVLHmD5U6lBVwpVhQhtChNa3TPXr692yDaGWkocne3h266QRl0hyRuXgC9hdVibzLEGYlGqMRllbX98qacSSyM64Uke2CId4iSOWNGKJpCjQtxqFNepuyW2VNOISSWhTiFZZ1ed6wG1OGnFJJDg2SM6IHMTXN5Kr6dksQZj9Zkdj415JY1VdHWtDIZri/tZKAoGWaqq4JDI2GCTL1/d+WUebojSUNyQufawPNY+zESPZQnB066QRHB0kpzTHTSNz8OX0vfNk9j9LECbjwtEo60KhVkljpVcC2REON68XEOHg3FzG5+ZyUG4uQwMBN2Vnt8wHAgzKyupTzzNEQhEaNjYkLH3Ur69v7hU3XqAk0JIwRuW0zJfmEBwVJHtkNv6gtYOYjnWUIPpO66Lp0QI+X3OJoa2dcW0dsaSxqq6Oubt2UReNJt6fCEMCAYbEJY22SST+/eBAoEe3h/iDfvLG55E3PvEgUU3VTTSUN7RMm1rmQ+tD7H5td6vbdmMCQwIJE0jz+5E5+PMsiZjErARherS6SISKcJiKxkZ2hMNuPjY1Nu71fnckknA/AhS3TSgdJJUhgQDZvayqq6mmicbNjS5pbAq1TiheUonv2yoma3BWwhJI/Hvr76rvshKE6bXy/H4O9Ps5MMkOBxuj0eZEsqOdJFIRDrOiro7XwmEqw2Ha+4k0yO9naHY2JYEAw7KzGZ6dzbDsbIYFAs3zsdc8f+YvoFkFWWSNz2q3FAIQqYvQsHnvUkhsqn6nmvCO8F7bZRVlNSeLQEmAwJAOpqIA4u+5pTWTPEsQpk/J9vkYkZOTdKeEEVV2xiWQRElle2Mjq+rqeK2qisqmvX+BAxT4/S5ZxCWP9pJKbgaTiT/PT964PPLGdZBEQpEOSyK1y2oJ7wgTrUtc/YcPAoM7SCBD917mH+DvU21KfYUlCNOv+UVctVKS3YuEo1G2h8Nsa2xka2Mj27xpa2Mj27zly+vqmFdV1eqW33gD/f7WCaSDpBLMQDLxB/3kHpxL7sEdj1YYqYsQrgwT3tFmqmj9vn5tPXve2UN4RxgNJy6vSUA6TSKtpuIAvlyfJZU0swRhTBcEfD5G5uQwMokSSmM0ynYvceyVTLzlS2treaWxkV3tJJNBXjIZHtc2MsRrKxkS9z7WtrI/q7r8eX78eX6Co5Kr/lNVItWRvRJI26mxopGaxTWEd4Rdm0k7dYCSLWQVZpFVlOVek5wCRQGyCrPsNuEkWIIwJk2yfT5Kg8GkBmxqiCWTNqWRrXHLVtTVscOrBmuncodcn2+vpJEokcTmiwMBAvupMV5EyBqYRdbArISlE1WlPhqlLhKhNvYajrB7dwPVVWH27G6kujpMdW2YhtomBu2Bwp1QuF0ZtC1K/pYwofUhmqqaaNrV1G5pJcYX9HWeUDpIPr7svp9gLEEY0wPk+HyMCgaTGio2qkpVU1NzsohvlG+1rLGRdd5zJu3d3QUtjfEdJZLYNDgri7AqtZEIddFo69d2ljVf7L3liZbFXjuV700J+IChgQAl2dkMC+Qz1B9gaDSLIWE/g0M+imuFomph8C4YVAn+nRGXTLwpXOmqw5oTTFMnCSbXR1ZRFoHBAbKKswgUBzqfHxzoVYnFEoQxvYxPhMHesx2HJrlNYzTa3BjfUWLZ3NDAopoaKhobaUjBLfB5Ph95fj/5sVe/nzyfz1WH5eS49wk+b/Wa4PMsEXZ4paztbV5j82tD1WxvbKQ2lnhyvGkIMK6l+q7Ea+8pyc5tbvsZGggwVLMorvcxuMZH7u4okarWCaWpqonwTlcNFq4MU7eqjqZKN99R6cVf4G9JHMUBsgYnMV+Y1WHXK1FVfGloj7EEYUw/kO3zMTwnJ+mBoVSVutgtw94zKDvCYXY2NZEt0uHFPDYf9PnSctGKGZqdzYT8dooTcWojkebqu7ZJJPa6vK6OVzu4Sy3o81EyOMCw4S23PZdkZzPIn0sEaFJtnsLRKOHGKI2hCI31ERobIoQbojQ2RAmHI4TDSjgcJdzUQFOknnDEbdOkEPFDpB4iWyCyHaI+t6wpCyIBiGZBJMtbzw8RHzQJHBXOpWz6cSk+w5YgjDEJiAj5XhJI9hmUnirf72dsbi5jczu+KwvcXWqxW5u3xV7bJJOPGht5r6aG7eFwq/7FfECWyF5TIChk5bZdntU8nyNCgc9HFuCPgC8M/kbF16j4GhRfA0hDFF+V4qtXfHVRpC6K1HpTvXLgAIXpqT93liCMMcYT6MJzNKpKKBolSwS/SFpLSx2JNkaJ1LXfxtQdliCMMWYfiEhGH3qM8WX70tbw3Xua040xxuxXliCMMcYk1Gd6cxWRCmBjN3YxBNiRonB6OzsXrdn5aM3OR4u+cC4OVNWhiT7oMwmiu0SkrL0ub/sbOxet2flozc5Hi75+LqyKyRhjTEKWIIwxxiRkCaLFA5kOoAexc9GanY/W7Hy06NPnwtogjDHGJGQlCGOMMQlZgjDGGJNQv08QInKOiKwSkQ9E5NZMx5NJIjJKROaJyHIRWSYi12c6pkwTEb+IvCci/8h0LJkmIoUiMkdEVorIChE5IdMxZZKI3Oj9P1kqIo+LSO/u1TCBfp0gRMQP3At8HJgIfE5EJmY2qoxqAm5S1YnA8cA1/fx8AFwPrMh0ED3EPcCLqnoYcCT9+LyIyEjgOmCaqk4C/MBnMxtV6vXrBAEcC3ygqutUtRF4Ajg/wzFljKpuUdWF3nw17gIwMrNRZY6IlAKfAB7KdCyZJiKDgI8BDwOoaqOqVmU0qMzLAnJFJAvIAz7KcDwp198TxEhgU9z7cvrxBTGeiIwBpgLvZDiUTLob+Ca0OwR0fzIWqAD+4FW5PSQinY/W00ep6mbg58CHwBZgt6r+O7NRpV5/TxAmAREpAP4G3KCqezIdTyaIyCeB7aq6INOx9BBZwFHA71V1KlAL9Ns2OxEpwtU2jAVGAPki8oXMRpV6/T1BbAZGxb0v9Zb1WyISwCWHx1T16UzHk0EnAZ8SkQ24qsfTReQvmQ0po8qBclWNlSjn4BJGf3UmsF5VK1Q1DDwNnJjhmFKuvyeI+cA4ERkrItm4RqbnMxxTxoiI4OqYV6jqLzMdTyap6m2qWqqqY3B/F/9R1T73CzFZqroV2CQi471FZwDLMxhSpn0IHC8ied7/mzPog432/XpEOVVtEpFrgZdwdyE8oqrLMhxWJp0EXAosEZFF3rJvq+oLmQvJ9CBfBx7zfkytAy7LcDwZo6rviMgcYCHu7r/36IPdblhXG8YYYxLq71VMxhhj2mEJwhhjTEKWIIwxxiRkCcIYY0xCliCMMcYkZAnCmB5ARE61HmNNT2MJwhhjTEKWIIzpAhH5goi8KyKLROR+b7yIGhH5lTc2wCsiMtRbd4qIvC0i74vIM17/PYjIISLysogsFpGFInKwt/uCuPEWHvOe0DUmYyxBGJMkEZkAzAROUtUpQAS4BMgHylT1cOC/wHe9Tf4EfEtVjwCWxC1/DLhXVY/E9d+zxVs+FbgBNzbJQbgn243JmH7d1YYxXXQGcDQw3/txnwtsx3UHPttb5y/A0974CYWq+l9v+R+Bp0RkADBSVZ8BUNUQgLe/d1W13Hu/CBgDvJ72b2VMOyxBGJM8Af6oqre1Wijyf23W29f+axri5iPY/0+TYVbFZEzyXgEuEpESABEZLCIH4v4fXeSt83ngdVXdDewSkVO85ZcC//VG6isXkU97+8gRkbz9+SWMSZb9QjEmSaq6XERuB/4tIj4gDFyDGzznWO+z7bh2CoAvAfd5CSC+99NLgftF5E5vHxfvx69hTNKsN1djuklEalS1INNxGJNqVsVkjDEmIStBGGOMSchKEMYYYxKyBGGMMSYhSxDGGGMSsgRhjDEmIUsQxhhjEvr/8ghbqGu0zq8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "s, (at, al) = plt.subplots(2,1)\n",
    "at.plot(history.history['accuracy'], c= 'b')\n",
    "at.plot(history.history['val_accuracy'], c='r')\n",
    "at.set_title('model accuracy')\n",
    "at.set_ylabel('accuracy')\n",
    "at.set_xlabel('epoch')\n",
    "at.legend(['LSTM_train', 'LSTM_val'], loc='upper left')\n",
    "\n",
    "al.plot(history.history['loss'], c='m')\n",
    "al.plot(history.history['val_loss'], c='c')\n",
    "al.set_title('model loss')\n",
    "al.set_ylabel('loss')\n",
    "al.set_xlabel('epoch')\n",
    "al.legend(['train', 'val'], loc = 'upper left')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model will output a prediction score between 0 and 1. We can classify two classes by defining a threshold value for it. In our case, I have set 0.5 as THRESHOLD value, if the score above it. Then it will be classified as **POSITIVE** sentiment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "25/25 [==============================] - 2s 42ms/step\n"
     ]
    }
   ],
   "source": [
    "def decode_sentiment(score):\n",
    "    return \"Positive\" if score>0.5 else \"Negative\"\n",
    "\n",
    "scores = model.predict(x_test, verbose=1, batch_size=10000)\n",
    "y_pred_1d = [decode_sentiment(score) for score in scores]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Confusion Matrix\n",
    "Confusion Matrix provide a nice overlook at the model's performance in classification task"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_confusion_matrix(cm, classes,\n",
    "                          title='Confusion matrix',\n",
    "                          cmap=plt.cm.Blues):\n",
    "    \"\"\"\n",
    "    This function prints and plots the confusion matrix.\n",
    "    Normalization can be applied by setting `normalize=True`.\n",
    "    \"\"\"\n",
    "\n",
    "    cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n",
    "\n",
    "    plt.imshow(cm, interpolation='nearest', cmap=cmap)\n",
    "    plt.title(title, fontsize=20)\n",
    "    plt.colorbar()\n",
    "    tick_marks = np.arange(len(classes))\n",
    "    plt.xticks(tick_marks, classes, fontsize=13)\n",
    "    plt.yticks(tick_marks, classes, fontsize=13)\n",
    "\n",
    "    fmt = '.2f'\n",
    "    thresh = cm.max() / 2.\n",
    "    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\n",
    "        plt.text(j, i, format(cm[i, j], fmt),\n",
    "                 horizontalalignment=\"center\",\n",
    "                 color=\"white\" if cm[i, j] > thresh else \"black\")\n",
    "\n",
    "    plt.ylabel('True label', fontsize=17)\n",
    "    plt.xlabel('Predicted label', fontsize=17)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAbAAAAFhCAYAAAARLP+gAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAA130lEQVR4nO3deZwcVb338c93JhtLWCSAQBIIkLgAjywRUa4YURBFwPugCIoSEFGvKC5wL5uAgCLCg1yvqHCRgAtEFIQAwQBCQFaTsIjsIQskbNkhISSZmd/zx6kmnU7P0j3dmV6+77zqNVOnTlWd6un0r89SpxQRmJmZ1ZuWvi6AmZlZORzAzMysLjmAmZlZXXIAMzOzuuQAZmZmdckBzMzM6lK/vi6AmZmtG60bbRvRtrzs/WP5vEkRcUAFi9QrDmBmZk0i2pYz8F2Hlb3/W49eMqSCxek1BzAzs6YhUOP0HDmAmZk1CwFSX5eiYhzAzMyaSQPVwBrnSszMrHtS+Uu3h9YBkp6RNF3SyZ3kOUzSk5KekHR1Xnq7pEezZUJPLsU1MDMz6zVJrcAlwH7AHGCKpAkR8WRenpHAKcDeEbFI0hZ5h1geEbuWck4HMDOzplHVQRx7AtMjYgaApPHAIcCTeXm+ClwSEYsAIuK13pzQTYjW8CR9O2uyWC4pJH1nHZxzlqRZ1T5PM5E0WZKf/9Rb1WtC3AZ4MW99TpaWbxQwStJ9kh6UlH9P2SBJU7P0z/TkUlwDs4qR9G7gm8BHgWHAesB84BHgeuD3EbFiHZfpcOC/szJcDKwAHlyXZbAkCz53R8SYvi5L0xK9rYENkTQ1b/2yiLishP37ASOBMcBQ4B5Ju0TEYmDbiJgraXvgTkmPR8Tz3R3MrNcknQGcSarVPwBcBSwFtiS9WS8HvgGMXsdF+3TuZ0S8tA7P+7F1eK5m8WVg/b4uRH3r2WCMLsyPiM7+D88lfXHNGZql5ZsDPBQRq4CZkp4lBbQpETEXICJmSJoM7AY4gFl1SToV+CGp+eBzEfFQkTyfBr6/rssGbA2wjoMX3X1ztNJFxAt9XQbr0hRgpKQRpMB1OPCFgjw3AEcA4yQNITUpzpC0KfBmRKzI0vcGftrdCd0HZr0iaTvgLGAV8KliwQsgIm4G1ppDLRtSe4+kJVkf1eOSTpE0sEjeWdmygaQLJL0gaUU2ZPe/pNVfLSWdlTVZfTRbj9ySK3e2fmUn17VWf4uSoyTdL2mepLckvShpkqTPFytrkeMOlHRydp1vSnpd0t8lrTW/T34Zs9/HS5qfnXdq9qWgx7JjTZa0paQrJL0qaVl2PR/O8uRe29nZa/uEpM8VOdbGkk6SdKekOZJWZq/JBEkfLMg7Nu+1/Ej+30LSWUWudZSkP0p6TVKHpDFZnjX+JpIGSJqS7XdwkTL+Ntv2g1Jep4anlvKXLkREG3A8MAl4Crg2Ip6QdHbe32cSsEDSk8BdwEkRsQB4DzBV0mNZ+k/yRy92xjUw662jgf7A+Ij4V1cZC/u/JP2YNKR2PnA1qcnxk8CPgU9I2j8iVhYcpj/pP8HWwK1AG/AZ4CfAIFJNEGBy9nMssG1eem/8KCvvTOBaYAmwFfB+4HPAH7vaWdKArOwfAZ4mDTleH/gs8EdJu0bEqUV23Rb4BzAD+B3wDuDzwI2SPh4Rd5VwDZsA9wFvANdkxzocmJQFnkuztJtJr/URWdlejIj8vsP3ZK/HPcAtwCJgOHAw8ElJB0XEX7O8j5Je/zOB2cCVeceZXFC+HYCHgGeBP5D6UV8vdiERsTL74vAI6Rv9rhHxIoCko4EvAX/Lymk5VZyJIyImAhML0s7I+z2A72VLfp77gV1KPZ/S8czKI+lvwL7AVyPi8hL2+yBwP6nZcc+IeCVL7wf8hdR3dVpE/Dhvn1mkD/NbgUMjYnmWvgXpAw9g86x9PbfPZOAjEbHG/9qs5jgTuCoixhYp31r7SVoALAdGRcSbBfmHRMT8grISEdvlpZ1CCs63Agdn31hz5f9Hdm17Z/+Z88sIcFZE/DDvWJ8A/grcGhGfKix/MXm1l0uB/4iIjiz9S8BvSUHoPlIz8FvZtg+TgtQNEfHvecfaGOiff81Z+tDsWpZExHuKnL/oII6Caz2vWCDv4m95GOnLw72k/tZRwFTSF6L35d5bBi0bbh0Dd/1K2fu/dd+507roA1vn3IRovbVV9nNOifsdk/08N/8DJvtQ/z7QARzbyb7fzgWvbJ/XgBuBjYF3lViOUq0C2gsTCz/IO3EMEMD3csEr2/c14Jxstdg1zwbOLTjfJOAF0r03pXiT1GzTkZd2NakmuylwQi54Zef5OzAL2LXg/EuKXXNEzAH+DLxb0vASywbwKiXWliPiWlJQ/jfgfFLteD3gSw5ejc0BzPrK7tnPOws3RMSzpIA4Ivumn29JREwvcrzc/SebVq6Ia/kDsB3wpKTzlKbNKSxfUZIGAzsCL0XE00Wy5F6H3YpsezQi1gqapGsu9XqfjYg38hOyY78KLM7dhFpgLmlE2Rok7S3p2qwfcIVW9zF+K8tSeA9QTzxW5q0W3wEeJ3352ZnUh3JbGcdpbLnJfKs0ldS65j4w662XSf0hpX5Y5T74X+7iuMNJfTZL8tIXd5I/V6NpLbEcpfguqR/qaODkbGmTNBH4fieBNacn1wvpegst7mSfNkr/Erqkk/S2brat8Vkh6d9JNa23gNtJw52XkWrOY0j9fGsNxOmBsmpMEfGWpFtI/ShtpP5FK8aT+Zq97d7sZ6n3PeU+LN/ZyfatCvJVWq4JrbMvcZsUJkREe0RcHBHvI93fdiipv+5g4K8qMnIyT19fb6WdA6wERkfEZyLi+xFxRkScBTzTi+OW1Skv6d+Ak0gDgvoBV0g1WGXoc6raKMS+UHslsnozjtQvdKik93aVseAD/pHs55gi+XYkNVnNzO7Qr4ZF2c9hhRskbUQaCNCpiHgtIq6PiMNIzX87kJquOsv/BqmWso3ShKaFPpr9fLgHZa8FOwJPRsRT+YmSWkh9UcV0UIUasqTNSCMqV5EGFP0B2B/4r0qfqyG0qPylxjiAWa9ExCzSfWADgFskFR2hpDTn2a15SVdkP0+XtHlevlbgQtJ78zdVKDLwdkB5Gtg7P/Bm57+INAiAvPSBkvYuPI6k/qRh55AGSHTlClIvxAXZeXLHGAL8IC9PPZhFuml161xCVuM5C+jsi8wCinxhqIBxpC88342Ix0kzvkwHzpH0oSqcz2qE+8Cs1yLix9nw9zNJj1C4n9XDmLcE9iFNFzM1b5/7Jf0U+E/gX5L+TOpD+SSpJnMvcEGVi34BKUjeJ+lPpP6cj5Luf3oMeF9e3vWAeyVNB6aRRgYOIj064j3AhMLaSBEXkq7vEOCxrO9sfdI9ZFsAP42Ie7vYv5b8DPg18Iik60i1n71Jwesm4KAi+/wNOFzSTaSa5irgnoi4p9xCKE3MfBBwXUT8GtKXk+z+sAeAa7L7wxZ1cZjm0fu5EGtK41yJ9amIOJsUeH5BGrBwNKlP4kBS09mxFDQtRcR/kW6UfY40z923Se/J04H9itzEXOkyX5GV6yXgKOAw0r1pe7P2wIllpCap6cCHgBNI0+S8TvrGv9ZsFUXOt5IU8E7Lkr6Vnfc54AvZ61EXIuJS0t/4ZdI1fJE0KvIDdN4MegKpqW9P0t/4HFKTX1kk7UEaNj+bgtsPIuJh0vtvOKmGZjkNNArRNzKbmTWJlo2GxsA9jy97/7f+dopvZDYzM+st94GZmTWTGmwKLJcDmJlZM2mgQRwOYGZmzaJGB2OUywHMzKyZuAZmlaB+64UGDO7rYliN2e095Uzibo3u4YenzY+IzbvP2TwcwPqQBgxm4LvWehCvNbn7HvpFXxfBatB6/TW7IgdyE6KZmdUfuQnRzMzqlGtgZmZWdzwXopmZWd9zDczMrGm4D8zMzOqV+8DMzKwuNVANrHGuxMzMmoprYGZmzcRNiGZmVnfkQRxmZlavXAMzM7N6pAYKYI1TlzQzs6biGpiZWZMQjVUDcwAzM2sWypYG4QBmZtY05BqYmZnVp0YKYB7EYWZmdck1MDOzJtJINTAHMDOzJuIAZmZm9afBRiG6D8zMzOqSa2BmZk1CHkZvZmb1ygHMzMzqkgOYmZnVpUYKYB7EYWZmFSHpAEnPSJou6eRO8hwm6UlJT0i6Oi/9KEnPZctRPTmfa2BmZs2iisPoJbUClwD7AXOAKZImRMSTeXlGAqcAe0fEIklbZOnvAM4ERgMBTMv2XdTVOV0DMzNrIpLKXrqxJzA9ImZExEpgPHBIQZ6vApfkAlNEvJalfwK4PSIWZttuBw7o7oQOYGZmTSI3jL4XAWyIpKl5y3F5h98GeDFvfU6Wlm8UMErSfZIelHRACfuuxU2IZmZNpJeDOOZHxOhe7N8PGAmMAYYC90japdyDuQZmZmaVMBcYlrc+NEvLNweYEBGrImIm8CwpoPVk37U4gJmZNRP1YunaFGCkpBGSBgCHAxMK8txAqn0haQipSXEGMAnYX9KmkjYF9s/SuuQmRDOzZqHq3QcWEW2SjicFnlbgioh4QtLZwNSImMDqQPUk0A6cFBELACSdQwqCAGdHxMLuzukAZmbWRKp5I3NETAQmFqSdkfd7AN/LlsJ9rwCuKOV8bkI0M7O65BqYmVkTaaSppBzAzMyahB+nYmZm9atx4pcDmJlZ06jiKMS+4EEcZmZWl1wDMzNrIo1UA3MAMzNrIg5gZmZWnxonfjmAmZk1k0aqgXkQh5mZ1SXXwMzMmkQPn6xcNxzAzMyaiAOYmZnVpUYKYO4DMzOzuuQamJlZM2mcCpgDmJlZM2mkJkQHMDOzZtFgk/k6gJmZNQkBDRS/PIjDzMzqk2tgZmZNwzcym5lZnWqg+OUAZtW134few4UnfZbWlhauvOF+Lhx3+1p5Dt1vN077+qeIgMefncvYU68EYNg7N+WXZ3yBoVtuShB85vhf8cLLC9fxFVg1tAj6ZR0Y7R3QHmtubxW05nVwrGqHgiwMbIW2Ivta11wDM+uBlhZx8cmHceA3fsHcVxdz7x9O4ua7H+fpGa+8nWeH4Ztz4jH7s+/Yi1j8xnI233TDt7ddfs6XOf/ySdz50NNssN4AOsKfVI2iX8vqoDSgFToKAlR7QHt7+j0X7FZ1rN7evwU6/HYonRqrBuZBHFY17995O55/cT6z5i5gVVs7f5r0MJ8e83/WyHPMv3+IS6+9h8VvLAdg3qKlALx7+3fSr7WFOx96GoBly1ey/K1V6/YCrCoERKwOWO0dKUj1VIvSvo5f1lQ1MEm/Btoi4vje5LGe2XqLjZnz6qK31+e+uog9d95ujTwjt90CgDvHfZfWlhbOvXQit9//FCOHb8HiN5Yz/sJj2XabzbjroWc4/ec30uGv3XVPWjP4BFkA66QZUcDK9tXp/VrSej9//S6ZSC0jjaJm3wKSJktaIWmppCWSHpF0aG+OGRFfzw9MkmZJOrKrPFZdra2t7Dh8C/b/6n/z5VOu5Jc/+AIbb7ge/fq1sPduO3Dyz/7Cvx15ASOGDuFLB+/V18W1dag9UqBa1bE6WPVrSf1eVj6p/KXW1GwAy5wTERsCmwHXAH+UNKqPy2Q99NJrSxi65aZvr2+z5abMnbdkjTxzX1vMzXc/TltbB7NfWsBzs19jx+GbM/fVxfzz2TnMmruA9vYOJtz1GLu+e9i6vgSrgog1p+PLNSl2piNWNzG2KPV/DWxNNbR+Lemn9VzumWDlLLWm1gMYABHRBvwSaAV2kfQNSc9kNbMHJX04l1fSbpLuzbYtlHS/pE2zbVdKujz7/SZgOHB5Vsu7rUieCyTdkF8WSWMkvSFpg2x9Z0mTJM2T9IKk8yT1r/6rUvumPjGbHYdvzrZbb0b/fq187hO7c8vkf66R56a7HmOf0SMB2GyTDRi57RbMnLuAqU/MZuPB6zEkG9Qx5v3vWmPwh9WvIPtGn623FhmQkf9RmevzglQjW5Et7eFRiM2uLvrAJA0AvgmsAt4LnAAcCEwDjgL+Kum9ETEbuAT4K/ARUoDeA1hZeMyIOEjSLOD0iPh9J6ceBzwqafOImJelHQ1cGxHLJG0B3A2cChwEbA7cCCwHzu71hde59vYOvnv+tdz0y2/S2iKuuvFBnprxCj/4xoE8/OQL3HL349x+/1N8/IPv4eHrTqO9PTj14htYuGQZAKdcdAMTf/0tJPHIUy9wxfX39fEVWaW0dUD/1vR7e0cKUP2yQNYRKajlal0RacSiVUCNNgWWS1GjQ5MlTQY+AKwgBaDpwHnAt4ApEXFaXt4HgAkRcV6233Tg3IiYVXDMK0kDNI7N1mdREMCK5HkIGB8RP5M0GHgF2D8i7pN0IvCpiNg3b/9DgfMjYsdOrus44DgA+m+4x6Cdjirn5bEGtmjKL/q6CFaD1uuvaRExujfHWH/rUbHjsb8se//Hz9mv12WopFpvQvxRRGwSEVtExIci4iZgGDCzIN/zWTqkGlILcK+kmZLOkdSbmuY4YGz2+2HAnIjIVQVGAHtLWpxbgCuAd3Z2sIi4LCJGR8Ro9VuvF8UyMytV+f1f7gOrjBeB7QrSts/SiYiZEXFMRAwFDgaOBb7cybF6Mp5pPDBK0u6kQDYub9ts4I4syOaWjbOBJ2ZmNcejEPvWlcDXJO0pqZ+ko4FdgasBJB0laess72KgDeisBf0VYGRXJ4uIxcBfgHOBvYCr8jb/Fhgt6RhJgyS1SNpe0gHlXJiZmfVc3QWwiLga+CHwe2AB8A1SP9TsLMu+wDRJy4AHSIHtd50c7lzgSEmLJN3axWnHAZ8EJkXEy3lleQX4KPAZYBawiBTsti/r4szMqqyRmhBrdhRiRIzpYtsvgKI93RHR6aiIiBhbsD4RmNhVniztdtYc2Zu/7UlSU6WZWW2r0abActVsADMzs8pKT2RunAjmAGZm1kQaKH7VXx+YmZkZuAZmZtZU3IRoZmZ1qYHilwOYmVnTUGPVwNwHZmZmFSHpgOxJIdMlnVxk+9jsyR2PZsuxedva89In9OR8roGZmTWJNIy+SseWWklPA9kPmANMkTQhu1c23x87eWjw8ojYtZRzOoCZmTWNqs6osScwPSJmAEgaDxwCFAawinEToplZE+nlZL5DJE3NW47LO/Q2ZJOqZ+ZkaYUOlfRPSX+WlP+Y9UHZMR+U9JmeXItrYGZmTaSXNbD5vXwe2E3ANRGxQtLXSJOj556nuG1EzJW0PXCnpMcj4vmuDuYamJmZVcJcVj+XEWBolva2iFgQESuy1cuBPfK2zc1+zgAmA7t1d0IHMDOzZtGL5sMeVNymACMljZA0ADgcWGM0oaSt8lYPBp7K0jeVNDD7fQiwNz3oO3MToplZk6jmZL4R0SbpeGAS0ApcERFPSDobmBoRE4BvSzqY9JzGhax+2v17gEsldZAqVj8pMnpxLQ5gZmZNpJo3MnfyiKoz8n4/BTilyH73A7uUej43IZqZWV1yDczMrIk00ExSDmBmZs2kkeZCdAAzM2sWPRtNWDccwMzMmoSqO5XUOudBHGZmVpdcAzMzayINVAFzADMzayYtDRTBOg1gkmYCUeLxIiJ26F2RzMysWhoofnVZA7ub0gOYmZnVqDSnYeNEsE4DWESMXYflMDMzK4n7wMzMmkhL41TAShtGL+kdks6RdJ+k5yR9MEvfTNIZkt5dnWKamVklSCp7qTU9roFJGg7cC2wJPANsD6wH6SFlko4AhgDfrkI5zcysAmowDpWtlCbE84FBwK7APOC1gu03AgdVplhmZmZdKyWA7QdcHBFPSdqsyPaZrPk4aTMzqyEiTSfVKEoJYBuwdq0r3+BelsXMzKqsWQdxPAPs1cX2TwL/6l1xzMysanoxgKMWB3GUEsAuBY6UdAzQmqWFpMGSfgaMAX5Z4fKZmVkFSeUvtabHTYgR8StJOwGXA29myX8GNiYFwp9HxO8rX0QzM7O1lXQjc0QcL+kPwOeBkaTANR0YHxH3VaF8ZmZWIaJJJvPtTEQ8ADxQhbKYmVmVNVD8Kj2ASdoI+BgwIkuaAdwVEUsqWTAzM6u8WhyMUa6SApikk4AzSTNw5L8KyyX9MCJ+WsnCmZlZ5dTqYIxylTKV1HdJs3H8Hfgf0rB6gHcD3wLOk7QqIn5W8VKamZkVKKUG9m3gLuDjEZH/nLDHJV0H3JHlcQAzM6tRjTSIo5T7wLYEri8IXgBERAdwXZbHzMxqlHqx1JpSamCPAzt0sX2HLI+ZmdWoZh3EcSJwo6RHgd9ntS4ktQBfBsYCB1e6gGZmZsV0GsAk3VYkeT4wDrhQ0owsbXtgM9INzT8APlHpQpqZWe+lG5n7uhSV01UNbBSwVn8X8EL2M9fftSxbBpBm5zAzs1pUo5PylqvTABYR263DcpiZ2TrQQPGr9Jk4zMysfjVFDawrkgazehb6NUTEC2vvYWZmVlmlTiX1FeA/gR27yNbaxTYzM+sjjTaIo8c3Mks6Cvhf0iCO00mvxcWk6aVeAx4Bjql8Ec3MrFKa9YnM3wEmR8R+wGVZ2i0RcSqwM7AFMLiyxTMzs0pqpJk4Sglg7wJuyH7vyH72B4iIBaTa2fEVK5mZmVWUlOZCLHepNaUEsOXAquz3N0hBLH/uw1eB4RUql5mZWZdKCWDTSbUwIqINeAI4FECpcfQzwJwKl8/MzCoo90ywcpZaU0oA+yvweUn9s/WLgIMkTQeeAw4ALqlw+czMrIIaaRBHKcPozwV+DrQBRMRVkt4CPk9qTvxhRPyu8kU0M7NKqcE4VLYe18AiYlVELMh/HlhE/DEi/m9EfNbBy8ysuUk6QNIzkqZLOrnI9rGS5kl6NFuOzdt2lKTnsuWonpzPU0mZmTUJUb3RhJJaSd1I+5HGQ0yRNCEinizI+seIOL5g33cAZwKjSZPIT8v2XdTVObt6nMoVZVxDRMRXytjPzMyqrbqDMfYEpkfEDABJ44FDgMIAVswngNsjYmG27+2kcRXXdLVTVzWwfSn+OJWulJrfzMzWoSoOxtgGeDFvfQ7wgSL5DpW0D/As8N2IeLGTfbfp7oR+nEof2vldw5h45//r62JYjdn04J/3dRGsgZUy9LyIIZKm5q1fFhGXdZp7bTcB10TECklfA64iVZbK4j4wMzPrqfkRMbqTbXOBYXnrQ7O0t2WzNuVcDvw0b98xBftO7q4wvQzGZmZWL0RV7wObAoyUNELSAOBwYMIa55e2yls9GHgq+30SsL+kTSVtCuyfpXXJNTAzsyZSrcepRESbpONJgacVuCIinpB0NjA1IiYA35Z0MOl+4oXA2GzfhZLOIQVBgLNzAzq64gBmZtZEqvk8sIiYCEwsSDsj7/dTgFM62fcKoKTR7w5gZmZNIs1p2DhTcbgPzMzM6lJZNTBJI0kPsPxXRCypbJHMzKxaqtmEuK6VVAOT9HlJs4GngXuAPbL0Idn8VZ+rQhnNzKxCmvJxKpIOIU3r8QLwA/KeMB0R80nDIb9U6QKamVlliOZ9IvPpwD0R8WHg0iLbHwLeV5FSmZmZdaOUALYTcG0X218FtuxdcczMrJpaerHUmlIGcbwFDOpi+7bA4l6VxszMqqoGWwLLVkpQvRc4otgGSRsDxwB3VqJQZmZWeepF/1ct9oGVUgM7C7hX0l3A1VnaaEnvBb4PbAScU9nimZlZJdVgHCpbj2tgEfEw6aFj72T1II6fAD8HVgKfiIinOtndzMysokq6kTki/g68R9L7gFGkADgdeDgi/DBLM7Ma10g3Mpc1E0dEPAY8VuGymJlZFeXuA2sUPQ5g2SOguxUR95RfHDMzq6YGil8l1cAmAz1pJmwtryhmZlZVat4mxI8WSWsFRgBfBzro5DkvZmZmldbjABYRd3e2TdKVwP3Av+F7wczMapZonCpYRWYHiYh20r1hx1XieGZmVnlpEEf5S62p5BOZ1wM2q+DxzMyswmoxEJWr1wFM0kak/rETgSm9LpGZmVkPlDKMvoPORyEKmAV8swJlMjOzKlEDjaMvpQZ2NmsHsAAWkWbjuC3rCzMzsxqU6wNrFKWMQjyriuUwM7NqU2PdyNyjUYiS1pe0QNJJ1S6QmZlVTyM9TqVHASwi3iTdqLy0usUxMzPrmVLuA7sJOKhaBTEzs+pq5vvALgKukXQ98GvgeWB5YaaIeKlCZTMzswqrwZbAsnUZwCSdAVwfEf8C/pkl7wQc0sVunszXzKwmiZYGmkqquxrYWaQh8v+i+DB6MzOzPuFh9GZmTUI0UROimZk1kBodjFGungQwNxuamTWIWryfq1w9CWC/lXRVD48XETGwNwUyM7PqaMYmxPuBGdUuiJmZWSl6EsAujYirq14SMzOrumZrQjQzswbRQPHLAczMrFmI0uYPrHUOYGZmzUJN9EDLiGikYG1mZg3ENTAzsybSOPUvBzAzs6aRHqfSOCHMAczMrIk0TvhqrAEpZmbWRBzAzMyaiFT+0v2xdYCkZyRNl3RyF/kOlRSSRmfr20laLunRbPl1T67FTYhmZk1DVRtGL6kVuATYD5gDTJE0ISKeLMg3GDgBeKjgEM9HxK6lnNM1MDOzJpG7kbncpRt7AtMjYkZErATGA4cUyXcOcD7wVq8upmdlMjOzRiGp7AUYImlq3nJc3qG3AV7MW5+TpeWfe3dgWETcUqRoIyQ9IuluSR/uybW4CdHMzHpqfkSMLmdHSS3ARcDYIptfBoZHxAJJewA3SNopIl7v6piugZmZNRH1YunGXGBY3vrQLC1nMLAzMFnSLGAvYIKk0RGxIiIWAETENOB5YFR3J3QNzMysWVR3LsQpwEhJI0iB63DgC7mNEbEEGPJ2UaTJwIkRMVXS5sDCiGiXtD0wkh48h9IBzMysSVRzNvqIaJN0PDAJaAWuiIgnJJ0NTI2ICV3svg9wtqRVQAfw9YhY2N05HcDMzJpINWejj4iJwMSCtDM6yTsm7/frgOtKPZ/7wMzMrC65BmZm1kQaaS5EBzCrqoH9xCbr90OIZSvaeWNF+xrbNxjQwoaDWomAABYta6OtI1hvQAuDB7a+na9/q3jtjVWsao91fAVWDfvtsS0Xfm0fWlvElZOe4MI/TVsrz6EfHslpX/wAEcHjM+cz9qeTGL7FYMaffiAtEv37tfCrmx7j8on/6oMrqF8NNBm9A5hV16br92fe0pW0d8AWg/uzfFUHbR2rg9CbKztYtrIDgEH9W9hk/X7MX7qK5Ss7WJ6l92sRQzbs7+DVIFpaxMX/MYYDT/sLc+cv5d6LP8/ND87k6RdX99nvsPXGnHjYaPY98U8sXrqCzTdeD4CXFy5jzPf+xMq2djYY1J9pv/oitzw4k5cXLuury6kraRBH40Qw94FZ1QxoFW0dQXuKQyxf1cF6A9Z8y+WHpM7+W60/oIU3V7Z3stXqzftHbcnzLy1m1iuvs6qtgz/d8xyf/uD2a+Q55oCdufTmf7J46QoA5i1ZDsCqtg5WtqX3wsD+rQ31bCsrnWtgBbJZkNsi4vi+Lku9a20R7Xm1rfaOYEDr2t+ZNhjYwuCB/UAw/41Va21ff0Ar85eunW71aevNNmTO/KVvr8+dv5Q937XlGnlGbrMJAHde+FlaW1o49w8Pcfu02QAMHbIh1//wYHbYamNOveI+175K1Egxv89qYJImZ9Pp71OQPl3S2HVUhlmSjsxPi4ivO3itW8tWdPDK6ytZ8mYbgwe1rrFtQKsIYo1mR2t8ra0t7Lj1Juz/X9fz5fP/yi+/vS8bbzAAgDnzl7LnN69m52N/y5EfezdbbLJeH5e2nqhX/2pNXzchLgAuVDVvTLA+094RtLas/tO2toj26DwQFWtiXG9AC29mfWHWGF5asJShQzZ8e32bIRsyd8Gatai585dy80MzaGvvYParr/Pc3MXsuPUma+R5eeEynpi9gL13WmO+WOtGNZ8Htq71dQD7X9J8WUcU2yhpZ0mTJM2T9IKk8yT1z9v+AUnTJL0h6V5JZ2RzbOW2nyDp6Wx7bv/WbNtNwHDgcklLJd2WpV8p6fLs9wsk3VBQpjHZ8TboSRmb2cr2oF+LyLUarte/5e2BGTn98gLcoP4ttBUM1Fh/QKv7vxrM1GdfZcetN2HbLTeif78WPrfPSG55cM1Zg256YAb77DIUgM02GsTIbTZh5iuvs81mGzJoQKqlb7LhQD6009Y8O3fROr+GepUbxFHuUmv6ug9sGXAG8GNJ10XEitwGSVsAdwOnAgcBmwM3AstJU45sQrrj+yfAxaRJIm8G8jtL5gCfBGYBuwJ/zX6/NCIOyoLd6RHx+07KNw54VNLmETEvSzsauDYilnVXxnJekEaz+M02hmzYPw2jX9lOW0ew0aBWVrYHb63qYIOBLQzq30IEdAQsXNb29r4D+605CMQaQ3tH8N1fTeamcw+htaWFq257gqdeWMgPjvwADz/3Grc8NJPbp83m47sP5+FfH0l7Rwen/uZeFr7xFvvuNoyfHHswEYEkLr7uYZ6YtaCvL8n6iKKLJp2qnjhN5HgHcB7wGPDbiPippOnAuaRJHz8VEfvm7XMocH5E7Jj1Xf0I2C6yi5B0DvCliNiuk3NeSJqy/7BsfRYFAUzSlaRBHMdm6w8B4yPiZ9mTRF8B9o+I+ySd2FUZOynDccBxANsMHbbHg/98rpSXzZrAyC9e2tdFsBr01q0nTCv3USY5o3beNf7n2tvL3v+AnbbodRkqqa9rYGSzD58EXCPpN3mbRgB7S1qclybSJJGQHpT2QqwZgWfnH1vSEcD3gO1J1zoAeLDEIo4DvgH8DDgMmBMR9/WwjGuJiMuAywD+z257eGSCma1TtdiXVa6+7gMDICJuJU3Fnz/p42zgjojYJG/ZOCJyvb9zgeEFA0CG536RNAz4Pak2t1VEbAxcwpq3G/WkcWo8MCp7kuhYUkDraRnNzGqKRyFWx4nA10j9SAC/BUZLOkbSIEktkraXdEC2/WbSA9K+J6m/pF1J/VM5G5Kubx6wStJewJcKzvkK6bkznYqIxcBfSIFwL+CqvM3dldHMrGYIaFH5S62pmQAWEY8B1wAbZeuvAB8FPkMaeLGIFEi2z7YvBg4Evpht+wVwJbAi2/4UcCZpUMVi4OTs+PnOBY6UtEjSrV0UbxxpMMikiHg5r8xdltHMzKqnz/rA8p8Fk5d2NHm1qIh4Eji4i2M8AOyeW5d0Hnn9YBFxNl2MBuzk2TVji+S7nU5mOuqujGZmtaQWmwLLVTM1sHJI2l/SVlnT3T6k0X2FtSwzM8s00o3MfT4KsZd2JvVDbQS8BFzAmn1UZmaWp5FqYHUdwCLiIuCivi6HmZmte3UdwMzMrOdyoxAbhQOYmVnTqM37ucrlAGZm1ixqdDBGuRzAzMyaSAPFr/oeRm9mZs3LNTAzsyaRBnE0Th3MAczMrIk0TvhyADMzay4NFMEcwMzMmkgjDaP3IA4zM6tLroGZmTWRBhrD4QBmZtZMGih+OYCZmTWVBopg7gMzM7O65BqYmVmTEI01CtEBzMysWXgyXzMzq1cNFL8cwMzMmkoDRTAP4jAzs7rkGpiZWdPwE5nNzKxOeRCHmZnVHdFQXWAOYGZmTaWBIpgHcZiZWUVIOkDSM5KmSzq5i3yHSgpJo/PSTsn2e0bSJ3pyPtfAzMyaSLUGcUhqBS4B9gPmAFMkTYiIJwvyDQZOAB7KS3svcDiwE7A1cIekURHR3tU5XQMzM2siUvlLN/YEpkfEjIhYCYwHDimS7xzgfOCtvLRDgPERsSIiZgLTs+N1yQHMzKyJqBdLN7YBXsxbn5OlrT63tDswLCJuKXXfYtyEaGZmPTVE0tS89csi4rKe7CipBbgIGFupwjiAmZk1i96Po58fEaM72TYXGJa3PjRLyxkM7AxMVmqPfCcwQdLBPdi3KDchmpk1EfXiXzemACMljZA0gDQoY0JuY0QsiYghEbFdRGwHPAgcHBFTs3yHSxooaQQwEvhHdyd0DczMrEmI6s3EERFtko4HJgGtwBUR8YSks4GpETGhi32fkHQt8CTQBnyzuxGI4ABmZtZUqnkfc0RMBCYWpJ3RSd4xBes/An5UyvnchGhmZnXJNTAzs2bSQFNJOYCZmTURP07FzMzqkh+nYmZmdamB4pcHcZiZWX1yDczMrJk0UBXMAczMrEmkmaQaJ4I5gJmZNYuePRalbrgPzMzM6pJrYGZmTaSBKmAOYGZmTaWBIpgDmJlZ0+jRY1HqhgOYmVkT8SAOMzOzPuYamJlZkxAN1QXmAGZm1lQaKII5gJmZNZFGGsThPjAzM6tLroGZmTWRRhqF6ABmZtZEGih+OYCZmTWNBpvM1wHMzKypNE4E8yAOMzOrS66BmZk1CeEmRDMzq1MNFL8cwPrS448+PH/YOwbN7uty1IghwPy+LoTVHL8vVtu2EgdxDcwqIiI27+sy1ApJUyNidF+Xw2qL3xeV55k4zMzM+phrYGZmzaRxKmAOYFYzLuvrAlhN8vuiwhoofjmAWW2ICH9Q2Vr8vqgsNdhMHO4DMzOzuuQamPUZSb8G2iLi+N7kMfP7pOc8CtGajqTJklZIWippiaRHJB3am2NGxNfzP3AkzZJ0ZFd5rG9l74OQtE9B+nRJY9dRGfw+6Q31YqkxDmBWinMiYkNgM+Aa4I+SRvVxmWzdWwBcKDVSb0rzaKD45QBmpYuINuCXQCuwi6RvSHomq5k9KOnDubySdpN0b7ZtoaT7JW2abbtS0uXZ7zcBw4HLs1rebUXyXCDphvyySBoj6Q1JG2TrO0uaJGmepBcknSepf/Vflabyv8BQ4IhiG7v7G0j6gKRp2d/tXklnSJqVt/0ESU9n23P7t2bb/D7ppdxAjnKWWuMAZiWTNAD4JrAKeC9wDvBlUs3sf4G/SspNe3MJcBvwDmBL4HvAysJjRsRBwAvAsRGxYUTsX+TU44BPScqfweRo4NqIWCZpC+Bu4HpgG+CDwH7AKb27YiuwDDgD+LGkgfkbuvsbSNoEmAiMJ70nvgV8reD4c4BPAhsBhwDHAMeC3ye2JgcwK8VpkhaTPmAOAQ4FPgxcGhEPRURbRPwG+CfwhWyflaRvzMMiYlVEPBgRy8o5eUQ8CTwCHAkgaTDwWeCKLMuXgcci4tKIWBkRc4HzsnSrrHHAUuCEgvTu/gafzva7MHs/PMLqvx8AEXFdRMyM5BHgd8DHelowv0+6ol79qzUOYFaKH0XEJhGxRUR8KCJuAoYBMwvyPZ+lQ/rm2wLcK2mmpHMk9Wb06zhgbPb7YcCciLgvWx8B7C1pcW4hfWi9sxfnsyIioh04CThV0mZ5m7r7G2wDvBARkbfPGhNaSzpC0hRJCyQtIdX2S5031O+TInKPU3ETolnyIrBdQdr2WTrZN+ljImIocDCpKaizb7odPTjfeGCUpN1JH1Dj8rbNBu7Igmxu2TgbeGIVFhG3AlNIzYk53f0N5gLDCwaADM/9ImkY8HvgXGCriNiY1Aydn9/vEwMcwKz3rgS+JmlPSf0kHQ3sClwNIOkoSVtneRcDbUB7J8d6BRjZ1ckiYjHwF9IH3F7AVXmbfwuMlnSMpEGSWiRtL+mAci7MeuREUh9WrobU3d/gZmAw8D1J/SXtSqql52xI+lyaB6yStBfwpYJz+n3SC66BmWUi4mrgh6RvzQuAbwCfiohcs9C+wDRJy4AHSIHtd50c7lzgSEmLJN3axWnHkTr5J0XEy3lleQX4KPAZYBawiPQhtn1ZF2fdiojHSLdUbJStd/k3yALLgcAXs22/IH0JWpFtfwo4E7iR9IXn5Oz4+fw+qVGSDshGJE+XdHKR7V+X9LikR7MRqO/N0reTtDxLf1TpxvTuz7dmU7SZ2bol6Txgj05GFFoF7bb76Jh83z/K3n+T9VundfZ8tuxWh2dJIzrnkJqXj8gG1eTybBQRr2e/Hwz8R0QcIGk74OaI2LmU8rgGZmbrlKT9JW2VNd3tAxzH2rUsq4ZeNB/2oAlxT2B6RMyIiJWkfshD8jPkgldmA6BXNSjPhWhm69rOpH6ojYCXgAtYs4/KqqTKM2psQzZ4KzMH+MBaZZC+SbofdACpiyFnhKRHgNeB0yPi792d0AHMzNapiLgIuKivy2FlGSJpat76ZaU+8iYiLgEukfQF4HTgKOBlYHhELJC0B3CDpJ0KamxrcQAzM2smvauCze+sD4x0i8SwvPWhWVpnxgO/AoiIFaweyDNN0vPAKGBq57u7D8zMrKlUcSaOKcBISSOy6eYOByascW4p//aHA4HnsvTN8+a73J50m8SM7k7oGpiZWROp1v1cEdEm6XhgEmmi7ysi4glJZwNTI2ICcLykj5PmUV1Eaj4E2Ac4W9Iq0o3qX4+Ihd2d08Pozbqg9IyrccCIiJiVpU0GiIgxfVWuQj0tU2/Knu07NCJ2LHXfLo55FnBmRNTgbbKNZ/c9Rse9D0wpe/8NBrZ0Ooy+L7gJ0WqWpLFKD0/MLe2SXpE0XnX4HLLser7d1+UwaxRuQrR6cA7pBsmBwB7AV4CPS9olf4aFdajcG27Hkjq2f165opiVqIHqug5gVg9ui4h7s99/I+kZ4GJSQDiv2A6SNij3sS3dyW7SNKtLtfhYlHK5CdHq0R3ZzxGQ+lGyJsZdJF0haT7pJkqy7R+TdKfSE3mXSbpbeU+Nzsu3l9ITo99SekrvyRT5vippcq4vKS9Nkr6m9KThN7N5+u6VdEi2fRbwEWCHvCbRWXn795d0mtKTiFdkTaWXSXpHkfP8p6TZ2dxxD0j6UHkv49vH/L6kvys9nXhFVoYTpeLd/ZJ2knRX9lq+LOlHKvKIHEmfU3pC95uSXpd0i6RdelNW651Ge5yKa2BWj3KDCOYXpF9DClxnkmY1R9JhWfrdwA9I/4fHAn+T9PGIuCfL915SYHyDNFnsStIUR0t7WKZfkWZln5ydZxXwfuATpIlpv0OqLW5KmsGd3LGzQHEdaQ653ANBtyc9rXhPSXtFxFvZPmcAZwF/I81gMZI0w/si1pwFoRTfA24B/kx6WsB+2bE3BU4ryDsYuB24FfhTlvdU0tOVv5HLJOnE7Bh/IU3evGG2/T5JoyPi2TLLar3w8MPTJq3XX0N6cYjC/3N9KyK8eKnJhRRognS/yBBga+Ag0gzi7cDuWb6zsnw3ko2szdI3IM2Q/4eC464HTAfuy0u7jhR0RuWlbU6aET2A7fLSJwOT89b3yfKMyz9/tk0F+00vcp1HZPvvV5C+f5b+1Wx9COlmzzuB1rx8x2X5Jhceu8i5JhfmA9Yvku9yUoAdWLBvAD8syPsH0tDnd2frw7LX8kcF+bYEFub/PXJ/u75+r3mpz8VNiFYPbiY9H2ou6cbIQcCXIuLhgny/ioj8+0L2I9UMfi9pSG4hBbY7gL0krZ/dQHkAMDHyagYRMY/04dydz2U/Tys4P4Xrnfg86abNRwrK+TCwhNXzxe1Hmj/ufyI9ETlnXJavLBHxJoDS89w2zc49mfQ6vaswO/DfBWkXk2q2B2brh5Jad64puJ520iN19sWsAtyEaPXgu8C/SB+A84CnCj7Ac54vWM8NtZ/YxbE3I9UW1geeKbK9WFqhHYGFEfFSD/IWM4rUZDivk+1bZD+3LVamiFglqdtZCzoj6VOkpsk9WPszYZOC9fmx9g2mufKMyH7mXvfHOzllT56obNYtBzCrB1Nj9SjEriwvWM+1MHwFeKGTfeax9of0utYCPE3q8ypmUbVOnA0AuYlUM/oPUi13JbA7cD7lDfTK7fNpsvntzKrBAcwa2fTs5/yIuKOzTJLmAW+ydnMZnaQVO88BkrbuphbWWXPidNJjJ+6MiK5qJ7mnXL8LyH9IYH9S7eexHpS10OdIAevjsXqgSG4+umKGSHpHQS0s9xrNzH7mXvcXI+KfZZTJrEfcB2aNbBJpEMbpkgYWbpS0OUDWHDkJ+FT+DB/Z9i/24Dx/yn7+qHDoecH6MorX9saTBmh8p0gZW/OG0t9OCjbfkpT/f/foTo7bEx2kwNqad85BdF4bFHBCQdp3sp+5ptrrSKMZf1hQztzxNy+zrGZrcA3MGlZEvCEp97TfxyX9gfQAxW1I92QBfDT7eQZpyPvdkn5B6hc7jjTi8X3dnOceSZcDxwLbSbqZ1HS2B6lm980s6zTgQEkXZr8vjYibSANFDgX+X3Z/2t2kALBDln4GcGVEzJd0PmmY/m2SbiD1v32ZHszc3YkJpD7GOyT9jjRM/ijgrU7yvwZ8VdLQ7Bo+Dvw76blQT2Wvx0xJ/0l65tc/JF1PGn49nDRY5l+kEaZmvdPXwyC9eOlsYfUw+n/rJt9ZWb6hnWz/EGkk40LSB/Ms4FrggCL5HsjyvACcTKrddDmMPksTKVA9lu2/EPg7cFBeno2BP5L6tAKYlbetlVSTeZTUl7eEdD/YBaQH/eWf52TSPV/LgQezcq9Vpk5ei2Jl/yLwRN51n00a8RjAmIJ9p5OeqHwXKTi/Qrq/rX+Rcx1IGvL/epZ3OnAlsFfh366v32te6nPxbPRmZlaX3AdmZmZ1yQHMzMzqkgOYmZnVJQcwMzOrSw5gZmZWlxzAzMysLjmAmZlZXXIAMzOzuuQAZmZmdckBzMzM6tL/B1xcZxBBVxxIAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x432 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "cnf_matrix = confusion_matrix(test_data.sentiment.to_list(), y_pred_1d)\n",
    "plt.figure(figsize=(6,6))\n",
    "plot_confusion_matrix(cnf_matrix, classes=test_data.sentiment.unique(), title=\"Confusion matrix\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Classification Scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "    Negative       0.64      0.66      0.65    119960\n",
      "    Positive       0.65      0.63      0.64    120041\n",
      "\n",
      "    accuracy                           0.64    240001\n",
      "   macro avg       0.64      0.64      0.64    240001\n",
      "weighted avg       0.64      0.64      0.64    240001\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(classification_report(list(test_data.sentiment), y_pred_1d))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}