Financial News Analysis using NLP .ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Financial News Analysis using NLP "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**About this file**\n",
    "\n",
    "\n",
    "This dataset contains the sentiments for financial news headlines from the perspective of a retail investor. Further details about the dataset can be found in: Malo, P., Sinha, A., Takala, P., Korhonen, P. and Wallenius, J. (2014): “Good debt or bad debt: Detecting semantic orientations in economic texts.” Journal of the American Society for Information Science and Technology."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### NLP\n",
    "Natural language processing (NLP) is a subfield of linguistics, computer science, and artificial intelligence concerned with the interactions between computers and human language, in particular how to program computers to process and analyze large amounts of natural language data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "NLP is currently the focus of significant interest in the machine learning community. Some of the use cases for NLP are listed here:\n",
    "\n",
    "    -- Chatbots\n",
    "    -- Search(text and Audio)\n",
    "    -- Text Classification\n",
    "    -- Sentiment Analysis\n",
    "    -- Recomendation System\n",
    "    -- Quesstion Answering\n",
    "    -- Speech recognation\n",
    "    -- NLU (Natural Language Understanding)\n",
    "    -- NLG ( Natural Language Generation)\n",
    "\n",
    "\n",
    "You encounter many of these use cases in everyday life : when you visit web pages or perform an online search for books, or recommendation regarding movies"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**NLP Techniques**\n",
    "\n",
    "The earliest approach for solving NLP task involves rule-based approachers, which dominated the indusytry for decades. Examples of techniques using rule based approaches include Regular Expressions (RegExs) and Context Free Grammars (CFGs). RegExs are sometimes used in order to remove HTML tags from text that has been scraped from a web page or uwanted special characters from a document.\n",
    "\n",
    "The second approach involved training a machine learning model with some data that is based on some user defined features. This technique requires a considerable amount of features engineering ( a nontrivial task), and includes analyzing the text to remove undersired and superfluous content(including stop words), as well as transforming the word (e.g., converting upercase to lowercase).\n",
    "\n",
    "The most recent approach involves deep learning, whereby a neural network learns the features instead of relying on human to perform feature engineering. One of the key ideas involves mapping words to numbers, which enables us to map sentence to vector of number. After transforming documents to vector, we can perform a myriad of operations on those vector. For example we use the notion of vector space to define vector space model, where the distance between two vector can be measured by the angle between them (related to cosine similarity). If two vector are closed to each other, then it's likelier that the coresponding sentence are similar meaning. Their similarity is based on the distributional hypothesis, which asserts that words in the same contexts tent to have similar meaning.The NLP models that use deep learning can comprise CNNs, RNNs, LSTMs, and bidirectional LSTMs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**This work tries to classifying sentiment from financial news using Combination Doc2vec and LSTM**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What is Sentimen Analysis ?\n",
    "Sentiment analysis refers to the use of natural language processing, text analysis, computational linguistics, and biometrics to systematically identify, extract, quantify, and study affective states and subjective information. Sentiment Analysis [2]: the process of understanding if a given text is talking positively or negatively about a given subject, aiming for brand monitoring.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Different Levels of Analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Document level:\n",
    "\n",
    "The task at this level is to classify whether a whole opinion document expresses a positive or negative sentiment (Pang, Lee and Vaithyanathan, 2002; Turney, 2002). For example, given a product review, the system determines whether the review expresses an overall positive or negative opinion about the product. This task is commonly known as document-level sentiment classification. This level of analysis assumes that each document expresses opinions on a single entity (e.g., a single product). Thus, it is not applicable to documents which evaluate or compare multiple entities."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sentence level:\n",
    "\n",
    "The task at this level goes to the sentences and determines whether each sentence expressed a positive, negative, or neutral opinion. Neutral usually means no opinion. This level of analysis is closely related to subjectivity classification (Wiebe, Bruce and O'Hara, 1999), which distinguishes sentences (called objective sentences) that express factual information from sentences (called subjective sentences) that express subjective views and opinions. However, we should note that subjectivity is not equivalent to sentiment as many objective sentences can imply opinions, e.g., “We bought the car last month and the windshield wiper has fallen off.” Researchers have also analyzed clauses (Wilson, Wiebe and Hwa, 2004), but the clause level is still not enough, e.g., “Apple is doing very well in this lousy economy.”"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Entity and Aspect level:\n",
    "\n",
    "Both the document level and the sentence level analyses do not discover what exactly people liked and did not like. Aspect level performs finer-grained analysis. Aspect level was earlier called feature level (feature-based opinion mining and summarization) (Hu and Liu, 2004). Instead of looking at language constructs (documents, paragraphs, sentences, clauses or phrases), aspect level directly looks at the opinion itself. It is based on the idea that an opinion consists of a sentiment (positive or negative) and a target (of opinion). An opinion without its target being identified is of limited use. Realizing the importance of opinion targets also helps us understand the sentiment analysis problem better."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What is LSTM ?\n",
    "Long Short Term Memory (LSTM) is a special kind of Recurrent Neural Network (RNN), capable of learning long-term dependencies. These long-term dependencies have a great influence on the meaning and overall polarity of a document. Long short-term memory networks (LSTM) address this long-term dependency problem by introducing a memory into the network. It was first introduced by Hochreiter & Schmidhuber."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The LSTM architecture has a range of repeated modules for each time step as in a standard RNN. At each time step, the output of the module is controlled by a set of gates, as a function of the old hidden state ℎ𝑡−1 and the input at the current time step 𝑥𝑡 : the forget gate 𝑓𝑡, the input gate 𝑖𝑡 , and the output gate 𝑂𝑡 . These gates collectively decide how to update the current memory cell 𝐶𝑡 and the current hidden state ℎ𝑡 . The LSTM transition functions are defined as follows:\n",
    "\n",
    "    𝑖𝑡=(𝑊𝑖[ℎ𝑡−1,𝑥𝑡]+𝑏𝑖)\n",
    "\n",
    "    𝐶´𝑡=𝑡𝑎𝑛ℎ(𝑊𝑐[ℎ𝑡−1,𝑥𝑡]+𝑏𝐶)\n",
    "\n",
    "    𝑓𝑡=(𝑊𝑓[ℎ𝑡−1,𝑥𝑡]+𝑏𝑓)\n",
    "\n",
    "    𝑂𝑡=(𝑊𝑜[ℎ𝑡−1,𝑥𝑡]+𝑏𝑜)\n",
    "\n",
    "    𝐶𝑡= 𝑓𝑡∗𝐶𝑡−1+𝑖𝑡∗𝐶´𝑡\n",
    "\n",
    "Here 𝜎 is logistic sigmoid function that has an output in 𝑛 [0,1] ,tanh denotes the hyperbolic tangent function that has an output ℎ 𝑖𝑛 [−1,1], and ∗ denotes the pointwise multiplication.\n",
    "\n",
    "As I mentioned before to deal with Deep learning we have to map sentence to vector of number. In this work inspired by [3] Doc2vec as the embedding used for extracting information context. The Doc2vec is an unsupervised algorithm that learns fixed-length feature representations from variable-length pieces of texts such as sentences, paragraphs, and documents."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Let's Explore the given theory concepts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\tqdm\\std.py:697: FutureWarning: The Panel class is removed from pandas. Accessing it from the top-level namespace will also be removed in the next version\n",
      "  from pandas import Panel\n"
     ]
    }
   ],
   "source": [
    "## Import Libraries\n",
    "import pandas as pd # Dataframe Handling\n",
    "import numpy as np # Numirical calculations and mapping\n",
    "from tqdm import tqdm # \n",
    "from keras.preprocessing.text import Tokenizer # To tocanised the data\n",
    "tqdm.pandas(desc=\"progress-bar\") \n",
    "from gensim.models import Doc2Vec # Doc Vector model\n",
    "from sklearn import utils # It helps to generate the charts\n",
    "from sklearn.model_selection import train_test_split # To split the data\n",
    "from keras.preprocessing.sequence import pad_sequences # To specify the length of word or sentence\n",
    "import gensim # to represant the data in vectors\n",
    "from sklearn.linear_model import LogisticRegression # Logistic regression\n",
    "from gensim.models.doc2vec import TaggedDocument\n",
    "import re\n",
    "import seaborn as sns # To visualised the data\n",
    "import matplotlib.pyplot as plt # To visualised the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>neutral</th>\n",
       "      <th>According to Gran , the company has no plans to move all production to Russia , although that is where the company is growing .</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>neutral</td>\n",
       "      <td>Technopolis plans to develop in stages an area...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>negative</td>\n",
       "      <td>The international electronic industry company ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>positive</td>\n",
       "      <td>With the new production plant the company woul...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>positive</td>\n",
       "      <td>According to the company 's updated strategy f...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>positive</td>\n",
       "      <td>FINANCING OF ASPOCOMP 'S GROWTH Aspocomp is ag...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    neutral  \\\n",
       "0   neutral   \n",
       "1  negative   \n",
       "2  positive   \n",
       "3  positive   \n",
       "4  positive   \n",
       "\n",
       "  According to Gran , the company has no plans to move all production to Russia , although that is where the company is growing .  \n",
       "0  Technopolis plans to develop in stages an area...                                                                               \n",
       "1  The international electronic industry company ...                                                                               \n",
       "2  With the new production plant the company woul...                                                                               \n",
       "3  According to the company 's updated strategy f...                                                                               \n",
       "4  FINANCING OF ASPOCOMP 'S GROWTH Aspocomp is ag...                                                                               "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Load and read the data\n",
    "df = pd.read_csv('all-data.csv',delimiter=',',encoding='latin-1')\n",
    "df.head() # For top 5 rows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Rename the column becxause the length of name of column in large\n",
    "df = df.rename(columns={'neutral':'sentiment','According to Gran , the company has no plans to move all production to Russia , although that is where the company is growing .':'Message'})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(4845, 2)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Dimension of the data\n",
    "df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "111964"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Find the len of the message\n",
    "df.index = range(4845)\n",
    "df['Message'].apply(lambda x: len(x.split(' '))).sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\seaborn\\_decorators.py:36: FutureWarning: Pass the following variables as keyword args: x, y. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAt0AAAEuCAYAAABS2a7YAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAiHElEQVR4nO3debRddX338fcHRAYBBQ2ICQGUQcEBMEa0PlalCNpasE6hMtQpirN1VUGtRSutWouPVKWiIPAIAq0yOKAiohQEaUCRSUqUAIHIoFKjIkLyff44O/X0eu/NDrn7npNz36+1zjp7f/fe53xu1srKN/v+9u+XqkKSJElSd9YbdABJkiRp1Nl0S5IkSR2z6ZYkSZI6ZtMtSZIkdcymW5IkSeqYTbckSZLUsWlpupNslOSyJFcmuSbJ+5r6lknOS3JD875F3zVHJFmc5Pok+/bVn5zkqubYMUkyHT+DJEmS9EBN153ue4HnVNWTgN2B/ZLsBRwOnF9VOwHnN/sk2RVYAOwG7Ad8Msn6zWcdCywEdmpe+03TzyBJkiQ9INPSdFfPr5rdDZpXAfsDJzX1k4ADmu39gdOq6t6quhFYDMxPsg2weVVdUr1VfU7uu0aSJEkaSg+ari9q7lRfDuwIfKKqvpdk66paBlBVy5Js1Zw+G7i07/KlTe2+ZntsfVKPeMQjavvtt1/7H0KSJEmawOWXX35XVc0a79i0Nd1VtQLYPcnDgDOTPH6S08cbp12T1P/wA5KF9IahMHfuXBYtWrRmgSVJkqQ1kOSmiY5N++wlVXU38G16Y7Fvb4aM0Lzf0Zy2FNi277I5wG1Nfc449fG+57iqmldV82bNGvc/HJIkSdK0mK7ZS2Y1d7hJsjHwJ8CPgHOAQ5vTDgXObrbPARYk2TDJDvQemLysGYqyPMlezawlh/RdI0mSJA2l6Rpesg1wUjOuez3gjKr6cpJLgDOSvAq4GXgJQFVdk+QM4FrgfuANzfAUgMOAE4GNgXOblyRJkjS00psEZLTNmzevHNMtSZKkLiW5vKrmjXfMFSklSZKkjtl0S5IkSR2z6ZYkSZI6ZtMtSZIkdWzaFscZRQd97CuDjiBNic+95U8HHUGSpJHmnW5JkiSpYzbdkiRJUsdsuiVJkqSO2XRLkiRJHbPpliRJkjpm0y1JkiR1zKZbkiRJ6phNtyRJktQxm25JkiSpYzbdkiRJUsdsuiVJkqSO2XRLkiRJHbPpliRJkjpm0y1JkiR1zKZbkiRJ6phNtyRJktQxm25JkiSpYzbdkiRJUsdsuiVJkqSO2XRLkiRJHbPpliRJkjpm0y1JkiR1bFqa7iTbJrkgyXVJrknylqZ+ZJJbk/ygeT2/75ojkixOcn2SffvqT05yVXPsmCSZjp9BkiRJeqAeNE3fcz/w9qq6IslmwOVJzmuOfbSqPtJ/cpJdgQXAbsCjgG8m2bmqVgDHAguBS4GvAvsB507TzyFJkiStsWm5011Vy6rqimZ7OXAdMHuSS/YHTquqe6vqRmAxMD/JNsDmVXVJVRVwMnBAt+klSZKktTPtY7qTbA/sAXyvKb0xyQ+TnJBki6Y2G7il77KlTW12sz22LkmSJA2taW26k2wKfAF4a1X9kt5QkccAuwPLgH9edeo4l9ck9fG+a2GSRUkW3XnnnWsbXZIkSXrAWjXdSXZNsnWzvWmS9yV5b5JN2n5Rkg3oNdynVNUXAarq9qpaUVUrgU8D85vTlwLb9l0+B7itqc8Zp/4Hquq4qppXVfNmzZrVNqYkSZI05dre6T4VeFiz/RHgmcDTgE+1ubiZYeR44LqqOrqvvk3faS8Erm62zwEWJNkwyQ7ATsBlVbUMWJ5kr+YzDwHObvkzSJIkSQPRdvaS7avq+qbRfSG9WUXuAW5sef0fAQcDVyX5QVN7F3Bgkt3pDRFZArwWoKquSXIGcC29mU/e0MxcAnAYcCKwMb1ZS5y5RJIkSUOtbdN9bzPV367ALVV1V5IHARu1ubiqLmL88dhfneSao4CjxqkvAh7fKrUkSZI0BNo23acC3wI2Az7e1Pak/Z1uSZIkacZq1XRX1duSPBe4r6ouaMorgbd1lkySJEkaEa1XpKyqbzTLue9VVZc2wzwkSZIkrUbbKQPnJrkY+BHwzab24iSf6TKcJEmSNAraThn4KeAr9MZ039fUzgP26SKUJEmSNEraDi+ZD/xpVa1MUgBV9d9JHtpdNEmSJGk0tL3TfTuwY38hya7AzVOeSJIkSRoxbZvujwBfTvIK4EFJDgROBz7UWTJJkiRpRLSdMvCEJD8HFgK30Ft+/W+r6qwOs0mSJEkjYU2mDDwLOKuzJJIkSdKIajtl4DFJnj6m9vQk/7eTVJIkSdIIaTum+0Bg7GI4lwN/ObVxJEmSpNHTtumucc5dfw2ulyRJkmastk3zfwAfSLIeQPN+ZFOXJEmSNIm2D1K+BfgysCzJTcBcYBnwgq6CSZIkSaOi7ZSBS5PsCTwVmENv2sDLqmpll+EkSZKkUbAmUwauBC5ZNcQEesNMbLwlSZKkybWdMnDPJJck+TVwX/O6v3mXJEmSNIm2d7pPAr4EvBL4TXdxJEmSpNHTtuneDnh3VVWXYSRJkqRR1HbKwDOB53YZRJIkSRpVbe90bwScmeQi4Kf9B6rqkClPJUmSJI2Qtk33tc1LkiRJ0hpqO0/3+7oOIkmSJI2qtmO6SbJPkuOTfKnZn5fkOd1FkyRJkkZD23m63wQcC9wAPLMp3wN8oKNckiRJ0shoe6f7rcCfVNUHgVUrUP4I2KWLUJIkSdIoadt0bwbc0myvmqt7A+B3bS5Osm2SC5Jcl+SaJG9p6lsmOS/JDc37Fn3XHJFkcZLrk+zbV39ykquaY8ckScufQZIkSRqItk33hcDhY2pvBi5oef39wNur6nHAXsAbkuzafOb5VbUTcP6q72iOLQB2A/YDPplk/eazjgUWAjs1r/1aZpAkSZIGom3T/SbghUmWAJsluR54CfDXbS6uqmVVdUWzvRy4DpgN7E9viXma9wOa7f2B06rq3qq6EVgMzE+yDbB5VV3SrI55ct81kiRJ0lBa7ZSBSdYDHgf8H+AJ9JaEvwW4rKpWTnbtBJ+3PbAH8D1g66paBr3GPMlWzWmzgUv7Llva1O5rtsfWJUmSpKG12qa7qlYmObuqNgMua14PSJJNgS8Ab62qX04yHHu8AzVJfbzvWkhvGApz585d87CSJEnSFGk9pjvJXmvzRUk2oNdwn1JVX2zKtzdDRmje72jqS4Ft+y6fA9zW1OeMU/8DVXVcVc2rqnmzZs1am+iSJEnSWmm7DPxNwLlJzqY3tOR/7i5X1XtXd3Ezw8jxwHVVdXTfoXOAQ4EPNu9n99VPTXI08Ch6D0xeVlUrkixv/gPwPeAQ4F9a/gySJEnSQLRtujcGzmq2++80jzu0Yxx/BBwMXJXkB03tXfSa7TOSvAq4md7DmVTVNUnOAK6lN/PJG6pqRXPdYcCJTaZzm5ckSZI0tNo8SLk+vbvbR1XVvQ/kS6rqIsYfjw2w9wTXHAUcNU59EfD4B5JDkiRJGoTVjulu7jC/nt7MIZIkSZLWUNsHKU8GXtdlEEmSJGlUtR3TPR94U5J38IcPUj6zi2CSJEnSqGjbdH+6eUmSJElaQ62a7qo6afVnSZIkSRpPq6Y7ySsnOlZVJ0xdHEmSJGn0tB1ecvCY/UcCjwEuBmy6JUmSpEm0HV7y7LG15u7346Y8kSRJkjRi2k4ZOJ4TgVdNUQ5JkiRpZLUd0z22Od8EOAi4e6oDSZIkSaOm7Zju++mbm7txK7BwauNIkiRJo6dt073DmP1fV9VdUx1GkiRJGkVrcqf7N1X1i1WFJFsAG1fVbZ0kkyRJkkZE2wcpzwLmjKnNAc6c0jSSJEnSCGrbdO9SVVf1F5r9x059JEmSJGm0tG2670iyY3+h2f/Z1EeSJEmSRkvbpvsE4AtJ/izJrkleAPw78JnuokmSJEmjoe2DlB8E7gM+AmwL3AwcDxzdUS5JkiRpZLRdBn4l8E/NS5IkSdIaaDW8JMnhSZ4ypjY/yTu6iSVJkiSNjrZjut8CXDumdi3w1ilNI0mSJI2gtk33g+mN6e73O2CjqY0jSZIkjZ62TfflwOvH1F4HXDG1cSRJkqTR03b2krcB5yU5GPgxsCOwNbBPV8EkSZKkUdF29pJrkuwM/Bm9KQO/CHy5qn7VZThJkiRpFLS90w2wDXATcHlV3dBRHkmSJGnkrHZMd5K/SLIEuB64GPhRkiVJXtx1OEmSJGkUTNp0J/lT4LPAJ4FHAxsDjwGOBT6T5M/afEmSE5LckeTqvtqRSW5N8oPm9fy+Y0ckWZzk+iT79tWfnOSq5tgxSbJGP60kSZI0AKu70/23wGur6sNVtaSq7m3ePwQc1hxv40Rgv3HqH62q3ZvXVwGS7AosAHZrrvlkkvWb848FFgI7Na/xPlOSJEkaKqtruncDzpzg2BeBXdt8SVVdCPy8Zab9gdOaBv9GYDEwP8k2wOZVdUlVFXAycEDLz5QkSZIGZnVN973A5hMcexi9BXLWxhuT/LAZfrJFU5sN3NJ3ztKmNrvZHluXJEmShtrqmu6vAf84wbF/AL6+Ft99LL3x4bsDy4B/burjjdOuSerjSrIwyaIki+688861iClJkiStndVNGfhO4KIkPwS+QK853gZ4Eb074M94oF9cVbev2k7yaeDLze5SenOBrzIHuK2pzxmnPtHnHwccBzBv3rwJm3NJkiSpa5Pe6a6qW4E9gbPpPbT4N8372cCeVbV0kssn1YzRXuWFwKqZTc4BFiTZMMkO9B6YvKyqlgHLk+zVzFpySJNDkiRJGmqrXRynqn5Bb5aStjOV/IEknweeBTwiyVLg74BnJdmd3hCRJcBrm++7JskZwLXA/cAbqmpF81GH0ZsJZWPg3OYlSZIkDbU1WZHyAauqA8cpHz/J+UcBR41TXwQ8fgqjSZIkSZ1b7YqUkiRJktaOTbckSZLUsQmb7iSX9m3/3fTEkSRJkkbPZHe6d06yUbP99ukII0mSJI2iyR6kPBv4ryRLgI2TXDjeSVX1zC6CSZIkSaNiwqa7ql6R5BnA9sBTmGS2EUmaTj/99MsGHUGaEo98zemDjiBpmkw6ZWBVXURvRcoHV9VJ05RJkiRJGimt5umuqhOSPBs4GJgN3Ap8rqq+1WU4SZIkaRS0mjIwyauB04GfAl8ElgGnJnlNh9kkSZKkkdB2Rcp3APtU1ZWrCklOB74AfLqLYJIkSdKoaLs4zsOBa8fUrge2nNo4kiRJ0uhp23RfBBydZBOAJA8B/gn4blfBJEmSpFHRtul+HfBE4L+T3A7cDTwJeG1HuSRJkqSR0Xb2kmXAHyeZAzwKuK2qlnaaTJIkSRoRbR+kBKBptG22JUmSpDXQdniJJEmSpAfIpluSJEnq2Gqb7iTrJXlOkgdPRyBJkiRp1Ky26a6qlcDZVfW7acgjSZIkjZy2w0suTLJXp0kkSZKkEdV29pKbgHOTnA3cAtSqA1X13i6CSZIkSaOibdO9MXBWsz2nmyiSJEnSaGq7OM4rug4iSZIkjarWi+MkeRzwYmDrqnpjkl2ADavqh52lkyRJkkZAqwcpk7wEuBCYDRzSlDcDju4olyRJkjQy2s5e8n5gn6p6HbCiqV0JPKmTVJIkSdIIadt0b0WvyYbfz1xSfduSJEmSJtC26b4cOHhMbQFwWZuLk5yQ5I4kV/fVtkxyXpIbmvct+o4dkWRxkuuT7NtXf3KSq5pjxyRJy/ySJEnSwLRtut8MfCDJd4CHJPk68PfA21pefyKw35ja4cD5VbUTcH6zT5Jd6TX0uzXXfDLJ+s01xwILgZ2a19jPlCRJkoZOq6a7qn4EPBb4BPAe4LPAE6rqhpbXXwj8fEx5f+CkZvsk4IC++mlVdW9V3QgsBuYn2QbYvKouqaoCTu67RpIkSRparacMrKrfJLkYuBG4rap+tZbfvXVVLWs+e1mSrZr6bODSvvOWNrX7mu2xdUmSJGmotZ0ycG6S/wCWAF8BliS5KMl2HWQab5x2TVIf/0OShUkWJVl05513Tlk4SZIkaU21HdN9Er2HKR9WVVsBWwD/ye+HhzwQtzdDRmje72jqS4Ft+86bA9zW1OeMUx9XVR1XVfOqat6sWbPWIqYkSZK0dto23U8G/qaqfg3QDC15Z1N/oM4BDm22DwXO7qsvSLJhkh3oPTB5WTMUZXmSvZpZSw7pu0aSJEkaWm2b7kuB+WNq84BL2lyc5PPNubskWZrkVcAHgX2S3ADs0+xTVdcAZwDXAl8D3lBVqxbkOQz4DL2HK38MnNsyvyRJkjQwEz5ImeT9fbs/Br6a5CvALfSGfzwfOLXNl1TVgRMc2nuC848Cjhqnvgh4fJvvlCRJkobFZLOXbDtm/4vN+1bAvcCZwEZdhJIkSZJGyYRNd1W9YjqDSJIkSaOq9TzdSTYBdgQ27a9X1XenOpQkSZI0Slo13UkOAT4O/A64p+9QAXM7yCVJkiSNjLZ3uj8MvKiqzusyjCRJkjSK2k4Z+Dvg2x3mkCRJkkZW26b7b4GjkzyiyzCSJEnSKGrbdP8X8Of0lm5f0bxWJlmxugslSZKkma7tmO7/B5wMnM7/fpBSkiRJ0mq0bbofDry3qqrLMJIkSdIoaju85LPAwV0GkSRJkkZV2zvd84E3Jnk3cHv/gap65pSnkiRJkkZI26b7081LkiRJ0hpq1XRX1UldB5EkSZJGVdtl4F850bGqOmHq4kiSJEmjp+3wkrEPUT4SeAxwMWDTLUmSJE2i7fCSZ4+tNXe/HzfliSRJkqQR03bKwPGcCLxqinJIkiRJI6vtmO6xzfkmwEHA3VMdSJIkSRo1bcd03w+MXY3yVuA1UxtHkiRJGj1tm+4dxuz/uqrumuowkiRJ0ihq+yDlTV0HkSRJkkbVpE13kgv4w2El/aqq9p7aSJIkSdJoWd2d7s9NUJ8NvJneA5WSJEmSJjFp011Vx/fvJ3k4cAS9ByhPB97fXTRJkjRMXnn6hAtUS+uUE142/Ws7tpqnO8nmSf4eWAxsDexZVQurammn6SRJkqQRMGnTnWTjJEcAP6G3+uQzqurgqvrxtKSTJEmSRsDqxnTfCKwPfBhYBGydZOv+E6rqW2sTIMkSYDmwAri/quYl2ZLe8JXtgSXAS6vqF835R9BbCXMF8Oaq+vrafL8kSZLUtdU13b+lN3vJYRMcL+DRU5Dj2WPm/T4cOL+qPpjk8Gb/nUl2BRYAuwGPAr6ZZOeqWjEFGSRJkqROrO5Byu2nKcdY+wPParZPAr4NvLOpn1ZV9wI3JlkMzAcuGUBGSZIkqZVWD1J2rIBvJLk8ycKmtnVVLQNo3rdq6rOBW/quXdrUJEmSpKHVdhn4Lv1RVd2WZCvgvCQ/muTcjFMbd/GepoFfCDB37ty1TylJkiQ9QAO/011VtzXvdwBn0hsucnuSbQCa9zua05cC2/ZdPge4bYLPPa6q5lXVvFmzZnUVX5IkSVqtgTbdSR6SZLNV28BzgauBc4BDm9MOBc5uts8BFiTZMMkOwE7AZdObWpIkSVozgx5esjVwZpJVWU6tqq8l+U/gjCSvAm4GXgJQVdckOQO4FrgfeIMzl0iSJGnYDbTprqqfAE8ap/4zYO8JrjkKOKrjaJIkSdKUGfiYbkmSJGnU2XRLkiRJHbPpliRJkjpm0y1JkiR1zKZbkiRJ6phNtyRJktQxm25JkiSpYzbdkiRJUsdsuiVJkqSO2XRLkiRJHbPpliRJkjpm0y1JkiR1zKZbkiRJ6phNtyRJktQxm25JkiSpYzbdkiRJUsdsuiVJkqSO2XRLkiRJHbPpliRJkjpm0y1JkiR1zKZbkiRJ6phNtyRJktQxm25JkiSpYzbdkiRJUsdsuiVJkqSO2XRLkiRJHbPpliRJkjq2TjbdSfZLcn2SxUkOH3QeSZIkaTLrXNOdZH3gE8DzgF2BA5PsOthUkiRJ0sTWuaYbmA8srqqfVNXvgNOA/QecSZIkSZrQuth0zwZu6dtf2tQkSZKkoZSqGnSGNZLkJcC+VfXqZv9gYH5VvWnMeQuBhc3uLsD10xpUU+URwF2DDiHNUP79kwbDv3vrru2qatZ4Bx403UmmwFJg2779OcBtY0+qquOA46YrlLqRZFFVzRt0Dmkm8u+fNBj+3RtN6+Lwkv8EdkqyQ5IHAwuAcwacSZIkSZrQOnenu6ruT/JG4OvA+sAJVXXNgGNJkiRJE1rnmm6Aqvoq8NVB59C0cIiQNDj+/ZMGw797I2ide5BSkiRJWtesi2O6JUmSpHWKTbckSZLUMZtuSZIkqWM23ZIkANJzUJL3Nvtzk8wfdC5ppkiycZJdBp1D3bDp1tBIsjzJL8d5LU/yy0Hnk2aATwJPAw5s9pcDnxhcHGnmSPIC4AfA15r93ZO4DskIWSenDNRoqqrNBp1BmuGeWlV7Jvk+QFX9olmETFL3jgTmA98GqKofJNl+gHk0xWy6NbSSbAVstGq/qm4eYBxpJrgvyfpAASSZBawcbCRpxri/qv47yaBzqCMOL9HQSfLnSW4AbgS+AywBzh1oKGlmOAY4E9gqyVHARcA/DDaSNGNcneQvgfWT7JTkX4DvDjqUpo6L42joJLkSeA7wzaraI8mzgQOrauGAo0kjL8ljgb2BAOdX1XUDjiTNCEk2Ad4NPLcpfR34QFX9dnCpNJVsujV0kiyqqnlN871HVa1McllVOYuC1KEkHwNOryrvrknTLMkeVfX9QedQdxzTrWF0d5JNgQuBU5LcAdw/4EzSTHAF8J4kO9MbZnJ6VS0acCZppjg6yTbAvwGnVdU1gw6kqeWdbg2dJA8B7qH3zMHLgYcCp1TVzwYaTJohkmwJvAhYAMytqp0GHEmaEZI8Engp8DJgc3r/8f3AYFNpqth0a6g0Myd8var+ZNBZpJmqWRDnZcABwLVV9YLBJpJmliRPAN4BvKyqnLZzRDh7iYZKVa0AfpPkoYPOIs00ST7UzBz0fuAa4Mk23NL0SPK4JEcmuRr4OL2ZS+YMOJamkGO6NYx+C1yV5Dzg16uKVfXmwUWSZoQbgadV1V2DDiLNQJ8FPg88t6puG3QYTT2Hl2joJDl0nHJV1cnTHkaaAZI8tqp+lGTP8Y5X1RXTnUmSRo13ujWMHlZVH+svJHnLoMJIM8BfAwuBfx7nWNGbN19SB5KcUVUvTXIVzWqwqw7Ru+H0xAFF0xTzTreGTpIrqmrPMbXvV9Ueg8okzQRJNhq7EMd4NUlTJ8k2VbUsyXbjHa+qm6Y7k7rhg5QaGkkOTPIlYIck5/S9LgCcLlDq3niL4rhQjtShqlrWbL6+qm7qfwGvH2Q2TS2Hl2iYfBdYBjyC//1r7uXADweSSJoBmrmBZwMbJ9mD3q+1oTdP8CYDCybNLPsA7xxTe944Na2jbLo1NJr/1d8EPG3QWaQZZl/gr+hNT3Z0X3058K5BBJJmiiSH0buj/egk/TeYNgMuHkwqdcEx3Ro6SZbz+4dJHgxsAPy6qjYfXCpp9CV5UVV9YdA5pJmkWZdiC+AfgcP7Di2vqp8PJpW6YNOtoZfkAGB+VXnHTepAkoOq6nNJ3s7/nj0BgKo6epzLJHUgyVbARqv2q+rmAcbRFPJBSg29qjoLpyyTuvSQ5n1Ter/SHvuS1LEkL2hWhL0R+A6wBDh3oKE0pbzTraGT5C/6dtcD5gF/XFWO9ZYkjaQkV9K7wfTNqtojybOBA6tq4YCjaYp4p1vD6AV9r33pPcy1/0ATSTNAkg8n2TzJBknOT3JXkoMGnUuaIe6rqp8B6yVZr6ouAHYfcCZNIWcv0dCpqlcMOoM0Qz23qt6R5IXAUuAlwAXA5wYbS5oR7k6yKXAhcEqSO4D7B5xJU8g73Ro6SXZu7rJd3ew/Mcl7Bp1LmgE2aN6fD3zemROkabU/cA/wNuBrwI/p/cZXI8Ix3Ro6Sb4D/A3wqVVLvye5uqoeP9hk0mhL8kHgAHr/8M8HHgZ8uaqeOsBYkjQSvNOtYbRJVV02puav2KSOVdXh9BanmldV9wG/xucppGmRZHmSX4553ZLkzCSPHnQ+rT3HdGsY3ZXkMTTzBSd5Mb3l4SV1KMkGwMHAM5NAb9qyfx1oKGnmOBq4DTgVCLAAeCRwPXAC8KyBJdOUcHiJhk7zP/rjgKcDv6A3Z+nLm2XiJXUkyWfojes+qSkdDKyoqlcPLpU0MyT53tihXEkuraq9klxZVU8aVDZNDe90axjdCnyW3qwJWwK/BA4F3j/IUNIM8JQx/7B/q5k7WFL3ViZ5KfDvzf6L+455h3QEOKZbw+hsek9s30fvV22/oje2VFK3VjRDu4D/+a3TigHmkWaSl9P77dIdwO3N9kFJNgbeOMhgmhoOL9HQcaYSaTCS7E3vt0w/aUrbA69oFumQJK0F73RrGH03yRMGHUKagS4GPgWsbF6fAi4ZaCJphnCNitHnnW4NnSTXAjvSe4DyXnpPcVdVPXGgwaQRl+QMes9QnNKUDgS2qKqXDC6VNDO4RsXo80FKDaPnDTqANEPtMuZBygt8kFKaNptU1WXNdJ2ruEbFCLHp1tBxakBpYL6fZK+quhQgyVPpDTmR1D3XqBhxDi+RJAGQ5DpgF+DmpjQXuI7e+G6HeEkdco2K0WfTLUkCIMl2kx33H3+pO0k2pDc39/b8fo2KqirXqBgRDi+RJAE21dKAnQ3cDVxBb40KjRjvdEuSJA2YM5WMPufpliRJGjzXqBhx3umWJEkaMNeoGH023ZIkSQM20YPMPmsxOmy6JUmSpI45pluSJEnqmE23JEmS1DGbbkkSAEnOTXLooHNI0ihyTLckzUBJjgR2rKqDhiDLicDSqnrPoLNIUle80y1JkiR1zKZbktYBSd6Z5NYky5Ncn2TvJOslOTzJj5P8LMkZSbZszt8+SSU5NMnNSe5K8u7m2H7Au4CXJflVkiub+reTvLrZ/qskFyf5aJK7k/wkydOb+i1J7ugfipJkwyQfab7r9iT/mmTj5tizkixN8vbmumVJXtEcWwi8HHhHk+VL0/nnKknTxaZbkoZckl2ANwJPqarNgH2BJcCbgQOAPwYeBfwC+MSYy58B7ALsDbw3yeOq6mvAPwCnV9WmVfWkCb76qcAPgYcDpwKnAU+ht4DHQcDHk2zanPshYGdg9+b4bOC9fZ/1SOChTf1VwCeSbFFVxwGnAB9usrxgjf5wJGkdYdMtScNvBbAhsGuSDapqSVX9GHgt8O6qWlpV9wJHAi9O8qC+a99XVfdU1ZXAlcBEDfZ4bqyqz1bVCuB0YFvg/VV1b1V9A/gdsGOSAK8B3lZVP6+q5fSa+gV9n3Vfc+19VfVV4Ff0/jMgSTPCg1Z/iiRpkKpqcZK30muqd0vydeCvge2AM5Os7Dt9BbB13/5P+7Z/A2xKe7f3bd/TZBlb2xSYBWwCXN7rv4HeEtbr9537s6q6fy2ySNI6zTvdkrQOqKpTq+oZ9Brtojec4xbgeVX1sL7XRlV1a5uPnMJ4d9FrwHfry/HQqmrbVDuNlqSRZ9MtSUMuyS5JnpNkQ+C39BrcFcC/Akcl2a45b1aS/Vt+7O3A9knW+t+BqloJfBr4aJKtmiyzk+y7BlkevbY5JGmY2XRL0vDbEPggvTvKPwW2ojf7yMeAc4BvJFkOXErv4cc2/q15/1mSK6Yg4zuBxcClSX4JfJP2Y7aPpzde/e4kZ01BFkkaOi6OI0mSJHXMO92SJElSx2y6JUmSpI7ZdEuSJEkds+mWJEmSOmbTLUmSJHXMpluSJEnqmE23JEmS1DGbbkmSJKljNt2SJElSx/4/7iZkAukp30EAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 864x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# VISUALIZING THE DATA\n",
    "cnt_pro = df['sentiment'].value_counts()\n",
    "plt.figure(figsize=(12,4))\n",
    "sns.barplot(cnt_pro.index, cnt_pro.values, alpha=0.8)\n",
    "plt.ylabel('Number of Occurrences', fontsize=12)\n",
    "plt.xlabel('sentiment', fontsize=12)\n",
    "plt.xticks(rotation=90)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "      sentiment                                            Message\n",
      "0             1  Technopolis plans to develop in stages an area...\n",
      "1             2  The international electronic industry company ...\n",
      "2             0  With the new production plant the company woul...\n",
      "3             0  According to the company 's updated strategy f...\n",
      "4             0  FINANCING OF ASPOCOMP 'S GROWTH Aspocomp is ag...\n",
      "...         ...                                                ...\n",
      "4840          2  LONDON MarketWatch -- Share prices ended lower...\n",
      "4841          1  Rinkuskiai 's beer sales fell by 6.5 per cent ...\n",
      "4842          2  Operating profit fell to EUR 35.4 mn from EUR ...\n",
      "4843          2  Net sales of the Paper segment decreased to EU...\n",
      "4844          2  Sales in Finland decreased by 10.5 % in Januar...\n",
      "\n",
      "[4845 rows x 2 columns]\n"
     ]
    }
   ],
   "source": [
    "#Convert sting to numeric\n",
    "sentiment  = {'positive': 0,'neutral': 1,'negative':2} \n",
    "\n",
    "df.sentiment = [sentiment[item] for item in df.sentiment] \n",
    "print(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Finnish Talentum reports its operating profit increased to EUR 20.5 mn in 2005 from EUR 9.3 mn in 2004 , and net sales totaled EUR 103.3 mn , up from EUR 96.4 mn .\n",
      "Message: 0\n"
     ]
    }
   ],
   "source": [
    "def print_message(index):\n",
    "    example = df[df.index == index][['Message', 'sentiment']].values[0]\n",
    "    if len(example) > 0:\n",
    "        print(example[0])\n",
    "        print('Message:', example[1])\n",
    "print_message(12)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Technopolis plans to develop in stages an area of no less than 100,000 square meters in order to host companies working in computer technologies and telecommunications , the statement said .\n",
      "Message: 1\n"
     ]
    }
   ],
   "source": [
    "# Print first message\n",
    "print_message(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# To clean the message\n",
    "from bs4 import BeautifulSoup # it helps to clean the sentence\n",
    "def cleanText(text):\n",
    "    text = BeautifulSoup(text, \"lxml\").text\n",
    "    text = re.sub(r'\\|\\|\\|', r' ', text) \n",
    "    text = re.sub(r'http\\S+', r'<URL>', text)\n",
    "    text = text.lower()\n",
    "    text = text.replace('x', '')\n",
    "    return text\n",
    "df['Message'] = df['Message'].apply(cleanText)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 4845 unique tokens.\n"
     ]
    }
   ],
   "source": [
    "df['Message'] = df['Message'].apply(cleanText)\n",
    "train, test = train_test_split(df, test_size=0.000001 , random_state=42)\n",
    "import nltk # natural language tool kit helps to tocanised the data\n",
    "from nltk.corpus import stopwords\n",
    "def tokenize_text(text):\n",
    "    tokens = []\n",
    "    for sent in nltk.sent_tokenize(text):\n",
    "        for word in nltk.word_tokenize(sent):\n",
    "            #if len(word) < 0:\n",
    "            if len(word) <= 0:\n",
    "                continue\n",
    "            tokens.append(word.lower())\n",
    "    return tokens\n",
    "train_tagged = train.apply(\n",
    "    lambda r: TaggedDocument(words=tokenize_text(r['Message']), tags=[r.sentiment]), axis=1)\n",
    "test_tagged = test.apply(\n",
    "    lambda r: TaggedDocument(words=tokenize_text(r['Message']), tags=[r.sentiment]), axis=1)\n",
    "\n",
    "# The maximum number of words to be used. (most frequent)\n",
    "max_fatures = 500000\n",
    "\n",
    "# Max number of words in each complaint.\n",
    "MAX_SEQUENCE_LENGTH = 50\n",
    "\n",
    "#tokenizer = Tokenizer(num_words=max_fatures, split=' ')\n",
    "tokenizer = Tokenizer(num_words=max_fatures, split=' ', filters='!\"#$%&()*+,-./:;<=>?@[\\]^_`{|}~', lower=True)\n",
    "tokenizer.fit_on_texts(df['Message'].values)\n",
    "X = tokenizer.texts_to_sequences(df['Message'].values)\n",
    "X = pad_sequences(X)\n",
    "print('Found %s unique tokens.' % len(X))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape of data tensor: (4845, 50)\n"
     ]
    }
   ],
   "source": [
    "# make tose words in numirical dots\n",
    "X = tokenizer.texts_to_sequences(df['Message'].values)\n",
    "X = pad_sequences(X, maxlen=MAX_SEQUENCE_LENGTH)\n",
    "print('Shape of data tensor:', X.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([TaggedDocument(words=['upm-kymmene', 'has', 'generated', 'four', 'consecutive', 'quarters', 'of', 'positive', 'free', 'cash', 'flow', '.'], tags=[0]),\n",
       "       TaggedDocument(words=['in', 'the', 'method', 'the', 'smelt', 'spouts', '2', 'are', 'separated', 'from', 'the', 'working', 'area', '6', 'by', 'a', 'shielding', 'wall', '8', ',', '10', 'arranged', 'movable', 'in', 'relation', 'to', 'the', 'smelt', 'spouts', '.'], tags=[1]),\n",
       "       TaggedDocument(words=['``', 'that', \"'s\", 'a', 'very', 'high', 'figure', 'on', 'the', 'european', 'scale', ',', '``', 'noop', 'said', ',', 'recalling', 'however', 'that', 'this', 'also', 'includes', 'beer', 'bought', 'by', 'finnish', 'tourists', '.'], tags=[1]),\n",
       "       ...,\n",
       "       TaggedDocument(words=['proha', 'plc', '(', 'euronet', ':7327', ')', 'announced', 'today', '(', '19', 'may', ')', 'that', 'its', 'fully-owned', 'subsidiary', ',', 'safran', 'software', 'solutions', 'as', ',', 'has', 'agreed', 'to', 'sell', 'its', '49', '%', 'share', 'of', 'safran', 'north', 'america', 'llc', 'to', 'a', 'sna', 'holding', 'as', ',', 'an', 'investment', 'group', 'based', 'in', 'norway', '.'], tags=[1]),\n",
       "       TaggedDocument(words=['the', 'building', 'will', 'house', 'product', 'development', 'and', 'test', 'laboratories', '.'], tags=[1]),\n",
       "       TaggedDocument(words=['(', 'adp', 'news', ')', '-', 'oct', '31', ',', '2008', '-', 'finnish', 'food', 'company', 'raisio', 'oyj', '(', 'om', ':', 'raivv', ')', 'said', 'today', 'that', 'its', 'net', 'profit', 'jumped', 'to', 'eur', '16.4', 'million', '(', 'usd', '20.9', 'm', ')', 'for', 'the', 'first', 'nine', 'months', 'of', '2008', 'from', 'eur', '1.1', 'million', 'for', 'the', 'same', 'period', 'of', '2'], tags=[0])],\n",
       "      dtype=object)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#train_tagged.values[2173]\n",
    "train_tagged.values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████████████████████████████████████████████████████████████████████| 4844/4844 [00:00<00:00, 404169.74it/s]\n"
     ]
    }
   ],
   "source": [
    "# apply D2V model\n",
    "d2v_model = Doc2Vec(dm=1,vector_size=20, dm_mean=1, window=8, min_count=1, workers=1, alpha=0.065, min_alpha=0.065,epochs=5)\n",
    "d2v_model.build_vocab([x for x in tqdm(train_tagged.values)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████████████████████████████████████████████████████████████████████| 4844/4844 [00:00<00:00, 606266.67it/s]\n",
      "100%|██████████████████████████████████████████████████████████████████████████| 4844/4844 [00:00<00:00, 807841.30it/s]\n",
      "100%|█████████████████████████████████████████████████████████████████████████| 4844/4844 [00:00<00:00, 1210366.29it/s]\n",
      "100%|██████████████████████████████████████████████████████████████████████████| 4844/4844 [00:00<00:00, 692687.21it/s]\n",
      "100%|██████████████████████████████████████████████████████████████████████████| 4844/4844 [00:00<00:00, 538518.04it/s]\n",
      "100%|██████████████████████████████████████████████████████████████████████████| 4844/4844 [00:00<00:00, 969471.23it/s]\n",
      "100%|█████████████████████████████████████████████████████████████████████████| 4844/4844 [00:00<00:00, 1616453.86it/s]\n",
      "100%|█████████████████████████████████████████████████████████████████████████| 4844/4844 [00:00<00:00, 1616839.77it/s]\n",
      "100%|██████████████████████████████████████████████████████████████████████████| 4844/4844 [00:00<00:00, 692050.16it/s]\n",
      "100%|█████████████████████████████████████████████████████████████████████████| 4844/4844 [00:00<00:00, 1212099.31it/s]\n",
      "100%|█████████████████████████████████████████████████████████████████████████| 4844/4844 [00:00<00:00, 1616196.69it/s]\n",
      "100%|██████████████████████████████████████████████████████████████████████████| 4844/4844 [00:00<00:00, 970165.63it/s]\n",
      "100%|█████████████████████████████████████████████████████████████████████████| 4844/4844 [00:00<00:00, 1616196.69it/s]\n",
      "100%|█████████████████████████████████████████████████████████████████████████| 4844/4844 [00:00<00:00, 1613757.63it/s]\n",
      "100%|██████████████████████████████████████████████████████████████████████████| 4844/4844 [00:00<00:00, 605724.42it/s]\n",
      "100%|██████████████████████████████████████████████████████████████████████████| 4844/4844 [00:00<00:00, 968593.09it/s]\n",
      "100%|█████████████████████████████████████████████████████████████████████████| 4844/4844 [00:00<00:00, 1215871.25it/s]\n",
      "100%|█████████████████████████████████████████████████████████████████████████| 4844/4844 [00:00<00:00, 1627199.15it/s]\n",
      "100%|█████████████████████████████████████████████████████████████████████████| 4844/4844 [00:00<00:00, 1616068.13it/s]\n",
      "100%|██████████████████████████████████████████████████████████████████████████| 4844/4844 [00:00<00:00, 973326.08it/s]\n",
      "100%|██████████████████████████████████████████████████████████████████████████| 4844/4844 [00:00<00:00, 970582.74it/s]\n",
      "100%|█████████████████████████████████████████████████████████████████████████| 4844/4844 [00:00<00:00, 1614527.06it/s]\n",
      "100%|█████████████████████████████████████████████████████████████████████████| 4844/4844 [00:00<00:00, 1210438.40it/s]\n",
      "100%|█████████████████████████████████████████████████████████████████████████| 4844/4844 [00:00<00:00, 1606865.59it/s]\n",
      "100%|██████████████████████████████████████████████████████████████████████████| 4844/4844 [00:00<00:00, 972440.94it/s]\n",
      "100%|██████████████████████████████████████████████████████████████████████████| 4844/4844 [00:00<00:00, 808034.07it/s]\n",
      "100%|██████████████████████████████████████████████████████████████████████████| 4844/4844 [00:00<00:00, 965555.01it/s]\n",
      "100%|█████████████████████████████████████████████████████████████████████████| 4844/4844 [00:00<00:00, 1626417.59it/s]\n",
      "100%|██████████████████████████████████████████████████████████████████████████| 4844/4844 [00:00<00:00, 694249.40it/s]\n",
      "100%|██████████████████████████████████████████████████████████████████████████| 4844/4844 [00:00<00:00, 972953.19it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 15.7 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "for epoch in range(30):\n",
    "    d2v_model.train(utils.shuffle([x for x in tqdm(train_tagged.values)]), total_examples=len(train_tagged.values), epochs=1)\n",
    "    d2v_model.alpha -= 0.002\n",
    "    d2v_model.min_alpha = d2v_model.alpha"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Doc2Vec<dm/m,d20,n5,w8,s0.001>\n"
     ]
    }
   ],
   "source": [
    "print(d2v_model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "11682"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(d2v_model.wv)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "<ipython-input-18-95e16a1341c7>:4: DeprecationWarning: Call to deprecated `docvecs` (The `docvecs` property has been renamed `dv`.).\n",
      "  for i, vec in enumerate(d2v_model.docvecs.vectors):\n"
     ]
    }
   ],
   "source": [
    "# save the vectors in a new matrix\n",
    "embedding_matrix = np.zeros((len(d2v_model.wv)+ 1, 20))\n",
    "\n",
    "for i, vec in enumerate(d2v_model.docvecs.vectors):\n",
    "    while i in vec <= 1000:\n",
    "    #print(i)\n",
    "    #print(model.docvecs)\n",
    "          embedding_matrix[i]=vec\n",
    "    #print(vec)\n",
    "    #print(vec[i])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('loss', 0.9108386039733887),\n",
       " ('sales', 0.7973217964172363),\n",
       " ('profits', 0.7576896548271179),\n",
       " ('turnover', 0.746253490447998),\n",
       " ('income', 0.7415260076522827),\n",
       " ('pharma', 0.692173182964325),\n",
       " ('larger-than-epected', 0.6914839148521423),\n",
       " ('ns1v', 0.6873822808265686),\n",
       " ('originally', 0.6847700476646423),\n",
       " ('bill', 0.680260419845581)]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Top 10 with profit word\n",
    "d2v_model.wv.most_similar(positive=['profit'], topn=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('appeal', 0.7885555624961853),\n",
       " ('property', 0.7696755528450012),\n",
       " ('karhinen', 0.7405270338058472),\n",
       " ('estate', 0.7381441593170166),\n",
       " ('organization', 0.7297702431678772),\n",
       " ('equity', 0.7217682003974915),\n",
       " ('warnings', 0.7113656997680664),\n",
       " ('rohwedder', 0.709439754486084),\n",
       " ('non-restricted', 0.7023128867149353),\n",
       " ('detailing', 0.7015187740325928)]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Top 10 with investment word\n",
    "d2v_model.wv.most_similar(positive=['investment'], topn=10)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('mold', 0.9611244797706604),\n",
       " ('gather', 0.8924813270568848),\n",
       " ('macro-economic', 0.8824651837348938),\n",
       " ('mursula', 0.8562044501304626),\n",
       " ('biscuits', 0.8543246388435364),\n",
       " ('aptly', 0.8493544459342957),\n",
       " ('else', 0.83978670835495),\n",
       " ('breakfast', 0.8378297686576843),\n",
       " ('umbrellas', 0.8366750478744507),\n",
       " ('campbell', 0.8338345289230347)]"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Top 10 with broke word\n",
    "d2v_model.wv.most_similar(positive=['broke'], topn=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Plot Similarity word in Doc2vec\n",
    "from sklearn.manifold import TSNE\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "def tsne_plot(model):\n",
    "    \"Creates and TSNE model and plots it\"\n",
    "    labels = []\n",
    "    tokens = []\n",
    "\n",
    "    for word in d2v_model.wv.vocab:\n",
    "        tokens.append(model[word])\n",
    "        labels.append(word)\n",
    "    \n",
    "    tsne_model = TSNE(perplexity=40, n_components=2, init='pca', n_iter=250, random_state=23)\n",
    "    new_values = tsne_model.fit_transform(tokens)\n",
    "\n",
    "    x = []\n",
    "    y = []\n",
    "    for value in new_values:\n",
    "        x.append(value[0])\n",
    "        y.append(value[1])\n",
    "        \n",
    "    plt.figure(figsize=(16, 16)) \n",
    "    for i in range(len(x)):\n",
    "        plt.scatter(x[i],y[i])\n",
    "        plt.annotate(labels[i],\n",
    "                     xy=(x[i], y[i]),\n",
    "                     xytext=(5, 2),\n",
    "                     textcoords='offset points',\n",
    "                     ha='right',\n",
    "                     va='bottom')\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Create the LSTM Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding (Embedding)        (None, 50, 20)            233660    \n",
      "_________________________________________________________________\n",
      "lstm (LSTM)                  (None, 50)                14200     \n",
      "_________________________________________________________________\n",
      "dense (Dense)                (None, 3)                 153       \n",
      "=================================================================\n",
      "Total params: 248,013\n",
      "Trainable params: 248,013\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "from keras.models import Sequential\n",
    "from keras.layers import LSTM, Dense, Embedding\n",
    "\n",
    "\n",
    "# init layer\n",
    "model = Sequential()\n",
    "\n",
    "# emmbed word vectors\n",
    "model.add(Embedding(len(d2v_model.wv)+1,20,input_length=X.shape[1],weights=[embedding_matrix],trainable=True))\n",
    "\n",
    "# learn the correlations\n",
    "def split_input(sequence):\n",
    "     return sequence[:-1], tf.reshape(sequence[1:], (-1,1))\n",
    "model.add(LSTM(50,return_sequences=False))\n",
    "model.add(Dense(3,activation=\"softmax\"))\n",
    "\n",
    "# output model skeleton\n",
    "model.summary()\n",
    "model.compile(optimizer=\"adam\",loss=\"binary_crossentropy\",metrics=['acc'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "pip install pydot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('Failed to import pydot. You must `pip install pydot` and install graphviz (https://graphviz.gitlab.io/download/), ', 'for `pydotprint` to work.')\n"
     ]
    }
   ],
   "source": [
    "from keras.utils import plot_model\n",
    "plot_model(model, to_file='model.png')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "pip install graphviz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(4118, 50) (4118, 3)\n",
      "(727, 50) (727, 3)\n"
     ]
    }
   ],
   "source": [
    "# to Train the model split the datapoints\n",
    "Y = pd.get_dummies(df['sentiment']).values\n",
    "X_train, X_test, Y_train, Y_test = train_test_split(X,Y, test_size = 0.15, random_state = 42)\n",
    "print(X_train.shape,Y_train.shape)\n",
    "print(X_test.shape,Y_test.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/50\n",
      "129/129 - 10s - loss: 0.5590 - acc: 0.5954\n",
      "Epoch 2/50\n",
      "129/129 - 3s - loss: 0.4324 - acc: 0.6858\n",
      "Epoch 3/50\n",
      "129/129 - 3s - loss: 0.3113 - acc: 0.7817\n",
      "Epoch 4/50\n",
      "129/129 - 3s - loss: 0.2439 - acc: 0.8278\n",
      "Epoch 5/50\n",
      "129/129 - 3s - loss: 0.1887 - acc: 0.8749\n",
      "Epoch 6/50\n",
      "129/129 - 4s - loss: 0.1404 - acc: 0.9271\n",
      "Epoch 7/50\n",
      "129/129 - 4s - loss: 0.0899 - acc: 0.9592\n",
      "Epoch 8/50\n",
      "129/129 - 4s - loss: 0.0560 - acc: 0.9784\n",
      "Epoch 9/50\n",
      "129/129 - 4s - loss: 0.0407 - acc: 0.9828\n",
      "Epoch 10/50\n",
      "129/129 - 4s - loss: 0.0304 - acc: 0.9866\n",
      "Epoch 11/50\n",
      "129/129 - 4s - loss: 0.0276 - acc: 0.9879\n",
      "Epoch 12/50\n",
      "129/129 - 3s - loss: 0.0244 - acc: 0.9900\n",
      "Epoch 13/50\n",
      "129/129 - 3s - loss: 0.0152 - acc: 0.9947\n",
      "Epoch 14/50\n",
      "129/129 - 3s - loss: 0.0324 - acc: 0.9857\n",
      "Epoch 15/50\n",
      "129/129 - 3s - loss: 0.0218 - acc: 0.9913\n",
      "Epoch 16/50\n",
      "129/129 - 4s - loss: 0.0123 - acc: 0.9956\n",
      "Epoch 17/50\n",
      "129/129 - 3s - loss: 0.0138 - acc: 0.9942\n",
      "Epoch 18/50\n",
      "129/129 - 3s - loss: 0.0094 - acc: 0.9976\n",
      "Epoch 19/50\n",
      "129/129 - 3s - loss: 0.0118 - acc: 0.9959\n",
      "Epoch 20/50\n",
      "129/129 - 3s - loss: 0.0107 - acc: 0.9964\n",
      "Epoch 21/50\n",
      "129/129 - 3s - loss: 0.0072 - acc: 0.9976\n",
      "Epoch 22/50\n",
      "129/129 - 3s - loss: 0.0100 - acc: 0.9968\n",
      "Epoch 23/50\n",
      "129/129 - 3s - loss: 0.0059 - acc: 0.9978\n",
      "Epoch 24/50\n",
      "129/129 - 3s - loss: 0.0074 - acc: 0.9976\n",
      "Epoch 25/50\n",
      "129/129 - 3s - loss: 0.0047 - acc: 0.9981\n",
      "Epoch 26/50\n",
      "129/129 - 3s - loss: 0.0158 - acc: 0.9932\n",
      "Epoch 27/50\n",
      "129/129 - 3s - loss: 0.0093 - acc: 0.9971\n",
      "Epoch 28/50\n",
      "129/129 - 3s - loss: 0.0126 - acc: 0.9942\n",
      "Epoch 29/50\n",
      "129/129 - 3s - loss: 0.0101 - acc: 0.9947\n",
      "Epoch 30/50\n",
      "129/129 - 4s - loss: 0.0074 - acc: 0.9971\n",
      "Epoch 31/50\n",
      "129/129 - 3s - loss: 0.0040 - acc: 0.9981\n",
      "Epoch 32/50\n",
      "129/129 - 4s - loss: 0.0043 - acc: 0.9976\n",
      "Epoch 33/50\n",
      "129/129 - 4s - loss: 0.0043 - acc: 0.9978\n",
      "Epoch 34/50\n",
      "129/129 - 4s - loss: 0.0063 - acc: 0.9976\n",
      "Epoch 35/50\n",
      "129/129 - 3s - loss: 0.0094 - acc: 0.9971\n",
      "Epoch 36/50\n",
      "129/129 - 3s - loss: 0.0049 - acc: 0.9985\n",
      "Epoch 37/50\n",
      "129/129 - 3s - loss: 0.0071 - acc: 0.9971\n",
      "Epoch 38/50\n",
      "129/129 - 3s - loss: 0.0028 - acc: 0.9988\n",
      "Epoch 39/50\n",
      "129/129 - 3s - loss: 0.0027 - acc: 0.9985\n",
      "Epoch 40/50\n",
      "129/129 - 3s - loss: 0.0028 - acc: 0.9978\n",
      "Epoch 41/50\n",
      "129/129 - 3s - loss: 0.0022 - acc: 0.9990\n",
      "Epoch 42/50\n",
      "129/129 - 3s - loss: 0.0031 - acc: 0.9983\n",
      "Epoch 43/50\n",
      "129/129 - 3s - loss: 0.0030 - acc: 0.9985\n",
      "Epoch 44/50\n",
      "129/129 - 3s - loss: 0.0016 - acc: 0.9993\n",
      "Epoch 45/50\n",
      "129/129 - 3s - loss: 0.0097 - acc: 0.9956\n",
      "Epoch 46/50\n",
      "129/129 - 3s - loss: 0.0070 - acc: 0.9964\n",
      "Epoch 47/50\n",
      "129/129 - 3s - loss: 0.0099 - acc: 0.9956\n",
      "Epoch 48/50\n",
      "129/129 - 3s - loss: 0.0037 - acc: 0.9983\n",
      "Epoch 49/50\n",
      "129/129 - 4s - loss: 0.0309 - acc: 0.9888\n",
      "Epoch 50/50\n",
      "129/129 - 3s - loss: 0.0089 - acc: 0.9976\n"
     ]
    }
   ],
   "source": [
    "# Set the batch size\n",
    "\n",
    "batch_size = 32 # 32 batches are made\n",
    "# Train those 32 batches separetly and check there accurecy\n",
    "history=model.fit(X_train, Y_train, epochs =50, batch_size=batch_size, verbose = 2) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEWCAYAAAB8LwAVAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAtSklEQVR4nO3deZhddZ3v+/ends2VSipDJWQkAUJImAKEQIMgiGIQEOHaCrZ26+2+XFpx6Ok07elz9PRwnj6PtrePLd7ItbnqVaA9agTtNKC0TDJkgEAGCAmZqkhIKlUZKjUP3/vHXhU2xa6kQmrXrqr9eT1Pntp7DXt9VyVZn/1bv7V+SxGBmZlZf0X5LsDMzEYmB4SZmWXlgDAzs6wcEGZmlpUDwszMsnJAmJlZVg4IM0DS9yT93SCX3SHp/bmuySzfHBBmZpaVA8JsDJFUnO8abOxwQNiokZza+QtJL0tqkfQvkqZJ+ndJzZJ+LWlixvIflrRR0kFJj0tamDHvAkkvJOv9K1Deb1s3SFqXrPuMpPMGWeP1kl6UdFhSnaSv9pv/nuTzDibzP51Mr5D0j5J2Sjok6elk2lWS6rP8Ht6fvP6qpJ9I+qGkw8CnJS2V9GyyjT2SviWpNGP9syX9SlKTpL2SvizpFEmtkiZnLHeRpAZJJYPZdxt7HBA22vxvwAeAM4EbgX8HvgxMIf3v+QsAks4E7ge+BNQCK4FfSCpNDpY/B/4/YBLwv5LPJVn3QuBe4P8EJgPfAR6SVDaI+lqA3wdqgOuBP5b0keRz5yT1/nNS02JgXbLe14GLgMuSmv4T0DvI38lNwE+Sbf4I6AH+hPTv5HeAa4DPJjVUA78GHgZmAGcAj0XEm8DjwMcyPveTwAMR0TXIOmyMcUDYaPPPEbE3It4AngKej4gXI6IDWAFckCz3ceDfIuJXyQHu60AF6QPwpUAJ8E8R0RURPwFWZ2zj/wC+ExHPR0RPRHwf6EjWO6aIeDwi1kdEb0S8TDqk3pvM/j3g1xFxf7LdxohYJ6kI+N+BL0bEG8k2n0n2aTCejYifJ9tsi4i1EfFcRHRHxA7SAddXww3AmxHxjxHRHhHNEfF8Mu/7pEMBSSngNtIhagXKAWGjzd6M121Z3o9LXs8AdvbNiIheoA6Ymcx7I94+UuXOjNenAn+WnKI5KOkgMDtZ75gkXSLpN8mpmUPAHaS/yZN8xutZVptC+hRXtnmDUdevhjMl/VLSm8lpp/8+iBoAHgQWSTqNdCvtUESsepc12RjggLCxajfpAz0AkkT64PgGsAeYmUzrMyfjdR3w9xFRk/GnMiLuH8R27wMeAmZHxARgOdC3nTrg9Czr7AfaB5jXAlRm7EeK9OmpTP2HZP6/gVeB+RExnvQpuOPVQES0Az8m3dL5FG49FDwHhI1VPwaul3RN0sn6Z6RPEz0DPAt0A1+QVCzpFmBpxrr/D3BH0hqQpKqk87l6ENutBpoiol3SUuATGfN+BLxf0seS7U6WtDhp3dwLfEPSDEkpSb+T9Hm8BpQn2y8B/ho4Xl9INXAYOCLpLOCPM+b9EjhF0pcklUmqlnRJxvwfAJ8GPgz8cBD7a2OYA8LGpIjYTPp8+j+T/oZ+I3BjRHRGRCdwC+kD4QHS/RU/y1h3Del+iG8l87cmyw7GZ4G/kdQM/FfSQdX3ubuAD5EOqybSHdTnJ7P/HFhPui+kCfgfQFFEHEo+87ukWz8twNuuasriz0kHUzPpsPvXjBqaSZ8+uhF4E9gCXJ0x/7ekO8dfSPovrIDJDwwys0yS/gO4LyK+m+9aLL8cEGZ2lKSLgV+R7kNpznc9ll8+xWRmAEj6Pul7JL7kcDBwC8LMzAbgFoSZmWU1pgb2mjJlSsydOzffZZiZjRpr167dHxH9760BxlhAzJ07lzVr1uS7DDOzUUPSzoHm+RSTmZll5YAwM7OsHBBmZpbVmOqDyKarq4v6+nra29vzXUpOlZeXM2vWLEpK/GwXMxsaYz4g6uvrqa6uZu7cubx98M6xIyJobGykvr6eefPm5bscMxsjcnaKSdK9kvZJ2jDAfEn6pqStSj9C8sKMecskbU7m3XUydbS3tzN58uQxGw4Akpg8efKYbyWZ2fDKZR/E94Blx5h/HTA/+XM76THs+8a7vzuZvwi4TdKikylkLIdDn0LYRzMbXjk7xRQRT0qae4xFbgJ+kDzV6zlJNZKmA3OBrRGxDUDSA8mym3JVq5nlTkRwoLWLtq4eplaXUZI69vfS7p5e9hxqp/5AGxFBcaqI4pQoKUp+pkRlaTHjyosZV1pMUdE7vxy1d/XQ0NzBvuZ2Gpo7ONLRw7iyFNXlJYwrK6a6vJjq8hKqylL09AbdPUFXb+/R1509vbR29NDc3sXh9m6OdHTT3N7FkfZuZk6s4Moza5kybjCPKH93v6+R8oUvn30QM3n7oxLrk2nZpmc+0ORtJN1OugXCnDlzBlosbw4ePMh9993HZz/72RNa70Mf+hD33XcfNTU1uSlslOjtDTbuPszTW/ez+c3DTJtQzpxJlZw6qYo5kyqZUVNOcb8DTkTQ0d1LW2cPTa2dyYGig32H22k40kHD4Q66e4Nx5cmBoix9sKguLyZVJA62dnGgtZODrV00tXRyoLWTw+3dFImjB6niVBHFRaK4SHT2pLfV3t1Le2cPbV09tHf1sOCUan53yWyuXTSN8pLUgPvY0xu8uOsA2xpaOGfmBBacUk0qy0HveLp6etm67wibdh9m98E2mvrtw4HWTjq6eikvSVFRkqK8NEV5cREVpSlKU0X09AZdvUF3T+/RA2Zvb1CU7Gfx0QN0et/LSlJUlBQd/ayK5HNbOnvYc7CN3Yfa2HOwnd2H2mjv6gWgSDC1upzpNeXMmFDBjJpyaipL2X2wjV1NrexqauWNA2109w5+jLi3DvjFdPcGDYc7aO7oPuHf34mQ4LyZE3jvgqlcvaCW82bVvOPvLCJo7+qls7uXqrLUO/6dQvrf97b9R1hXd4iX6w/yUt1BNu9tZtnZp/B3N5/LuLJjH6Ijgh+vqWP1jgN87aPnDXmw5DMgsu1JHGN6VhFxD3APwJIlS0bcyIMHDx7k29/+9jsCoqenh1Rq4IPGypUrc13aCWvp6KbxSCdzJlcef+EBdPf08ty2Jv5t/W4OtnYxo6aC6RPKmVFTkf4zoZz2rl5++/p+nt6yn2de38+B1i4Apk8oZ/+RDrp63vprThWJU8aXA9DW1ZMcqHsYaAzK0lQRtdVllKTEkY5uDrd309ndm3XZ8eXFTKoqpaaylAkVJUQEXT3p//CtnT1096YPpKXFRZSXpKipKKFifDnlJUWUpIp45vVGvnD/i4wvL+YjF8zkY0tmc/aM8UiitbObp7bs59eb9vIfr+6jsaXz6Hary4pZPKeGJadOYsnciZw3awKS0vuWhE9bVw9HOrqPBsLG3YfZvLf5bfsyrqyYiVUlTKwsZWJlKafXjqOsuCj5jN7076urhwMtnXR091KcEqmiIkqKRHFKjCtJB2bft+ru3l7auuLofveFcF8gdiTblmBqdRkzaipYOH087ztrKjNqKqgoTbHnUDu7D7ax51Abr+w5zK9f2UtHdy8TK0uYM6mSc2dO4IbzpjNnUiWzJlaSKtLRsOruSYdXXyA3t3fTnHyzb25P/0wViSvn11JbXUZtdRlTk5/VZSVvtQI6uo8u39LZQ0o6GvolRSJVlA7BqiR4xpUVMz75AlFVVszmN5t5fPM+frN5H9/6jy1887EtTKwsYf7U6vRnd6RbGs3t3W8LuoqS1FtfSspLKCkSm99sPhpmVaUpzp01gQ+dO52fv/gGL9Uf4lufuICzZ0zI+u9zX3M7f/XT9Tz26j4uPW0SbV09VJYO7SE9nwFRT/oZwX1mkX6OcOkA00elu+66i9dff53FixdTUlLCuHHjmD59OuvWrWPTpk185CMfoa6ujvb2dr74xS9y++23A28NG3LkyBGuu+463vOe9/DMM88wc+ZMHnzwQSoqKoal/p7e4Ldb97PixTd4ZOObtHb28IlL5vDX1y8c9D/G3t7ghV0H+MVLu/m39XvYf6STqtIU0yaU8/jmBtq6erKuN218Ge87axpXzJ/CZWdMZmp1OT29wZuH29nV2Epd3zfOg22kipT+Jpvxjba8OMWkqtK3HSgmVJS841tWR3dPxn/oXmoqS6mpKMn6je9E9PYGz7zeyP9aW8cDq+v4wbM7WTh9PNMnlPPbrfvp6O6luryY9501lfcvnMbC6dWsf+MQa3YcYO3OA/zTY68NGHSZaipLOHvGeD592VzOnjGes2eMZ/akSsqKB/4Ckgs9vUF7Vw+lxUXHPY3Up+9bdkXp8NZ6Ms6dNYFzZ03g89fM50BLJ09uaeCJzQ3UH2xjRk051eXVR1s148qLKU0V0drZczScDrd3c6S9m/auHm66YAbnz6ph8ewaTqsdd7QV8vEls/nCAy9y87ef4b/esIjfu2TO2/7dPrxhD3/1s/W0dPbwX25YxGcum5v1VNvJyulw30kfxC8j4pws864H7iT9CMZLgG9GxFJJxaSfw3sN6UcsrgY+EREbj7e9JUuWRP+xmF555RUWLlwIwH/7xUY27T58UvvU36IZ4/nKjWcPOH/Hjh3ccMMNbNiwgccff5zrr7+eDRs2HL0ctampiUmTJtHW1sbFF1/ME088weTJk98WEGeccQZr1qxh8eLFfOxjH+PDH/4wn/zkJ9+xrcx9PVmv7DnMihff4OcvvsG+5g6qy4u54bzplBWn+P6zO5g3pYr/+fELOHdW9m83APUHWvnhc7v4xUu7eeNgG2XFRVyzcCo3njeDq8+aSnlJiojgYGvX205HAFx2+mROrx03Ys7FnqxDrV089PJufrKmjoNtXVy9YCrXLprGxfMmDXgwPdTWxYu7DvDKnmaKi3T0NE5fCFaUpJg7pYrpE8rHzO/J3tJ4pIM//fFLPPFaA9efN51/uOVcAvjqgxv52YtvcO7MCXzjY+czf9pgHpU+MElrI2JJtnk5a0FIuh+4CpgiqR74ClACEBHLgZWkw2Er0Ap8JpnXLelO4BEgBdw7mHAYLZYuXfq2exW++c1vsmLFCgDq6urYsmULkydPfts68+bNY/HixQBcdNFF7NixY8jqae3sZsveI2ze28zmN5t5Lfm5r7mD4iJx1YKp3HLhTN6XHNABrl00jT/98Uvc/O3f8mfXLuD2K087+s0nIt1auPfpHTy88U0A3ntmLX/xwQW8f9G0d5xTlcTEqlImVpUO2JQeCyZUlvCpS0/lU5eeOvh1Kkq4asFUrlowNYeV2Ug1eVwZ/++nL+Y7T27j649uZsMbh+jq7mVvcwdfuGY+n3/fGYNuqb1bubyK6bbjzA/gcwPMW0k6QIbUsb7pD5eqqqqjrx9//HF+/etf8+yzz1JZWclVV12V9V6GsrK3rpZIpVK0tbUNenttnT08v72R+gNt7GvuoKG5g4bm9qTTtoM3D7+1vbLiIs6cVs0V82tZPKeG68+dzqSq0nd85mVnTOHhL13Bf16xgf/x8Ks8vnkfX/vo+ayrP8i/PL2dl+oOMr68mD+6Yh5/8DtzmVEzPKfDzMaaoiLxx1edzpK5E/nC/S9SUZLip398GYtn1wzL9sf8ndT5Vl1dTXNz9qc3Hjp0iIkTJ1JZWcmrr77Kc889NyTbbGju4Dev7uPRTXt5emvD0StIJJhcVUptdTm11WWcOa2a2RMrWXBKNQtOqWbOpMpBXz1TU1nKtz5xAVe/MJWvPLiBK7/2GwDmTanib286m1sunEXVca7AMLPBuXjuJJ74i6tJJZ3ow8X/g3Ns8uTJXH755ZxzzjlUVFQwbdq0o/OWLVvG8uXLOe+881iwYAGXXnrpu95OR3e6E+yWb/+WF+sOEgEzayr4+JLZXLNwGgtOqWZyVelJd7xmksRHL5rFxXMn8qPnd3HJvElcvWBqTjrLzApdafHwj606pp5JfbxO6rGmq7uXg21dHGrrorWzm727tvHNta28f+E0PrAofVWMOy/N7Fjy0kltudHbG+mbuNq6aEmun64oSaWvZJlQxi8+f+FxPsHMbHAcEKNM/YE2DrZ1UlacYtr4ciZUlBy9umh/kR/vYWZDpyACYiSNbXIyDrV1cbCtk2njy5laXfa2fRpLpwrNbGQY8185y8vLaWxsHPUH0O6eXt440EZFSYraLOHQ2NhIeXl5His0s7FmzLcgZs2aRX19PQ0NDfku5aQ0tXTS1tnD1PFlbG56Z673PVHOzGyojPmAKCkpGfVPWXvslb384U/W8IVr5vOnl52Z73LMrECM+VNMo92h1i6+vGI9Z51SzZ1Xn5HvcsysgIz5FsRo97f/ton9Rzr5lz+4OC83yphZ4fIRZwT7zeZ9/GRtPXe89zTOmTl2B7Izs5HJATFCHW7v4ss/W8/8qeP4wjXz812OmRUgB8QI9Y1HX2Pv4Xa+/rvnD/uDX8zMwAExIh3p6ObHa+q45cJZnD9Mw/qamfWX04CQtEzSZklbJd2VZf5ESSskvSxplaRzMubtkLRe0jpJa/qvO5Y9tG43rZ09/N4lc/JdipkVsFw+US4F3A18gPTzp1dLeigiNmUs9mVgXUTcLOmsZPlrMuZfHRH7c1XjSHX/ql2cdUr1sD0UxMwsm1y2IJYCWyNiW0R0Ag8AN/VbZhHwGEBEvArMlTSNArbhjUOsf+MQt148e0yMH2Vmo1cuA2ImUJfxvj6Zlukl4BYASUuBU4G+8SICeFTSWkm3D7QRSbdLWiNpzWgfTgPggdW7KCsu4uYLPGyGmeVXLgMi29ff/iPm/QMwUdI64PPAi0B3Mu/yiLgQuA74nKQrs20kIu6JiCURsaS2tnZoKs+T1s5ufv7ibq4/dzoTKkvyXY6ZFbhc3kldD8zOeD8L2J25QEQcBj4DoPT5lO3JHyJid/Jzn6QVpE9ZPZnDevPuly/v4UhHN7cudee0meVfLlsQq4H5kuZJKgVuBR7KXEBSTTIP4I+AJyPisKQqSdXJMlXAtcCGHNY6Ijywahen11Zx8dyJ+S7FzCx3LYiI6JZ0J/AIkALujYiNku5I5i8HFgI/kNQDbAL+MFl9GrAi6aQtBu6LiIdzVetIsPnNZl7YdZC/vn6hO6fNbETI6WB9EbESWNlv2vKM188C7xhHIiK2AefnsraR5v5VuyhNFXHLhe6cNrORwXdSjwDtXT2sePENPnjOKUyqKj3+CmZmw8ABMQL8+4Y9HGrr4raLZx9/YTOzYeKAGAHuX1XHqZMrufS0yfkuxczsKAdEnr3ecIRV25u49eI5FBW5c9rMRg4HRJ49sGoXxUXioxe5c9rMRhYHRB5FBD9ft5trFk6ltros3+WYmb2NAyKPXm9ooaG5g/edNTXfpZiZvYMDIo9WbW8CYOk8d06b2cjjgMijVdsbqa0uY+7kynyXYmb2Dg6IPIkInt/exNJ5kzy0hpmNSA6IPKk/0MaeQ+1cMm9SvksxM8vKAZEnzyf9D5e4/8HMRigHRJ6s2t5ITWUJ86eOy3cpZmZZOSDyZNX2Ji6eO8l3T5vZiOWAyIO9h9vZ0djq/gczG9EcEHnw/NH7HxwQZjZy5TQgJC2TtFnSVkl3ZZk/UdIKSS9LWiXpnMGuO5qt2t7IuLJiFk0fn+9SzMwGlLOAkJQC7gauAxYBt0la1G+xLwPrIuI84PeB/3kC645aq7Y3cdGpEylOuQFnZiNXLo9QS4GtEbEtIjqBB4Cb+i2zCHgMICJeBeZKmjbIdUelppZOXtt7xKeXzGzEy2VAzATqMt7XJ9MyvQTcAiBpKXAqMGuQ65Ksd7ukNZLWNDQ0DFHpubN6R9/9Dw4IMxvZchkQ2a7fjH7v/wGYKGkd8HngRaB7kOumJ0bcExFLImJJbW3tSZQ7PJ7f1kRZcRHnzpqQ71LMzI6pOIefXQ9kPmR5FrA7c4GIOAx8BkDpAYm2J38qj7fuaLVqRyMXzplIWXEq36WYmR1TLlsQq4H5kuZJKgVuBR7KXEBSTTIP4I+AJ5PQOO66o9Hh9i427T7s/gczGxVy1oKIiG5JdwKPACng3ojYKOmOZP5yYCHwA0k9wCbgD4+1bq5qHS5rdx6gN9z/YGajQy5PMRERK4GV/aYtz3j9LDB/sOuOdqu2N1FcJC6YMzHfpZiZHZcvxB9Gz29r5LxZE6godf+DmY18Dohh0tbZw8v1h7jkNA/vbWajgwNimLy46wDdveEOajMbNRwQw+T57U0UCS461f0PZjY6OCCGyartTSyaMZ7x5SX5LsXMbFAcEMOgs7uXF3YdYOlc9z+Y2ejhgBgGL+w6QEd3L5ee5v4HMxs9HBDD4Okt+0kViUtPdwvCzEYPB8QweGrrfs6fNcH9D2Y2qjggcuxQaxfr6w/ynvkjf6RZM7NMDogce+b1/fQGXDF/Sr5LMTM7IQ6IHHtq637GlRWzeHZNvksxMzshDogce3rLfi49bRIlfv60mY0yPmrl0K7GVnY1tfKeM3x6ycxGHwdEDj21Nf2MbHdQm9lolNOAkLRM0mZJWyXdlWX+BEm/kPSSpI2SPpMxb4ek9ZLWSVqTyzpz5ekt+5k+oZzTa6vyXYqZ2QnL2QODJKWAu4EPkH4+9WpJD0XEpozFPgdsiogbJdUCmyX9KCI6k/lXR8T+XNWYSz29wTOvN3LtommkH7dtZja65LIFsRTYGhHbkgP+A8BN/ZYJoFrpI+g4oAnozmFNw2b9G4c41NbFe3x5q5mNUrkMiJlAXcb7+mRapm+Rfi71bmA98MWI6E3mBfCopLWSbh9oI5Jul7RG0pqGhoahq/4kPb0lXcvl7qA2s1EqlwGR7bxK9Hv/QWAdMANYDHxL0vhk3uURcSFwHfA5SVdm20hE3BMRSyJiSW3tyOkMfmrLfhZNH8+UcWX5LsXM7F3JZUDUA7Mz3s8i3VLI9BngZ5G2FdgOnAUQEbuTn/uAFaRPWY0KLR3dvLDrgO+eNrNRLZcBsRqYL2mepFLgVuChfsvsAq4BkDQNWABsk1QlqTqZXgVcC2zIYa1DatX2Jrp6wv0PZjaq5ewqpojolnQn8AiQAu6NiI2S7kjmLwf+FviepPWkT0n9ZUTsl3QasCK5+qcYuC8iHs5VrUPtqS37KS0u4uK5fv6DmY1eOQsIgIhYCazsN215xuvdpFsH/dfbBpyfy9py6emtDSydO4nyklS+SzEze9d8J/UQ23u4ndf2HvHpJTMb9RwQQ+zpLen7+jz+kpmNdg6IIfb01v1Mripl0fTxx1/YzGwEc0AMoYjg6a37ueyMKRQVeXgNMxvdHBBDaPPeZhqaO7jCp5fMbAxwQAyho/0P7qA2szHAATGEnnitgTOmjmNGTUW+SzEzO2kOiCHS3tXDqu1NHl7DzMYMB8QQWbW9iY7uXq700+PMbIxwQAyRp7Y0UJoq4pLTPLyGmY0NDogh8tSW/SyZO5HK0pyOXmJmNmwcEENg7+F2Xn2zmSvP9OklMxs7HBBD4Knk8lZ3UJvZWOKAGAJPvtbAlHFlLDzFw2uY2dgxqICQdLOkCRnvayR9JGdVjSK9venhNa6Y7+E1zGxsGWwL4isRcajvTUQcBL6Sk4pGmU17DtPU0unTS2Y25gw2ILItd9zLdSQtk7RZ0lZJd2WZP0HSLyS9JGmjpM8Mdt2R4onXGgAPr2FmY89gA2KNpG9IOl3SaZL+L2DtsVaQlALuBq4DFgG3SVrUb7HPAZsi4nzgKuAfJZUOct0R4aktDSycPp6p1eX5LsXMbEgNNiA+D3QC/wr8GGgjfXA/lqXA1ojYFhGdwAPATf2WCaBa6YdPjwOagO5Brpt3LR3drN15gCvPdOvBzMaeQd3VFREtwIme5pkJ1GW8rwcu6bfMt4CHgN1ANfDxiOiVNJh1AZB0O3A7wJw5c06wxJPz3LZGunrCw2uY2Zg02KuYfiWpJuP9REmPHG+1LNOi3/sPAuuAGcBi4FuSxg9y3fTEiHsiYklELKmtHd4D9ZOvNVBeUsSSuROHdbtmZsNhsKeYpiRXLgEQEQeAqcdZpx6YnfF+FumWQqbPAD+LtK3AduCsQa6bd09t2c+lp02mrDiV71LMzIbcYAOiV9LR8zeS5jLAN/oMq4H5kuZJKgVuJX06KdMu4JrkM6cBC4Btg1w3r+qaWtm2v4UrfHrJzMaowY4s95+BpyU9kby/kuS8/0AiolvSncAjQAq4NyI2Srojmb8c+Fvge5LWkz6t9JcRsR8g27ontmu51Te8xnvdQW1mY9RgO6kflrSEdCisAx4kfSXT8dZbCazsN215xuvdwLWDXXckeWpLA9MnlHN67bh8l2JmlhODCghJfwR8kXRfwDrgUuBZ4H05q2wE6+7p5emt+/nQOdNJX6FrZjb2DLYP4ovAxcDOiLgauABoyFlVI9xL9Ydobu/mCp9eMrMxbLAB0R4R7QCSyiLiVdIdygXpma37keA9ZzggzGzsGmwndX1yH8TPgV9JOsAIvOx0uLzecISZNRXUVJbmuxQzs5wZbCf1zcnLr0r6DTABeDhnVY1wu5pamTOpMt9lmJnl1Ak/QDkinjj+UmPbrqY23r/wePcJmpmNbn6i3Alq7exm/5EOZrsFYWZjnAPiBNU1pW//cECY2VjngDhBu5paAdwHYWZjngPiBDkgzKxQOCBOUF1TK+PKiplYWZLvUszMcsoBcYLqmlqZPanSQ2yY2ZjngDhB6XsgKvJdhplZzjkgTkBE+CY5MysYDogT0NDcQUd3rwPCzApCTgNC0jJJmyVtlXRXlvl/IWld8meDpB5Jk5J5OyStT+atyWWdg9V3BdMsB4SZFYATHmpjsCSlgLuBD5B+xvRqSQ9FxKa+ZSLia8DXkuVvBP4kIpoyPubqvifMjQS+xNXMCkkuWxBLga0RsS0iOoEHgJuOsfxtwP05rOek7WpqRYKZNe6kNrOxL5cBMROoy3hfn0x7B0mVwDLgpxmTA3hU0lpJAz7/WtLtktZIWtPQkNtnGNU1tXHK+HLKS1I53Y6Z2UiQy4DIdqNADLDsjcBv+51eujwiLgSuAz4n6cpsK0bEPRGxJCKW1NbWnlzFx9F3D4SZWSHIZUDUA7Mz3s9i4IcM3Uq/00sRsTv5uQ9YQfqUVV75ElczKyS5DIjVwHxJ8ySVkg6Bh/ovJGkC8F7gwYxpVZKq+14D1wIbcljrcbV39fDm4XYHhJkVjJxdxRQR3ZLuBB4BUsC9EbFR0h3J/OXJojcDj0ZES8bq04AVyXAWxcB9EZHXJ9jVH0gP8+2AMLNCkbOAAIiIlcDKftOW93v/PeB7/aZtA87PZW0nqi65xHW2h9kwswLhO6kHadfRgHALwswKgwNikOqaWikvKaJ2XFm+SzEzGxYOiEHqu4LJw3ybWaFwQAySL3E1s0LjgBiEiPBNcmZWcBwQg9DU0klLZ49bEGZWUBwQg+BRXM2sEDkgBsGXuJpZIXJADMLRm+QmOiDMrHA4IAahrqmN2uoyKko9zLeZFQ4HxCD4ElczK0QOiEFwQJhZIXJAHEdndy97DrW5g9rMCo4D4jh2H2yjN3yJq5kVHgfEcfgeCDMrVA6I49jl50CYWYHKaUBIWiZps6Stku7KMv8vJK1L/myQ1CNp0mDWHS51B1opTRUxrbo8XyWYmeVFzgJCUgq4G7gOWATcJmlR5jIR8bWIWBwRi4G/Ap6IiKbBrDtc6ppamTWpgqIiD/NtZoUlly2IpcDWiNgWEZ3AA8BNx1j+NuD+d7luzvgSVzMrVLkMiJlAXcb7+mTaO0iqBJYBP30X694uaY2kNQ0NDSdddH+7Gh0QZlaYchkQ2c7JxADL3gj8NiKaTnTdiLgnIpZExJLa2tp3UebADrV2cbi92wFhZgUplwFRD8zOeD8L2D3Asrfy1umlE103ZzyKq5kVslwGxGpgvqR5kkpJh8BD/ReSNAF4L/Dgia6ba74HwswKWXGuPjgiuiXdCTwCpIB7I2KjpDuS+cuTRW8GHo2IluOtm6taB+IWhJkVspwFBEBErARW9pu2vN/77wHfG8y6w63uQCuTqkoZV5bTX5OZ2YjkO6mPYVdjK7Mn+g5qMytMDohj2NnUwtwpVfkuw8wsLxwQA+js7uWNA22cOtkBYWaFyQExgPoDrfQGzJ3sDmozK0wOiAHsbExfweQWhJkVKgfEAHY0pq+6PdUtCDMrUA6IAexsbGVcWTGTq0rzXYqZWV44IAawo7GFUydXInmYbzMrTA6IAexsbGWu+x/MrIA5ILLo7umlrqnV/Q9mVtAcEFnsPthOd2+4BWFmBc0BkYWvYDIzc0BktTMJCA+zYWaFzAGRxY7GVspLiphaXZbvUszM8sYBkcXOxhbmTq7yJa5mVtByGhCSlknaLGmrpLsGWOYqSeskbZT0RMb0HZLWJ/PW5LLO/nY0+gomM7OcPQlHUgq4G/gA6WdMr5b0UERsylimBvg2sCwidkma2u9jro6I/bmqMZve3mBXUyvXnNW/FDOzwpLLFsRSYGtEbIuITuAB4KZ+y3wC+FlE7AKIiH05rGdQ3jzcTmd3rwfpM7OCl8uAmAnUZbyvT6ZlOhOYKOlxSWsl/X7GvAAeTabfPtBGJN0uaY2kNQ0NDSdddN8lrh7m28wKXS4ftpythzeybP8i4BqgAnhW0nMR8RpweUTsTk47/UrSqxHx5Ds+MOIe4B6AJUuW9P/8E3Z0mG9f4mpmBS6XLYh6YHbG+1nA7izLPBwRLUlfw5PA+QARsTv5uQ9YQfqUVc7taGyhtLiI6ePLh2NzZmYjVi4DYjUwX9I8SaXArcBD/ZZ5ELhCUrGkSuAS4BVJVZKqASRVAdcCG3JY61E797cyZ1IlRUW+xNXMClvOTjFFRLekO4FHgBRwb0RslHRHMn95RLwi6WHgZaAX+G5EbJB0GrAiuQ+hGLgvIh7OVa2ZdjS2uP/BzIzc9kEQESuBlf2mLe/3/mvA1/pN20Zyqmk4RQQ7G1u5/Iwpw71pM7MRx3dSZ2ho7qCtq8ctCDMzHBBvs6PvCibfA2Fm5oDI9NY9EA4IMzMHRIadjS0UF4kZNb7E1czMAZFhR2MrsydVUpzyr8XMzEfCDDsbWzyKq5lZwgGRiAh27m91/4OZWcIBkWhq6aS5o9stCDOzhAMi0XeJq1sQZmZpDojEzuQS1zluQZiZAQ6Io3Y0tlIkmDWxIt+lmJmNCA6IxM7GFmbUVFBWnMp3KWZmI4IDIrGj0VcwmZllckAkfA+EmdnbOSCAg62dHGztcgvCzCyDA4KM51C7BWFmdlROA0LSMkmbJW2VdNcAy1wlaZ2kjZKeOJF1h8rRUVynuAVhZtYnZ0+Uk5QC7gY+ANQDqyU9FBGbMpapAb4NLIuIXZKmDnbdodTXgpgzyS0IM7M+uWxBLAW2RsS2iOgEHgBu6rfMJ4CfRcQugIjYdwLrDpkdjS1Mn1BOeYkvcTUz65PLgJgJ1GW8r0+mZToTmCjpcUlrJf3+CawLgKTbJa2RtKahoeFdFbqzsdX9D2Zm/eQyIJRlWvR7XwxcBFwPfBD4L5LOHOS66YkR90TEkohYUltb+64K3dnY4iuYzMz6yVkfBOlv/bMz3s8CdmdZZn9EtAAtkp4Ezh/kukOipze48sxaLj1tci4+3sxs1MplC2I1MF/SPEmlwK3AQ/2WeRC4QlKxpErgEuCVQa47JFJF4hsfW8xHLsh6BsvMrGDlrAUREd2S7gQeAVLAvRGxUdIdyfzlEfGKpIeBl4Fe4LsRsQEg27q5qtXMzN5JEVlP7Y9KS5YsiTVr1uS7DDOzUUPS2ohYkm2e76Q2M7OsHBBmZpaVA8LMzLJyQJiZWVYOCDMzy8oBYWZmWY2py1wlNQA73+XqU4D9Q1jOaOH9Lize78IymP0+NSKyjlM0pgLiZEhaM9C1wGOZ97uweL8Ly8nut08xmZlZVg4IMzPLygHxlnvyXUCeeL8Li/e7sJzUfrsPwszMsnILwszMsnJAmJlZVgUfEJKWSdosaauku/JdTy5JulfSPkkbMqZNkvQrSVuSnxPzWeNQkzRb0m8kvSJpo6QvJtPH+n6XS1ol6aVkv/9bMn1M73cfSSlJL0r6ZfK+UPZ7h6T1ktZJWpNMe9f7XtABISkF3A1cBywCbpO0KL9V5dT3gGX9pt0FPBYR84HHkvdjSTfwZxGxELgU+FzydzzW97sDeF9EnA8sBpZJupSxv999vkj66ZR9CmW/Aa6OiMUZ9z+8630v6IAAlgJbI2JbRHQCDwA35bmmnImIJ4GmfpNvAr6fvP4+8JHhrCnXImJPRLyQvG4mfdCYydjf74iII8nbkuRPMMb3G0DSLOB64LsZk8f8fh/Du973Qg+ImUBdxvv6ZFohmRYReyB9MAWm5rmenJE0F7gAeJ4C2O/kNMs6YB/wq4goiP0G/gn4T6QfY9ynEPYb0l8CHpW0VtLtybR3ve85eyb1KKEs03zd7xgkaRzwU+BLEXFYyvZXP7ZERA+wWFINsELSOXkuKeck3QDsi4i1kq7Kczn5cHlE7JY0FfiVpFdP5sMKvQVRD8zOeD8L2J2nWvJlr6TpAMnPfXmuZ8hJKiEdDj+KiJ8lk8f8fveJiIPA46T7n8b6fl8OfFjSDtKnjN8n6YeM/f0GICJ2Jz/3AStIn0Z/1/te6AGxGpgvaZ6kUuBW4KE81zTcHgL+IHn9B8CDeaxlyCndVPgX4JWI+EbGrLG+37VJywFJFcD7gVcZ4/sdEX8VEbMiYi7p/8//ERGfZIzvN4CkKknVfa+Ba4ENnMS+F/yd1JI+RPqcZQq4NyL+Pr8V5Y6k+4GrSA8BvBf4CvBz4MfAHGAX8LsR0b8je9SS9B7gKWA9b52T/jLpfoixvN/nke6QTJH+IvjjiPgbSZMZw/udKTnF9OcRcUMh7Lek00i3GiDdfXBfRPz9yex7wQeEmZllV+inmMzMbAAOCDMzy8oBYWZmWTkgzMwsKweEmZll5YAwyyNJV/WNOGo20jggzMwsKweE2SBI+mTyfIV1kr6TDIR3RNI/SnpB0mOSapNlF0t6TtLLklb0jb8v6QxJv06e0fCCpNOTjx8n6SeSXpX0o+TubyT9g6RNyed8PU+7bgXMAWF2HJIWAh8nPRDaYqAH+D2gCnghIi4EniB9ZzrAD4C/jIjzSN/B3Tf9R8DdyTMaLgP2JNMvAL5E+pkkpwGXS5oE3AycnXzO3+VyH82ycUCYHd81wEXA6mT47GtIH8h7gX9Nlvkh8B5JE4CaiHgimf594MpkjJyZEbECICLaI6I1WWZVRNRHRC+wDpgLHAbage9KugXoW9Zs2DggzI5PwPeTp3QtjogFEfHVLMsda9yaY40v3pHxugcojohu0iNx/pT0A14ePrGSzU6eA8Ls+B4DPpqMsd/3jN9TSf//+WiyzCeApyPiEHBA0hXJ9E8BT0TEYaBe0keSzyiTVDnQBpPnV0yIiJWkTz8tHvK9MjuOQn9gkNlxRcQmSX9N+kldRUAX8DmgBThb0lrgEOl+CkgPqbw8CYBtwGeS6Z8CviPpb5LP+N1jbLYaeFBSOenWx58M8W6ZHZdHczV7lyQdiYhx+a7DLFd8isnMzLJyC8LMzLJyC8LMzLJyQJiZWVYOCDMzy8oBYWZmWTkgzMwsq/8fDut6XVXFPrEAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAojElEQVR4nO3deXwc9X3/8ddHq/u2DtuSfEi+wAfYgDFnCQkQzGmuEGggzVVCH6EhaZKGtE2bpk1Lfm3TXBAHCA1pUggpMUdiwpVCAgZiA4b44LBlG8unLNm67/38/tixkY1sZFujlXbez8dDD+3OzM5+vj72vfP9znzH3B0REYmutGQXICIiyaUgEBGJOAWBiEjEKQhERCJOQSAiEnEKAhGRiFMQiAySmf3YzP55kNtuNLNzj3Y/IsNBQSAiEnEKAhGRiFMQSEoJumS+ZGavmVmbmf3IzMaZ2aNm1mJmT5rZmH7bX2pmq81sj5k9bWYz+607wcxeDl73cyD7gPe62MxWBq9dZmbHH2HNf25m68ys0cweNrPKYLmZ2X+a2U4zawraNCdYd6GZrQlq22JmXzyiPzARFASSmq4EzgNmAJcAjwJ/A5SR+Df/WQAzmwHcC3wOKAeWAo+YWaaZZQIPAv8NlAC/CPZL8NoTgbuBTwOlwA+Bh80s63AKNbMPAP8KXA1UAJuA+4LVHwTOCtpRDHwYaAjW/Qj4tLsXAHOA3x7O+4r0pyCQVPQ9d9/h7luA3wMvuvsr7t4FLAFOCLb7MPBrd3/C3XuAfwdygNOBU4EM4Nvu3uPu/wss7/cefw780N1fdPc+d78H6Apedzg+Atzt7i8H9X0FOM3MqoEeoAA4FjB3X+vu24LX9QCzzKzQ3Xe7+8uH+b4i+ygIJBXt6Pe4Y4Dn+cHjShLfwAFw9ziwGagK1m3x/Wdl3NTv8WTgC0G30B4z2wNMDF53OA6soZXEt/4qd/8t8H3gNmCHmd1hZoXBplcCFwKbzOwZMzvtMN9XZB8FgUTZVhIf6ECiT57Eh/kWYBtQFSzba1K/x5uBb7h7cb+fXHe/9yhryCPR1bQFwN2/6+4nAbNJdBF9KVi+3N0XAWNJdGHdf5jvK7KPgkCi7H7gIjM7x8wygC+Q6N5ZBjwP9AKfNbN0M7sCWNDvtXcCN5rZKcGgbp6ZXWRmBYdZw/8AHzezecH4wr+Q6MraaGYnB/vPANqATqAvGMP4iJkVBV1azUDfUfw5SMQpCCSy3P0N4Drge8AuEgPLl7h7t7t3A1cAHwN2kxhP+GW/164gMU7w/WD9umDbw63hKeCrwAMkjkKmAtcEqwtJBM5uEt1HDSTGMQCuBzaaWTNwY9AOkSNiujGNiEi06YhARCTiFAQiIhGnIBARiTgFgYhIxKUnu4DDVVZW5tXV1ckuQ0RkVHnppZd2uXv5QOtGXRBUV1ezYsWKZJchIjKqmNmmg61T15CISMQpCEREIk5BICIScaNujGAgPT091NXV0dnZmexSQpednc2ECRPIyMhIdikikiJSIgjq6uooKCigurqa/SeLTC3uTkNDA3V1ddTU1CS7HBFJESnRNdTZ2UlpaWlKhwCAmVFaWhqJIx8RGT4pEQRAyofAXlFpp4gMn5QJgvfS2dPHtqYOeuPxZJciIjKiRCYIunvj1Ld00d079EGwZ88ebr/99sN+3YUXXsiePXuGvB4RkcMRmSDITE80dTiDoK/v0DeNWrp0KcXFxUNej4jI4UiJs4YGIzOWCIKuEILglltuYf369cybN4+MjAzy8/OpqKhg5cqVrFmzhssuu4zNmzfT2dnJzTffzA033AC8M11Ga2srF1xwAWeeeSbLli2jqqqKhx56iJycnCGvVUTkQCkXBP/4yGrWbG0ecF17dx+xNCMr/fAOhGZVFvIPl8w+6Ppbb72VVatWsXLlSp5++mkuuugiVq1ate8Uz7vvvpuSkhI6Ojo4+eSTufLKKyktLd1vH2+99Rb33nsvd955J1dffTUPPPAA112nuw+KSPhSLggOJc0S5+KHbcGCBfud5//d736XJUuWALB582beeuutdwVBTU0N8+bNA+Ckk05i48aNodcpIgIpGASH+uZe19hOc1cvsyoKQ60hLy9v3+Onn36aJ598kueff57c3FzOPvvsAa8DyMrK2vc4FovR0dERao0iIntFZrAYEgPGvX1x+uJDe1RQUFBAS0vLgOuampoYM2YMubm5vP7667zwwgtD+t4iIkcr5Y4IDqX/mUM5mbEh229paSlnnHEGc+bMIScnh3Hjxu1bt3DhQhYvXszxxx/PMcccw6mnnjpk7ysiMhRsOPrMh9L8+fP9wBvTrF27lpkzZ77na9u7e1m3s5XJpXkU5YzeSdsG214Rkb3M7CV3nz/Qush1DQF09x76/H4RkSiJVBCkp6WRnmahXEsgIjJapUwQDLaLKzM9FsrVxcNltHXlicjIlxJBkJ2dTUNDw6A+JDPT00ZtEOy9H0F2dnaySxGRFJISZw1NmDCBuro66uvr33Pb5o4eWjp7ie/OHpVTOu+9Q5mIyFBJiSDIyMgY9B27Hnipji88/Cr/98WzqSnLe+8XiIikuJToGjock0tzAdjY0JbkSkRERobIBcGkIAjebmhPciUiIiND5IKgPD+L3MyYjghERAKhBoGZLTSzN8xsnZndMsD6s82sycxWBj9/H2Y9wXsyqSRXRwQiIoHQBovNLAbcBpwH1AHLzexhd19zwKa/d/eLw6pjINWleayrbx3OtxQRGbHCPCJYAKxz91p37wbuAxaF+H6DNrk0l7cb24kP8SykIiKjUZhBUAVs7ve8Llh2oNPM7FUze9TMBryZgJndYGYrzGzFYK4VeC+TSnPp7o2zvfnd9wUQEYmaMINgoKu1DvwK/jIw2d3nAt8DHhxoR+5+h7vPd/f55eXlR11YdWni+gENGIuIhBsEdcDEfs8nAFv7b+Duze7eGjxeCmSYWVmINQEwqUSnkIqI7BVmECwHpptZjZllAtcAD/ffwMzGWzDPg5ktCOppCLEmACqLc8iIGZsaFQQiIqGdNeTuvWZ2E/AYEAPudvfVZnZjsH4xcBXwF2bWC3QA1/gwTK8ZSzMmjsllk7qGRETCnWso6O5ZesCyxf0efx/4fpg1HMyk0lw2qWtIRCR6VxbvVV2ax6aGds3vLyKRF9kgmFSSS2tXL41t3ckuRUQkqSIbBNVle2chVfeQiERbZINgUkniWoK3GzVgLCLRFtkgmFiSgxkaMBaRyItsEGSlx6gsylEQiEjkRTYIIDFgrGsJRCTqIh0E1WW6lkBEJNJBMKkkj4a2blq7epNdiohI0kQ6CPbeyF7dQyISZQoCdOaQiERbxIMgcS2BgkBEoizSQZCflU5Zfqa6hkQk0iIdBLD3FFIdEYhIdEU+CCaX5vG2blAjIhGmICjNZWtTB129fckuRUQkKRQEpbm4w+bGjmSXIiKSFAqCfWcOacBYRKIp8kFQHQSB7ksgIlEV+SAYk5tBQXY6G3fpiEBEoinyQWBm1JTlsVFdQyISUZEPAkh0D23QEYGIRJSCAKguy2PrHp1CKiLRpCAAaspyiTts1oVlIhJBCgLeOXNowy4FgYhEj4IAqCkLTiHVOIGIRFCoQWBmC83sDTNbZ2a3HGK7k82sz8yuCrOegynOzaQ4N4MNOnNIRCIotCAwsxhwG3ABMAu41sxmHWS7bwKPhVXLYFSX5umIQEQiKcwjggXAOnevdfdu4D5g0QDb/SXwALAzxFre05QyBYGIRFOYQVAFbO73vC5Yto+ZVQGXA4tDrGNQqsvy2NrUSWePTiEVkWgJMwhsgGV+wPNvA19290N++prZDWa2wsxW1NfXD1V9+6ku020rRSSawgyCOmBiv+cTgK0HbDMfuM/MNgJXAbeb2WUH7sjd73D3+e4+v7y8PJRia/adQqruIRGJlvQQ970cmG5mNcAW4BrgT/tv4O41ex+b2Y+BX7n7gyHWdFDVZbmAgkBEoie0IHD3XjO7icTZQDHgbndfbWY3BuuTPi7QX0F2BmX5mRowFpHICfOIAHdfCiw9YNmAAeDuHwuzlsGoLs3TtQQiEjm6srifap1CKiIRpCDop6Ysj50tXbR19Sa7FBGRYaMg6Oed21bqqEBEokNB0M/eM4c2ahZSEYkQBUE/OiIQkShSEPSTl5XO2IIsXUsgIpGiIDiAzhwSkahREBygpjRPXUMiEikKggNUl+Wxq7Wbls6eZJciIjIsFAQHqNGZQyISMQqCA+ydjlpTTYhIVCgIDjC5RDeyF5FoURAcICczRkVRtoJARCJDQTAAzUIqIlGiIBiAriUQkShREAygpiyX3e09NLXrFFIRSX0KggHsnXNI3UMiEgUKggHUlOnMIRGJDgXBACaW5GKmG9mLSDQoCAaQnRGjsihHQSAikaAgOIiaMk0+JyLRoCA4iOqyXDbsasPdk12KiEioFAQHUV2aR0tnL41t3ckuRUQkVAqCg9h35pC6h0QkxSkIDmLfLKSajlpEUpyC4CAmjsklzXQtgYikPgXBQWSmpzF9bAGv1u1JdikiIqEKNQjMbKGZvWFm68zslgHWLzKz18xspZmtMLMzw6zncJ02tZTlGxvp6u1LdikiIqEJLQjMLAbcBlwAzAKuNbNZB2z2FDDX3ecBnwDuCqueI3H61FI6e+KsfHtPsksREQlNmEcEC4B17l7r7t3AfcCi/hu4e6u/c6J+HjCiTto/ZUopaQbPrW9IdikiIqEJMwiqgM39ntcFy/ZjZpeb2evAr0kcFbyLmd0QdB2tqK+vD6XYgRTlZHBcVRHPr981bO8pIjLcBhUEZnazmRVawo/M7GUz++B7vWyAZe/6xu/uS9z9WOAy4J8G2pG73+Hu8919fnl5+WBKHjKnTyvjlbf30NbVO6zvKyIyXAZ7RPAJd28GPgiUAx8Hbn2P19QBE/s9nwBsPdjG7v47YKqZlQ2ypmFx+tRSeuPO8o2NyS5FRCQUgw2Cvd/uLwT+y91fZeBv/P0tB6abWY2ZZQLXAA/vt1OzaWZmweMTgUxgRHXIz59cQmYsjWUaJxCRFJU+yO1eMrPHgRrgK2ZWAMQP9QJ37zWzm4DHgBhwt7uvNrMbg/WLgSuBj5pZD9ABfNhH2CxvOZkxTphUzDKNE4hIihpsEHwSmAfUunu7mZWQ6B46JHdfCiw9YNnifo+/CXxz0NUmyRnTyvjPJ99kT3s3xbmZyS5HRGRIDbZr6DTgDXffY2bXAX8HNIVX1shy+tRS3OGFWnUPiUjqGWwQ/ABoN7O5wF8Dm4CfhFbVCDN3YjG5mTGeW6cgEJHUM9gg6A367hcB33H37wAF4ZU1smTE0lhQU6JxAhFJSYMNghYz+wpwPfDrYPqIjPDKGnnOmFrG+vo2tjd1JrsUEZEhNdgg+DDQReJ6gu0krhD+t9CqGoFOm1oKwPO1OioQkdQyqCAIPvx/BhSZ2cVAp7tHZowAYFZFIcW5GRonEJGUM9gpJq4G/gB8CLgaeNHMrgqzsJEmLc04bUopz69v0A3tRSSlDPY6gr8FTnb3nQBmVg48CfxvWIWNRKdPK+PRVdvZ1NC+71aWIiKj3WDHCNL2hkCg4TBemzJOD8YJNN2EiKSSwX6Y/8bMHjOzj5nZx0hMGb30PV6TcqaU5TG+MJvndBqpiKSQQXUNufuXzOxK4AwSk83d4e5LQq1sBDIzTp9ayjNv1hOPO2lp7zXvnojIyDfYMQLc/QHggRBrGRVOn1bGL1/Zwhs7WphZUZjsckREjtohg8DMWhj49pEGuLtH7pOw/ziBgkBEUsEhxwjcvcDdCwf4KYhiCABUFudQU5bHc+s0TiAiqSFyZ/4MhbOml7Fs/S46uvuSXYqIyFFTEByB82ePp7MnzjNv1ie7FBGRo6YgOAILakoozs3g8dXbk12KiMhRUxAcgfRYGuccO44n1+6gp++Qd+wUERnxFARH6PzZ42ju7NVdy0Rk1FMQHKGzZpSTkxHjMXUPicgopyA4QtkZMc4+ppzHV+8gHtdspCIyeikIjsL5s8ezs6WLlXV7kl2KiMgRUxAchfcfO5b0NFP3kIiMagqCo1CUk8FpU0t5bNV23axGREYtBcFRWjhnPBsb2nlzR2uySxEROSIKgqN03qxxmKHuIREZtUINAjNbaGZvmNk6M7tlgPUfMbPXgp9lZjY3zHrCMLYgmxMnjVEQiMioFVoQmFkMuA24AJgFXGtmsw7YbAPwPnc/Hvgn4I6w6gnT+bPHsXprM5sb25NdiojIYQvziGABsM7da929G7gPWNR/A3df5u67g6cvABNCrCc0588eD6h7SERGpzCDoArY3O95XbDsYD4JPBpiPaGZXJrHseMLeHz1jmSXIiJy2MIMgoFu6DvgOZZm9n4SQfDlg6y/wcxWmNmK+vqROfXz+bPHs3xTI7tau5JdiojIYQkzCOqAif2eTwC2HriRmR0P3AUscvcBZ3Bz9zvcfb67zy8vLw+l2KN1/uzxuMOTa3RUICKjS5hBsByYbmY1ZpYJXAM83H8DM5sE/BK43t3fDLGW0M2sKGBiSY7GCURk1AktCNy9F7gJeAxYC9zv7qvN7EYzuzHY7O+BUuB2M1tpZivCqidsZsbC2eN5bl0DTe09yS5HRGTQQr2OwN2XuvsMd5/q7t8Ili1298XB40+5+xh3nxf8zA+znrBdOreK7r44v/7jtmSXIiIyaLqyeAjNqSpk2th8lrxSl+xSREQGTUEwhMyMy0+oYvnG3bq4TERGDQXBEFs0rxKAJa9sSXIlIiKDoyAYYhPG5HJKTQlLXtmiqalFZFRQEITgihOr2LCrjZWb9yS7FBGR96QgCMEFx1WQlZ6m7iERGRUUBCEozM7g3FnjeOTVrXT3xpNdjojIISkIQnLFCVXsbu/hmTdH5txIIiJ7KQhCctaMckrzMnVNgYiMeAqCkGTE0rhkbiVPrt1JU4emnBCRkUtBEKLLT6iiuzfOUk05ISIjmIIgRMdPKGJKeR5LXtbZQyIycikIQmRmXHFCFX/Y2KgpJ0RkxFIQhGzRvMTdOR/UNQUiMkIpCEI2sSSXBZpyQkRGMAXBMLjihCpqd7XxWl1TsksREXkXBcEwuOC4CjJixsOvvuuWzSIiSacgGAZFORm8b8ZYfvXaVvri6h4SkZFFQTBMLp1XyY7mLpZvbEx2KSIi+1EQDJNzZ44lJyOm7iERGXEUBMMkNzOd82aN49E/bqOnTzOSisjIoSAYRpfMrWR3ew/PrtuV7FJERPZREAyjs2aUUZidziMr1T0kIiOHgmAYZaXHuGBOBY+t3k5nT1+yyxERARQEw+7SeZW0dffx29d3JrsUERFAQTDsTp1SSll+Fo/o7CERGSEUBMMslmZcfHwFT72+k5ZO3bBGRJIv1CAws4Vm9oaZrTOzWwZYf6yZPW9mXWb2xTBrGUkumVtJd2+cx1fvSHYpIiLhBYGZxYDbgAuAWcC1ZjbrgM0agc8C/x5WHSPRiZOKqSrO0cVlIjIihHlEsABY5+617t4N3Acs6r+Bu+909+VApPpIzIxL5lby7LpdNLZ1J7scEYm4MIOgCtjc73ldsOywmdkNZrbCzFbU19cPSXHJduncSvrirvsZi0jShRkENsCyI5p6093vcPf57j6/vLz8KMsaGWZWFDBtbL66h0Qk6cIMgjpgYr/nEwB96gXMjEvnVrJ8YyPbmjqSXY6IRFiYQbAcmG5mNWaWCVwDPBzi+406l8ytxB1+saIu2aWISISFFgTu3gvcBDwGrAXud/fVZnajmd0IYGbjzawO+Cvg78yszswKw6pppKkpy+PcmWO58/e1NLVHarxcREaQUK8jcPel7j7D3ae6+zeCZYvdfXHweLu7T3D3QncvDh43h1nTSPP582bQ0tnLXc/WJrsUEYkoXVmcZLMri7jo+ArufnYDDa1dyS5HRCJIQTACfP7c6XT09LH4mfXJLkVEIkhBMAJMG1vAZSdU8ZPnN7GjuTPZ5YhIxCgIRojPnTODvrjz/d+uS3YpIhIxCoIRYlJpLlefPJH7lr/N5sb2ZJcjIhGiIBhB/vID0zAzvvfbt5JdiohEiIJgBKkoyuG6UybzwMtbqK1vTXY5IhIRCoIR5i/OnkpmLI1vP6mjAhEZHgqCEaa8IIuPn1HNI69t5fXtkbq2TkSSREEwAt1w1hTys9L53H0rdTqpiIROQTACFedm8v0/PZHNje1cfttzvLG9JdkliUgKUxCMUO+bUc7PP30avXHnqh8sY9m6XckuSURSlIJgBJtTVcSSz5xBRXE2f/Zff+CXL2u6ahEZegqCEa6qOIdf3Hg68yeX8Ff3v8p3n3oL9yO60ZuIyIAUBKNAUU4G93xiAVecUMW3nniTz/98pW56LyJDRkEwSmSmp/EfV8/l8+fO4JHXtnH2v/0f9yzbSG9fPNmlicgopyAYRcyMm8+dzm9u/hOOm1DEPzy8mou/9yzPr29IdmkiMoopCEah6eMK+OknT2HxdSfR2tXLtXe+wGf+52W27OkI9X2XvFLHV375Gtuawn0fkSjb1tQx7OOANtoGHufPn+8rVqxIdhkjRmdPHz98ppbbn15HV2+cquIcpo/LZ1p5fuL32AKmjc2nKCfjqN7nv1/YxFcfXAVAbmaMm8+ZzifOrCEjpu8SIkPlN6u2c+NPX+Km90/ji+cfM6T7NrOX3H3+gOsUBKmhbnc7D76yhbd2tvLWjlbW17fS1fvO+EFlUTYzKwo5tqIg8Xt8ITVlecTS7D33/ePnNvC1R9Zw7syx3HLBTG59dC1Prt3J9LH5fH3RHE6bWhpm00Qioamjh/O+9Qx72nvo7ouz+LqTWDhn/JDtX0EQQX1xp253O2/taOXNnS28sb2FtduaWV/fRl888XeenZHGJcdX8rnzZlBVnDPgfu76fS3//Ou1nD97HN+79kQy0xNHAE+u2cHXHllN3e4OFs2r5G8vnMnYwuxha59IqvnKL1/j58s384sbT+Prv1rLuh0tPHTTGUwbWzAk+1cQyD5dvX2s29nK2m0tvLRpNw+8XAcOHzl1Ep95/zTK8rP2bfuDp9fzzd+8zoXHjec715zwrm6gzp4+bn96PYufWU9mLI1PnzWFT/5JDbmZ6cPdLJFR7fn1DVx75wvccNYU/ubCmWxr6uCS7z1LYXYGD950BoXZR9e1CwoCOYStezr4zpNv8YuXNpOTEeOTZ9bwqbOmcM9zG/mPJ97kkrmV/OfVc0k/xFjAxl1t/MvStTy+ZgdjC7L4/Hkz+NBJEw75GhFJ6OzpY+G3f0fc4bHPnUVOZgyAF2sb+MhdL3L2MWO54/qTSBtEN+6hKAjkPa2vb+Vbj7/Jr/+4jbzMGG3dfVx+QhX/dtXxg/5AX7GxkX999HVe2rSbqeV5/PXCY/ngrHGY7f8PuKO7j8b2bsryM8lKj4XRnKRatn4X337iLSaMyeHKkyZw6pTSQY3FSDTd+ujrLH5mPT/71CmcMa1sv3V7x+c+f+4Mbj53+lG9j4JABu2PdU1856nEh9hXL5512B9g7s7ja3bwzd+8Tm19G3MnFlOen8mu1m4a2rpoaO2mvbsPgKz0NE6cNIZTp5Ry6pQS5k0q3i8Y3J3d7T1sbGjj7YZ29rR3U5CdQWFOBoXZ6RTlZlCYnUFxbsaI6I7a3dbNvyxdyy9eqqOyKJuWrl5aOnupKMrm8hOquPKkCUwtz9+3fWNbN2u3NbNmazNrtjXT0tlLTVku1WV51AQ/4wqyj/qboIxcq7Y0sei257jyxCr+31Vz37Xe3fnC/a+yZOUW7vrofM6ZOe6I30tBIMOuty/O/SvquGfZRmJpRml+JqV5mZTmZ1GSl8mY3EzW17fyQm0Da7Y14/5OMIzJy2BTQztvN7TT0tU7qPebXJrLnKoi5lQWcVxVEXOqCinOzSQed7bs6WBdfSvrdybOplpf38aY3AxOri5hfnUJsysLj+o0WHfn4Ve38vVH1tDU0cMNZ03hs+ckvr09sWYHD7xcx+/erCfuMHdiMWV5mazZ1sy2pnfuNVFRlE1BdjqbGtr3O9srJyPG5NJcppbnM6U8jynledSUJR4PRb/x4Wjq6OHNHYkTDzY1tDF9XAGnTy1lwpjcYa0jWV6r28O9f3ibjFgaNWV5TCnPZ0pZHpXFOfu+MHX29LFhVxvrdrby1s7EvzkMLjqugg8cO5bsjHe+6PT2xbns9ufY3tTFU3/1PopyB/777Ozp48ofLOPtxnYevulMasryjqj+pAWBmS0EvgPEgLvc/dYD1luw/kKgHfiYu798qH0qCFJPU3sPf9jYyAu1Dby4oYH2rj4mleYyuSSXSaV5TC7JpboslzG5mbR29dLc0UtzZw/NHT00d/ZQ39LF6q3N/HFLE3W737nYraIom93t3XT2vPPBWpKXyZSyPHa2dPF2YzuQOHvqhIljOLl6DOOLcmhs6wqOYLppaE0cxXT3xff/QC7LZ+rYPLp64nz1oVU8/UY9cycUceuVxzOzovBdbdzZ3MlDK7fy4Mot9PTFmV1ZxKyKQmZVFjKzopCSvEwA4nFne3MnG3a1UburjY272qitb2XDrjbebmwn3u+/a3lBFrMqCplTVcicyiLmVBUxYUzOu7riBsvd2dPew9amDrbt6WRrUwebG9t5c0crb+5o2S+40tOM3qCYSSW5nD61lNOmlnJa0A22eXfitZt3t7O5sYO63e1kpadxSk0pp0wpYVZF4agYQ3J3nn6znjueqeX52gbyMmOkme33BSUzPY3q0ly6e+P7/R2lGUwsyaWtq49drV3kZ6Vz/uzxLJpXyelTS7nr2Q3c+ujr3P6RE7nwuIpD1rG5sZ1Lv/8si+ZV8bVLZx9RW5ISBGYWA94EzgPqgOXAte6+pt82FwJ/SSIITgG+4+6nHGq/CgI5lD3t3azakgiFN7Y3U5qfxbSx+Uwbm8/U8vx9H7gAO5o7WbFxN8s3NrJiUyNrtjbv+09ckJWeOIrJz6I0L5P0mLFhVzsbdrXuFyyQuMDuS+cfw0dPqw51LCDxQdPG+vo2ausT3zpXb23irZ2t+04JLsrJYHZlIflZ6fTFnd640xuP09uXeNwXdxzAE7/dwXHau/rY1tRJR0/ffu+ZmZ7GtPJ8jh1fwIzxBRwzLvG7ojCbdfWtLFu3i2XrG3ihtoHmzoGP3krzMplQkktLRw+1u9oAyM9KZ371GE6pKWVOVSHpaWmYgQFpaYYBZnvrSwRk/3qz0tPISo+RkxkjOyNGdnoa2RkxeuNOa1cvrZ29tHb10NKZ6J7r7o0n9m+QFgSlmZEZM4pzE0erJXmZFOdmEkszunvjPPLqVu78fS2vb2+hoiibT5xRwzULJpKflc6u1u59AV27K/H3kZlu+y7gnD42n5qyPLIzYvTFnefXN/DQyi38ZtV2Wrp6KcvPpKWzl/fNKOeH1580qPDesKuNySW5R9xVmKwgOA34mrufHzz/CoC7/2u/bX4IPO3u9wbP3wDOdvdtB9uvgkDCkjja6KEkL3O/Q/j+4nFna1MHtfWJb+oNbd1cs2DSQa/DGA6dPX28sb2FVVubWLWliTVbm+nqjZMeM2JpaWSkGbE0Iz1mpJlh9s4HbeK3kZ2RRkVRDpXFOVQWZVNRnENlcTZleVmD+uDpiztrtjbz4oYG0syYWJLLpJJcJozJIS/rnfGbnc2dvLChkRdrG3hxQyPrdraG9wdzBMygOCcDB/a093DMuAJuOGsKl8yt3HcNzdHo7Onj6Td28tDKrdTWt/GTTy5g3DBdf5OsILgKWOjunwqeXw+c4u439dvmV8Ct7v5s8Pwp4MvuvuKAfd0A3AAwadKkkzZt2hRKzSIyvOpbuqitbyXuiW6Yvd/648FjgyC83gktgO6+OJ09ff1+4nT09JERS6MgK5387HTyg98FWelkpqftO7pw9+DIz+nqjbOnvYeGtm4aW7tobO+hsa2Lju44F8+t4OwZ5Ufc1TbSHCoIwjzVYqA/vQNTZzDb4O53AHdA4ojg6EsTkZGgvCCL8oKs995QQhXmaE0dMLHf8wnA1iPYRkREQhRmECwHpptZjZllAtcADx+wzcPARy3hVKDpUOMDIiIy9ELrGnL3XjO7CXiMxOmjd7v7ajO7MVi/GFhK4oyhdSROH/14WPWIiMjAQr0c092Xkviw779scb/HDnwmzBpEROTQRv4VHSIiEioFgYhIxCkIREQiTkEgIhJxo272UTOrB4700uIyYNcQljOaRLXtane0qN0HN9ndywdaMeqC4GiY2YqDXWKd6qLadrU7WtTuI6OuIRGRiFMQiIhEXNSC4I5kF5BEUW272h0tavcRiNQYgYiIvFvUjghEROQACgIRkYiLTBCY2UIze8PM1pnZLcmuJyxmdreZ7TSzVf2WlZjZE2b2VvB7TDJrDIOZTTSz/zOztWa22sxuDpandNvNLNvM/mBmrwbt/sdgeUq3ey8zi5nZK8HdDiPRbjPbaGZ/NLOVZrYiWHZU7Y5EEJhZDLgNuACYBVxrZrOSW1VofgwsPGDZLcBT7j4deCp4nmp6gS+4+0zgVOAzwd9xqre9C/iAu88F5gELg3t7pHq797oZWNvveVTa/X53n9fv2oGjanckggBYAKxz91p37wbuAxYluaZQuPvvgMYDFi8C7gke3wNcNpw1DQd33+buLwePW0h8OFSR4m33hL13gM8IfpwUbzeAmU0ALgLu6rc45dt9EEfV7qgEQRWwud/zumBZVIzbe+e34PfYJNcTKjOrBk4AXiQCbQ+6R1YCO4En3D0S7Qa+Dfw1EO+3LArtduBxM3vJzG4Ilh1Vu0O9Mc0IYgMs03mzKcjM8oEHgM+5e7PZQH/1qcXd+4B5ZlYMLDGzOUkuKXRmdjGw091fMrOzk1zOcDvD3bea2VjgCTN7/Wh3GJUjgjpgYr/nE4CtSaolGXaYWQVA8HtnkusJhZllkAiBn7n7L4PFkWg7gLvvAZ4mMUaU6u0+A7jUzDaS6Or9gJn9lNRvN+6+Nfi9E1hCouv7qNodlSBYDkw3sxozywSuAR5Ock3D6WHgz4LHfwY8lMRaQmGJr/4/Ata6+7f6rUrptptZeXAkgJnlAOcCr5Pi7Xb3r7j7BHevJvH/+bfufh0p3m4zyzOzgr2PgQ8CqzjKdkfmymIzu5BEn2IMuNvdv5HcisJhZvcCZ5OYlnYH8A/Ag8D9wCTgbeBD7n7ggPKoZmZnAr8H/sg7fcZ/Q2KcIGXbbmbHkxgcjJH4Yne/u3/dzEpJ4Xb3F3QNfdHdL071dpvZFBJHAZDo2v8fd//G0bY7MkEgIiIDi0rXkIiIHISCQEQk4hQEIiIRpyAQEYk4BYGISMQpCERCZmZn750dU2QkUhCIiEScgkAkYGbXBXP7rzSzHwaTubWa2X+Y2ctm9pSZlQfbzjOzF8zsNTNbsnf+dzObZmZPBvcHeNnMpga7zzez/zWz183sZ8GV0JjZrWa2JtjPvyep6RJxCgIRwMxmAh8mMaHXPKAP+AiQB7zs7icCz5C4UhvgJ8CX3f14Elcz713+M+C24P4ApwPbguUnAJ8jcT+MKcAZZlYCXA7MDvbzz2G2UeRgFAQiCecAJwHLgymdzyHxgR0Hfh5s81PgTDMrAord/Zlg+T3AWcEcMFXuvgTA3TvdvT3Y5g/uXufucWAlUA00A53AXWZ2BbB3W5FhpSAQSTDgnuCuT/Pc/Rh3/9oA2x1qTpZDzXnd1e9xH5Du7r0kZo58gMSNRH5zeCWLDA0FgUjCU8BVwRzve+8BO5nE/5Grgm3+FHjW3ZuA3Wb2J8Hy64Fn3L0ZqDOzy4J9ZJlZ7sHeMLh3QpG7LyXRbTRvyFslMghRuTGNyCG5+xoz+zsSd35KA3qAzwBtwGwzewloIjGOAImpfhcHH/S1wMeD5dcDPzSzrwf7+NAh3rYAeMjMskkcTXx+iJslMiiafVTkEMys1d3zk12HSJjUNSQiEnE6IhARiTgdEYiIRJyCQEQk4hQEIiIRpyAQEYk4BYGISMT9f5hWpC1i3c/FAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "<Figure size 432x288 with 0 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot the results (Accurecy and loss)\n",
    "\n",
    "plt.plot(history.history['acc'])\n",
    "plt.title('model accuracy')\n",
    "plt.ylabel('acc')\n",
    "plt.xlabel('epochs')\n",
    "plt.legend(['train', 'test'], loc='upper left')\n",
    "plt.show()\n",
    "plt.savefig('model_accuracy.png')\n",
    "\n",
    "# summarize history for loss\n",
    "plt.plot(history.history['loss'])\n",
    "#plt.plot(history.history['val_loss'])\n",
    "plt.title('model loss')\n",
    "plt.ylabel('loss')\n",
    "plt.xlabel('epochs')\n",
    "plt.legend(['train', 'test'], loc='upper left')\n",
    "plt.show()\n",
    "plt.savefig('model_loss.png')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "129/129 - 2s - loss: 0.0043 - acc: 0.9988\n",
      "23/23 - 0s - loss: 1.1497 - acc: 0.7166\n",
      "Train: 0.999, Test: 0.7166\n"
     ]
    }
   ],
   "source": [
    "# evaluate the model\n",
    "_, train_acc = model.evaluate(X_train, Y_train, verbose=2)\n",
    "_, test_acc = model.evaluate(X_test, Y_test, verbose=2)\n",
    "print('Train: %.3f, Test: %.4f' % (train_acc, test_acc))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[4.3916603e-07 9.9999940e-01 1.1227933e-07]\n",
      " [9.9999905e-01 1.1340623e-07 8.9236471e-07]\n",
      " [9.1908396e-07 9.9999869e-01 3.1600985e-07]\n",
      " ...\n",
      " [4.7750432e-02 5.0357246e-04 9.5174599e-01]\n",
      " [1.1223220e-06 9.9999869e-01 2.9577805e-07]\n",
      " [9.9974710e-01 2.1984818e-04 3.3064316e-05]]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\abc\\AppData\\Roaming\\Python\\Python38\\site-packages\\tensorflow\\python\\keras\\engine\\sequential.py:450: UserWarning: `model.predict_classes()` is deprecated and will be removed after 2021-01-01. Please use instead:* `np.argmax(model.predict(x), axis=-1)`,   if your model does multi-class classification   (e.g. if it uses a `softmax` last-layer activation).* `(model.predict(x) > 0.5).astype(\"int32\")`,   if your model does binary classification   (e.g. if it uses a `sigmoid` last-layer activation).\n",
      "  warnings.warn('`model.predict_classes()` is deprecated and '\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1 0 1 0 1 1 1 1 1 1 1 1 0 2 1 1 1 1 0 0 1 1 0 0 1 0 1 1 0 2 1 1 1 2 1 1 0\n",
      " 0 1 1 0 1 2 1 0 1 1 1 0 1 2 1 0 2 1 0 1 1 0 1 0 0 0 0 1 1 1 1 0 2 1 0 0 1\n",
      " 0 1 1 1 1 0 1 2 0 1 0 0 1 0 1 0 0 1 1 2 1 1 0 1 0 1 2 1 1 2 0 1 1 1 2 1 1\n",
      " 1 2 1 1 0 0 0 0 0 0 0 0 1 1 1 0 0 0 1 1 1 2 1 0 1 0 1 1 1 1 0 1 0 1 1 2 1\n",
      " 0 1 1 1 1 1 1 0 0 1 2 2 0 1 1 0 1 0 1 2 1 0 1 0 0 1 0 2 0 0 1 0 0 0 0 1 1\n",
      " 2 1 2 1 1 1 0 0 1 0 0 1 1 1 0 1 1 1 1 1 0 1 1 0 1 0 1 1 1 1 1 0 2 1 0 0 0\n",
      " 1 0 1 1 1 0 1 1 1 0 0 1 2 0 1 1 1 1 1 1 1 0 0 1 1 1 1 0 1 1 0 1 2 0 1 1 1\n",
      " 1 1 0 1 0 2 2 0 1 1 1 1 1 2 0 1 1 0 0 0 0 1 0 1 1 1 1 1 1 0 1 0 1 0 2 2 1\n",
      " 1 1 0 1 0 1 0 1 1 2 1 0 1 0 1 2 0 1 1 1 1 1 1 0 1 1 0 1 0 1 1 0 1 1 1 1 1\n",
      " 1 0 0 1 0 0 1 0 0 1 0 1 2 1 0 1 1 0 1 0 2 1 1 0 1 2 0 1 1 2 1 0 0 0 2 1 0\n",
      " 0 0 1 1 1 1 1 1 0 0 0 0 2 1 0 0 0 1 0 1 0 0 1 0 1 0 1 1 1 0 0 1 1 1 2 2 1\n",
      " 0 1 1 0 1 0 0 1 0 0 0 1 0 1 0 0 1 2 1 1 1 0 1 0 0 1 1 1 0 0 0 2 1 1 0 1 1\n",
      " 0 1 1 2 0 2 1 1 1 0 1 0 2 1 1 0 1 0 1 0 1 0 1 0 1 0 1 1 1 1 1 1 1 0 1 0 1\n",
      " 0 1 0 0 1 2 1 1 0 0 1 0 1 1 0 1 1 1 2 1 1 1 0 0 1 1 1 1 1 0 0 0 1 0 0 1 0\n",
      " 2 0 0 0 0 1 1 1 1 0 0 1 1 1 0 1 1 1 1 1 1 1 1 1 1 0 1 1 1 0 2 1 1 1 1 1 1\n",
      " 0 2 1 0 1 1 0 2 0 1 0 1 0 2 1 1 1 1 1 0 1 1 1 2 1 1 1 0 1 2 0 1 1 1 1 1 1\n",
      " 1 0 0 1 1 1 2 0 0 1 0 1 2 0 2 1 2 1 0 1 1 0 1 1 1 2 1 0 0 2 1 1 1 0 0 0 1\n",
      " 1 1 0 1 1 0 1 1 0 1 1 2 1 1 0 1 1 0 1 1 2 0 0 2 0 1 0 0 0 1 1 0 0 1 1 1 2\n",
      " 1 1 1 0 1 1 1 1 0 1 1 1 2 2 1 1 1 1 1 1 1 0 0 0 1 0 1 0 0 2 1 0 1 1 1 0 1\n",
      " 2 1 0 1 0 1 0 1 2 1 1 1 1 1 0 0 0 0 0 1 1 2 1 0]\n"
     ]
    }
   ],
   "source": [
    "# predict probabilities for test set\n",
    "yhat_probs = model.predict(X_test, verbose=0)\n",
    "print(yhat_probs)\n",
    "# predict crisp classes for test set\n",
    "yhat_classes = model.predict_classes(X_test, verbose=0)\n",
    "print(yhat_classes)\n",
    "# reduce to 1d array\n",
    "#yhat_probs = yhat_probs[:, 0]\n",
    "#yhat_classes = yhat_classes[:, 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 2, 1, 0, 1, 0, 0, 2, 2, 1,\n",
       "       0, 0, 1, 0, 1, 0, 0, 2, 1, 1, 0, 2, 1, 0, 2, 0, 1, 1, 0, 1, 2, 1,\n",
       "       0, 1, 1, 1, 0, 0, 2, 1, 0, 2, 1, 0, 1, 1, 1, 1, 0, 0, 0, 1, 2, 1,\n",
       "       0, 1, 1, 1, 1, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, 2, 0, 1, 0, 0, 0, 0,\n",
       "       0, 0, 0, 1, 2, 2, 1, 0, 1, 1, 0, 1, 2, 1, 1, 2, 1, 1, 1, 1, 1, 1,\n",
       "       1, 0, 2, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 1, 0, 1, 2, 0,\n",
       "       1, 1, 0, 1, 0, 2, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1,\n",
       "       1, 1, 1, 1, 2, 2, 0, 0, 1, 0, 1, 0, 1, 2, 1, 1, 1, 0, 1, 1, 0, 1,\n",
       "       1, 1, 1, 1, 0, 0, 1, 0, 1, 2, 1, 2, 1, 2, 1, 1, 0, 0, 1, 0, 2, 1,\n",
       "       2, 0, 1, 1, 0, 2, 1, 0, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 2, 0, 1,\n",
       "       0, 1, 1, 1, 2, 1, 1, 0, 1, 1, 1, 0, 0, 1, 2, 0, 1, 0, 1, 1, 1, 1,\n",
       "       1, 2, 0, 0, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 0,\n",
       "       2, 1, 2, 1, 1, 0, 1, 1, 2, 0, 1, 1, 1, 1, 1, 0, 1, 0, 1, 1, 0, 1,\n",
       "       1, 1, 2, 1, 0, 1, 0, 2, 2, 1, 1, 1, 0, 1, 0, 1, 1, 2, 1, 2, 1, 0,\n",
       "       1, 0, 1, 2, 1, 1, 0, 1, 1, 1, 1, 0, 1, 1, 0, 1, 0, 1, 1, 0, 1, 1,\n",
       "       1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 1, 1, 1, 2, 0, 2, 1, 0, 1,\n",
       "       1, 1, 1, 1, 0, 1, 2, 0, 1, 1, 1, 1, 0, 0, 1, 2, 1, 0, 0, 0, 1, 1,\n",
       "       1, 0, 1, 0, 2, 0, 0, 1, 2, 1, 0, 1, 0, 1, 1, 1, 1, 0, 1, 0, 1, 2,\n",
       "       2, 1, 0, 1, 1, 0, 1, 1, 2, 2, 0, 1, 1, 1, 1, 1, 0, 0, 1, 2, 0, 1,\n",
       "       1, 0, 2, 0, 1, 2, 2, 1, 1, 2, 1, 1, 0, 0, 1, 1, 1, 1, 1, 0, 2, 0,\n",
       "       1, 0, 1, 1, 1, 1, 0, 2, 1, 2, 1, 1, 1, 1, 1, 1, 2, 1, 1, 0, 1, 1,\n",
       "       1, 1, 1, 1, 1, 2, 1, 2, 1, 1, 1, 1, 1, 1, 0, 1, 1, 0, 1, 0, 1, 0,\n",
       "       1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 2, 1, 1, 0, 1,\n",
       "       1, 1, 1, 1, 2, 0, 0, 1, 0, 1, 0, 1, 1, 1, 1, 0, 0, 1, 1, 1, 0, 0,\n",
       "       1, 1, 2, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 0, 1, 1, 0, 0, 2, 0,\n",
       "       0, 1, 1, 1, 1, 0, 2, 1, 0, 1, 1, 0, 2, 2, 1, 0, 1, 0, 2, 0, 0, 1,\n",
       "       1, 1, 1, 1, 1, 1, 2, 1, 1, 0, 0, 1, 1, 0, 1, 1, 1, 1, 0, 1, 1, 0,\n",
       "       0, 1, 1, 1, 2, 1, 1, 1, 1, 1, 1, 0, 2, 1, 2, 1, 0, 1, 0, 0, 0, 1,\n",
       "       1, 2, 1, 0, 2, 2, 1, 0, 1, 0, 1, 1, 0, 1, 0, 0, 2, 0, 0, 1, 1, 0,\n",
       "       1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 2, 0, 0, 0, 0, 1, 0, 2, 0, 1, 1,\n",
       "       0, 1, 1, 1, 1, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 2, 1, 0,\n",
       "       1, 1, 1, 2, 0, 0, 1, 0, 1, 2, 1, 1, 1, 2, 1, 0, 1, 1, 1, 0, 1, 1,\n",
       "       2, 0, 1, 1, 1, 0, 1, 2, 1, 1, 1, 1, 0, 0, 0, 0, 1, 0, 1, 0, 1, 0,\n",
       "       0], dtype=int64)"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "rounded_labels=np.argmax(Y_test, axis=1)\n",
    "rounded_labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[149,  64,   2],\n",
       "       [ 85, 323,  18],\n",
       "       [ 15,  22,  49]], dtype=int64)"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import confusion_matrix\n",
    "cm = confusion_matrix(rounded_labels, yhat_classes)\n",
    "cm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUIAAAFNCAYAAAB1+2ZJAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAoaklEQVR4nO3dd5xU5fXH8c/ZZUGldxERUEEFKyqWaCygGEXRBBU1tpgfxhIllggodhITFY1GjUSN2FAUjYimGBILNkBCRKSIihFBmqKwwLLsnt8f9wID2TIMd7hzud+3r/ty5rntzOxw5jzPLWPujohImhXFHYCISNyUCEUk9ZQIRST1lAhFJPWUCEUk9ZQIRST1lAgjZmZzzKxnnrZ9uJnNzHi+m5n928yWmdllZvYHMxuSh/0ONrOHot5uvpnZKWb2hZktN7P9NmM708zsyOgi2/I2/uzIRty9oCdgDtCzmnmDgc+A5cBc4JmwfVrYthyoAFZlPB8MnAc4MGyj7Z0ctj9aQzyNgLuB/4bbmx0+b1FbvHl4bx4G7op4m0cCc7fg37dN+DrmA8uAGcBNQP0Itv0J0GdLvZYc4nst/Lzts1H7n8P2I7PcjgO7xv16kjwltiI0s3OBswmSTgPgAGAcgLt3dfcGYfubwKVrn7v7r8JNfAKcbmZ1MjZ7DjCrhn3WDffRFTiOICkeCiwBukf6ArPTniDpJ5KZNQPeAbYFDnH3hsAxQBNglwh2kYT3ZxbB5w4AM2sOHAwsimoHG33GpSpxZ+Isvu3mUEWFBfweuDuL9V8DfrpR23nAeOCvwAlhWzPgK+B2qqkIgZ8CC4AG2cRLkBzfAZYSVDy/B+qG8wy4C1gIfAt8AOwZzjse+IigQvoSuCpsP5KwWgP+yYbVbmfgUeDWjFj6AFOA7wgS/3Fh+/nA9HD7nwIXhu31gZVAJesr6B2AG4EnMrZ7EkGCWRq+v3ts9PqvCl/Pt8AzwDbVvFe3AlOBohrez0OBieG2JgKHbvS3vQV4K3wtfwdaAPXC2B0oBT4Jl9+gcsp8v8L1xoav6WuCL9CiKv6m9Qh6APPC6W6gXubfB7gy/LvOB86v5bN5fbhOcdh2KfBA2HZkFp+jNzJe53Lg9Iw4riH4TD/Ohp+dXcLX2C18vgOwmCwr0K1xSmxFCLwLnGNmV5vZAWZWnMM2HmP9t3E/4EWgrIblewJ/dfflWW6/AvgFwT+yQ4AewMXhvGOB7xMksCYEH+Al4byHCZJTQ2BPgqS3AXc/mg2r3Q0qWTPrHr6+q8Ptf5/gHzQE/0h7E1S05wN3mVk3dy8FfgDM8/UV9LyNttsZGAkMAFoCrwAvhdXyWqcRVMwdgb0Jvniq0hN43t0rq5oZVowvA/cAzYFhwMth1bTWmeFraAXUJfjSKPOgNwBBtzOb6vJKguTREmhNMIRS1fWn1xJUbPsC+xAkqesy5m8PNAbaAhcA95lZ0xr2O4/gS+/Y8Pk5BH+3TNV+jtz9++Ey+4R/r2cy4mhGUBX3z9yYu39CkCSfNLPtgD8RfPm/VkOcW7XEJkJ3fwL4OdALeB1YaGYDN3EzLwBHmlljqv4Abqw5wTdytjG+7+7vuvsad58DPAgcEc4uBxoCuwPm7tPdfX7GvC5m1sjdv3H3ydm/pHUuAB5x91fdvdLdv3T3GWFcL7v7Jx54naCSOjzL7Z4OvBxutxy4g6Bre2jGMve4+zx3/xp4iSBpVKW29/ME4GN3fzx8D0cSjCGemLHMn9x9lruvBEbVsK/alBOMV7Z393J3f9PDcmkjZwE3u/tCd19EMJ559kbbuTncxisEVdputez7MYIv9d2AJu7+TubMWj5H1akEbgi/FFZuPNPd/wh8DLwXvu5ra9neVi2xiRDA3Z90954EFc/PgJvNrNcmrL+SoOK4juBgx1u1rLKE4EOTFTPrbGZjzewrM/sO+BXBtzru/k+CLs59wAIzG25mjcJVf0TQPf7czF43s0Oy3WeGdgTd4ari+oGZvWtmX5vZ0nBfLbLc7g7A52ufhNXcFwQV0FpfZTxeATSgarW9nxvsK/R5jvuqze0EB77+bmaf1vClunFMn4dtay1x9zWbGNPzwNEEX+yPbzyzps9RDRa5+6palvkjQY/jXnevqSe01Ut0Ilwr/PZ9lnCcbRNXf4ygW/Q/H8Aq/APoZWb1s9z2AwQVTCd3b0TQ3bK1M939Hnffn+DgS2eCbizuPtHd+xB09/5MUOlsqi+o4oCDmdUDRhNUcq3dvQlB93ZtXLXdjmgeQXdr7faMIOl+mUOM/wBOMbPqPocb7Cu0U477giApbZfxfPu1D9x9mbtf6e47E1ScV5hZjyxi2ilsy5m7rwD+AlxE1Z/DGj9H1W22pplm1oBgfPNh4MZwGCK1kpIIS8xsm4ypjpmdZ2YnmFlDMysysx8QJJT3NnHbrxMcqbw3i2UfJ0gwo81s93C/zcPz7I6vYvmGBAcqlpvZ7gQfdADM7EAzO8jMSggGulcBFWZW18zOMrPGYdfzO4Ixok31MHC+mfUI42wbxlCXYMB/EbAmfN+OzVhvAdA8HC6oyijghHC7JQRfImXA2znEOIxgnHKEmbUHCOMcZmZ7EyTozmZ2Zvg3Px3oQnBQIxdTgDPNrNjMjiOje2lmvc1s1zCxr33Pq3rfRwLXmVlLM2tBcLDjiRzjyTQYOCLs+m6s2s9RaAGw8ybu73fA++7+U4Je0R82cf2tSlIS4SsERzPXTjcSfDAGE5zPtxT4LXCRu4/flA2H42TjwvGs2pYtIxjgnwG8GsYwgaCbUlUCvopgMH8ZQTfkmYx5jcK2bwi6V0sIqjQIxpzmhN2gnwE/3pTXFMY6gfBACMER19cJxr+WAZcRJLRvwvjGZKw3g+Af+6dmttTMdthouzPDeO4lONJ4InCiu6/OIcavCcYWy4H3zGwZwelJ3wKz3X0JwUGdKwnen18Cvd198abuK3R5GO9SgrG+P2fM60RQoS4nOEJ7fzUHD24FJhH0PqYCk8O2zRKOqVb32a3pcwTBv4cR4d/rtNr2ZWZ9CA5m/SxsugLoZmZn5RL71sCqHg8WEUmPpFSEIiJ5o0QoIqmnRCgiqadEKCKpp0QoIqlXsHelsNpP7BWRPPDaT9au0knWO6d/s2N8bE77i1LBJkKAe57I5RLbdLnsx92YPGdJ7QumXLcOwX0aVlZUeX8HyVScvo5iQSdCEUmOogSPtCkRikgkgqsTk0mJUEQioYpQRFKvSBWhiKSdqSIUkbRTRSgiqaeKUERSTxWhiKSejhqLSOrpPEIRST1VhCKSehojFJHU01FjEUm9omp/nrrwJTdyESkoluN/tW43+C3zCWb2HzObZmY3he3NzOxVM/s4/H/TjHUGmdlsM5tpZr1q24cSoYhEosiKcpqyUAYc7e77APsCx5nZwcBAYJy7dyL4PeyBAGbWBegHdCX4/eb7zay4xthzfdEiIluCB5aHT0vCyYE+wIiwfQRwcvi4D/C0u5e5+2fAbKB7TftQIhSRSBRhOU1m1t/MJmVM/TfetpkVm9kUYCHwqru/B7R29/kA4f9bhYu3Bb7IWH1u2FYtHSwRkUjketTY3YcDw2tZpgLY18yaAC+Y2Z41hlLFJmravhKhiERiS5xH6O5Lzew1grG/BWbWxt3nm1kbgmoRggqwXcZqOwLzatquusYiEomiHP+rjZm1DCtBzGxboCcwAxgDnBsudi7wYvh4DNDPzOqZWUegEzChpn2oIhSRSOTxWuM2wIjwyG8RMMrdx5rZO8AoM7sA+C9wKoC7TzOzUcBHwBrgkrBrXS0lQhGJRL6uNXb3D4D9qmhfAvSoZp2hwNBs96FEKCKR0LXGIpJ6utZYRFJPFaGIpJ4qQhFJvSTffUaJUEQiUZTFnWQKlRKhiETCVBGKSNqpIhSR1NMYoYikXjZ3my5UyU3hIiIRUUUoItEoSm5FqEQoItHQlSUiknamilBEUk8VoYiknirCrUOPg3eiw46NWblqDU+Nnb7BvP32aMVh++/IH5/9D6vKKigqMo46aCdaNdsOcN6YNJcvFyyvesNbudLly3jwrl8zd86nYMbPrhhM5y57AfDSs0/x5EO/Z/ioV2jUuEm8gRaQr+bP59pBA1myeDFmRt/TTuOss8+JO6zNo0S4dZj+6dd8MGsRxxzaYYP2BtuV0K5NI75bXraureuuzQEY+fJ0tq1Xh5OO3pVn/jJjS4ZbMEY8cDf7HnAwVwz5FWvKyykrWwXA4oULmPrvCbRo1TrmCAtPcZ1irvrlL9mjS1dKS0vp1/dHHHzIoeyy665xh5azPN6qP+/ydh6hme1uZteY2T1m9rvw8R752l8U5i1czqqy//1pg8P335G3J3+5QVuzxtsy96tlAKwsW0PZ6jW0br7dFomzkKwoLWX61CkcddyJANQpKaF+g4YAPPbg7zjrgksSPXaULy1btmKPLl0BqF+/PjvvvAsLFy6IOarNVGS5TQUgL4nQzK4Bnib4fdEJwMTw8UgzG5iPfeZLxx0bs3xlOYuXrtygffE3K+i4Y2PMoFH9urRqvh0NtqsbU5TxWfjVlzRq3IQH7hzKwIvP5cG7fs2qVSuZ9M6bNGvRkva7dIo7xIL35ZdfMmP6dPbae5+4Q9k8ZrlNBSBfXeMLgK7uXp7ZaGbDgGnAbVWtFP7CffAr9w8+CNsdkKfwslOn2Dhgz+15cdzH/zPvo0+W0LTxNpz+g91ZVrqa+YtKqfQaf0N6q1RRUcFns2dx3iVX0Gn3rjz6wF089/jDTJ86hWt/fXfc4RW8FaWlXHn5ZVw9aCANGjSIO5zNUyDVXS7ylQgrgR2AzzdqbxPOq1LmL94bOE9MzlN42WncsB6NGtTljBOCHn2D7erS7/g9GPWXGaxYtYbx76/vLvft1Zmly8qq29RWq3mLVjRr2ZJOuwfdvIMOO4rnnniYRV/N45cXBYP/Xy9axKBLzmfoPQ/RpFnzOMMtKOXl5Vwx4HKO730iPY85Nu5wNp9uuvA/BgDjzOxj4IuwbSdgV+DSPO0zckuWruLh56aue37uyV155i8zWFVWQZ1iA4w1FZW0274hlZXON9+uii/YmDRp1pzmLVoz74vP2aFdez6cMomOu+7GkN/cu26ZS8/5Ib+69xEdNc7g7tw45Dp23nlnzjnvvLjDiYROqN6Iu//VzDoD3YG2BOODc4GJtf3Qcpx6HdaBtq0bsk29Opx/yp6898F8PvpkSZXLbrtNCX167Io7lK5Yzatvb1z8psf5l/yC3//mJtasKafV9jvwsyuvjTukgvfvyZMZO2YMnTp35rRTTgHg5wMGcPgRR8Qc2WZIcCI0L9BxLQO/J+aucRJc9uNuTJ5TdbKW9bp1CLrkKyuqHZmR0DbFuWW0oZ3vyCmZXDvrqtgzqM4jFJFoJLgiVCIUkUjohGoRkQRTRSgi0VDXWERSL8FdYyVCEYmGKkIRST0lQhFJOx01FhHJ0224zKydmf3LzKab2TQzuzxsv9HMvjSzKeF0fMY6g8xstpnNNLNete1DFaGIRCN/FeEa4Ep3n2xmDYH3zezVcN5d7n7HhmFYF6Af0JXg5i//MLPONV3eq0QoItHI0xihu88H5oePl5nZdIJ7GFSnD/C0u5cBn5nZbIL7HrxT3QrqGotIJMwsp2kT99EB2A94L2y61Mw+MLNHzKxp2NaW9Xe9guCGLzUlTiVCEYlIjmOEZtbfzCZlTP2r2ryZNQBGAwPc/TvgAWAXYF+CivHOtYtWsXqNN4RQ11hEopFj1zjzhszVMbMSgiT4pLs/H663IGP+H4Gx4dO5QLuM1XcE5tW0fVWEIhKNPP1miQX954eB6e4+LKO9TcZipwAfho/HAP3MrJ6ZdQQ6Efx2UrVUEYpINPJ3QvX3gLOBqWY2JWwbDJxhZvsSdHvnABcCuPs0MxsFfERwxPmS2m4IrUQoIpHI1wnV7j6eqsf9XqlhnaHA0Gz3oUQoItHQJXYiknq6xE5EJLlUEYpINNQ1FpHUS24eVCIUkYgkeIxQiVBEImHqGotI6iU3DyoRikhE1DUWkdRT11hEUi+5eVCJUEQioq6xiKRegq9TUyIUkWioIhSRtEvy7xorEYpINJKbB5UIRSQiOn1GRFJPXWMRSb3k5kElQhGJSIK7xgk+80dEJBqqCEUkGsktCJUIRSQiOliSH5f9uFvcISRCtw7N4w4hMbYt1mhQbTzH9XRj1jwZ98G8uEMoeD323oETrXfcYRS8l3wsAF+vLI85kgTYtiS39ZKbBws7EYpIgqhrLCKpp66xiKRecvOgEqGIRERdYxFJvQQfkFciFJFoqCIUkbTTjVlFRBLcNU5w6CJSUMxym2rdrLUzs3+Z2XQzm2Zml4ftzczsVTP7OPx/04x1BpnZbDObaWa9atuHEqGIRCNPiRBYA1zp7nsABwOXmFkXYCAwzt07AePC54Tz+gFdgeOA+82suKYdKBGKSDSKcpxq4e7z3X1y+HgZMB1oC/QBRoSLjQBODh/3AZ529zJ3/wyYDXSvLXQRkc2Xv4owYxfWAdgPeA9o7e7zIUiWQKtwsbbAFxmrzQ3bqqVEKCKxMrP+ZjYpY+pfzXINgNHAAHf/rqZNVtFW4011dNRYRKKR4+kz7j4cGF7zpq2EIAk+6e7Ph80LzKyNu883szbAwrB9LtAuY/UdgRpvZaWKUESikacxQgtOUHwYmO7uwzJmjQHODR+fC7yY0d7PzOqZWUegEzChpn2oIhSRaOTvhOrvAWcDU81sStg2GLgNGGVmFwD/BU4FcPdpZjYK+IjgiPMl7l5R0w6UCEUkGnlKhO4+nurvbdOjmnWGAkOz3YcSoYhEI8EDbUqEIhINXWssIqmnRCgiqaeusYiknipCEUm95OZBJUIRiYh+xU5EUk9dYxFJveTmQSVCEYmIusYiknrqGotI6iU3Dyb5FEgRkWjUWhGaWT13L6utTURSLsFjhNlUhO9k2SYiabYFfrMkX6qtCM1se4IfPNnWzPZj/QhAI2C7LRCbiCRJYeS0nNTUNe4FnEdwv/87Wf8ylxHcHVZEZL0Ed42rTYTuPgIYYWY/cvfRWzAmEUmiAunm5iKbMcIdzayRBR4ys8lmdmzeIxORZMnTjzdtCdmE8ZPwN0SPJfgB5fMJfjRFRGS9rfFgSYa1kR4P/Mnd/xP+vJ6IyHoJTgvZJML3zezvQEdgkJk1BCrzG5aIJE6BdHNzkU0ivADYF/jU3VeYWXOC7vFWb9zYZ3lr3MtgRtudduaci6/hb39+ivH/eJmGjRoD0OfMn7Jnt4NjjnTLKqlXwq/f+A0l9UoorlPEW8+9xcgbn+K8355P9xO7s2b1GuZ/8hX3nH83pd+W0unAzlwy/FIAzIyRNz7Fu39O36mot95wHW+98QZNmzXjqdF/BmDWjBn8ZujNrC4ro7hOMVcPGkLXvfaKN9BcbeUVoQNdgN7AzUB9YJt8BlUIli5ZxL9eeZ7r73qUuvXq8cdhNzLprX8C0KN3X4456fSYI4xPeVk51x09mFWlqyiuU8xt43/L5L+8z5RXp/DYoBFUVlRy7m3n0XfQqYwY+Ciff/g5VxwwgMqKSppu35Tf/edeJrz0HpUV6epYnHDSyfTtdyY3X7f+7LPf330nF1x4EYcedjhvv/kGv7/7Th54+NH4gtwcCU6E2RSz9wOHAGeEz5cB9+UtogJSWVlB+eoyKioqWF1WRuNmzeMOqWCsKl0FQHFJHeqUFOPuTHn13+uS28x3Z9J8xxYArF5Ztq697jZ1wT2eoGO23/4H0CjsSaxlZpSWLgdg+fLltGzZKo7QopHgo8bZVIQHuXs3M/s3gLt/Y2Z1c92hmZ3v7n/Kdf0tpUnzlvQ88TSuveh0SurWY499DqDLPgfy6cxpvPbXF3jv9b+z0y6d+dE5F1O/QcO4w93iioqKGPb+3bTZtQ2v3PcysybM2mB+z58cw/hn3lj3vHP3zlz2yOW0bN+Ku84elrpqsDoDrr6GARdfyL3D7sArneEjnog7pNxt5RVhuZkVE3SRMbOWbN7Bkps2Y90tpnT5Mv4z8W1uuW8ktw1/jtVlq3jvjVf5/rEnccu9TzL49j/SuElzRj92f9yhxqKyspIB+13GT3Y8j07dO7NT1/br5p06+DQq1lTw2pOvrWubNWEWl+55CVce+Av6DjqVknolMURdeJ5/9hkuv+oaxvxtHJdf9UuG3nR93CHlLsGnz2STCO8BXgBamdlQYDzw65pWMLMPqpmmAq1rWK+/mU0ys0kMH74pryNyM6a+T4tW29OwcROK69Rh34MO59OZH9KoSTOKiospKirisJ69mTN7Rqxxxq3021I+fG0q3Y7rBsDR5xzNgb27c+dZd1S5/NwZc1lVuor2e7avcn7avPLSGI7q0ROAHsf24qMPp8Yc0WbYmrvG7v6kmb0P9CA4p/Bkd59ey2qtCa5V/majdgPermFfw4Hh4YLOB/NqCy9vmrVoxWcff8TqslWU1K3HjKmTab/Lbnz7zRIaNw3GCqdMeJMd2nWMLca4NGrRiIryCkq/LaXuNnXZp+e+jP7Nc3Tr1Y0fXtOXwUcMZPXK9Xdpa92hNYu+WERlRSUtd2pJ293asmDOwhhfQeFo0bIlkydNZP8DuzNpwnu02ym5XxBJPr04m/sRPu7uZwMzqmirzliggbtPqWJ7r+UQ5xbXsVMX9jv4CH71y/4UFRfTrkMnDuvZmyceuIO5c2ZjZjRruT1nXXhF3KFucc3aNGPAiF9QVFyEFRUxftSbTHp5Ig9+PJw69Uq4+dVbgeCAyQMX3cceh3VhyMC+rCmvwCsr+cPFD7BsyXcxv4otb8jAq5k8aSJLly7lxGN78H8XXcyg62/irt/eRkXFGurWrcegITfEHWYqmddyBM/MJrt7t4znxcBUd++S18DAx8VYESZFj7134ETrHXcYBe8lHwvA1yvLY46k8DXdtiSn0m7Y8PdyOh3giv4HxV5KVttDN7NBZrYM2NvMvgunZcBC4MUtFqGIJEKCj5VUnwjd/dfu3hC43d0bhVNDd2/u7oO2YIwikgBmltNUCLI5WKKkJyK1K5AjwLlIcOgiUkjyVRGa2SNmttDMPsxou9HMvjSzKeF0fMa8QWY228xmmlmvbGKvaYzwFTPrkM1GRETyOEj4KHBcFe13ufu+4fRKEIJ1AfoBXcN17g8P8NaoporwUeDvZnatmekyABGpUb7yoLu/AXydZRh9gKfdvczdPwNmA91rW6mmgyWjgP0IfrVukpldZWZXrJ2yDEpE0iLHTJh5RVk49c9yj5eGV6w9YmZNw7a2wBcZy8wN22pU28GScqAUqAfohqwiUi3L8VfsMq8o2wQPALcQ3APhFoJf2vwJVf+oaK3nN9b0u8bHAcOAMUA3d1+xiYGKSJpswTNh3H3But2a/ZHgajYIKsB2GYvuCNR6ZUZNFeG1wKnuPi2HOEUkZbbkOYFm1sbd54dPTwHWHlEeAzxlZsOAHYBOwITatlfT7xofvpmxikiK5CsPmtlI4EighZnNBW4AjjSzfQm6vXOACwHcfZqZjQI+AtYAl7h7RW37yObGrCIitctTJnT3M6pofriG5YcCQzdlH0qEIhKJQrlcLhdKhCISjQRfp6ZEKCKRSHJFmOAcLiISDVWEIhKNBFeESoQiEokE50ElQhGJRpLHCJUIRSQaCT7ioEQoIpFQRSgiokQoImmX4DyoRCgiEUlwJlQiFJFI5Hpj1kKgRCgikUhwQahEKCIRSXAmVCIUkUjo9BkRkeTmQSVCEYmGDpaISOolNw0qEYpIRDRGKCKpl+A8mOT7RYiIREMVoYhEIskVoRKhiETCEny4RIlQRCKhilBEUk+JUERST6fPiEjqJTcNFngi7LH3DnGHkAgv+di4Q0iMZtuWxB1CwfMc11NFKCKpl+A8WNiJcOmq8rhDKHhNtilhcenquMMoeC3q1wXg4wXLYo4kAVo3zGm1BOfBwk6EIpIc6hqLSOolOA/qWmMRiYaZ5TRlsd1HzGyhmX2Y0dbMzF41s4/D/zfNmDfIzGab2Uwz65VN7EqEIhIJy3HKwqPAcRu1DQTGuXsnYFz4HDPrAvQDuobr3G9mxbXtQIlQRCJhlttUG3d/A/h6o+Y+wIjw8Qjg5Iz2p929zN0/A2YD3Wvbh8YIRSQSW/hgSWt3nw/g7vPNrFXY3hZ4N2O5uWFbjVQRikgkcu0am1l/M5uUMfXfzDA2Vus54qoIRSRW7j4cGL6Jqy0wszZhNdgGWBi2zwXaZSy3IzCvto2pIhSRSORrjLAaY4Bzw8fnAi9mtPczs3pm1hHoBEyobWOqCEUkEvkaIzSzkcCRQAszmwvcANwGjDKzC4D/AqcCuPs0MxsFfASsAS5x94ra9qFEKCKRyNexEnc/o5pZPapZfigwdFP2oUQoIpHQrfpFJPWSfImdEqGIREKJUERSr0hdYxFJO1WEIpJ6SoQiknq6MauIpF5y06ASoYhERBWhiKRegvOgEqGIREOJUERST5fYiUjqJbki1P0IRST1VBGKSCR01FhEUi/BeVCJUESioYpQRFIvuWlQiVBEIpLgglCJUESioa6xiKRegvOgEqGIRENXlohI6qkiFJHU0xihiKRegvOgEqGIREOJUERSTwdLRCT1VBFuhW65/jrGv/EGTZs14+nn/wzA8Afu48XRo2nSrCkAF//8cr53+PdjjDJ+C776iluuH8ySxYspKiripB/25fQzf8zv77qT8W++RkmdEtq2a8e1N95Cw4aN4g43dhUVFQzofzbNW7Tixt/czaezZ3Hfnb9m5YoVtG6zA1cPuYXt6jeIO8ycJPlgie5HWI0T+pzM7x74w/+0n3H22Tw5ajRPjhqd+iQIUFxczM9/cRUjnx/D8BFP8vyop/ns00848OBDeGLUCzw+6nna7dSexx55KO5QC8KY50bSrn3Hdc/v+e2tnHfhpdw/4hkOOfxIRo98PMboNo9ZblMhyFsiNLPdzayHmTXYqP24fO0zSt32P4BGjRrHHUbBa9GyJbvt0QWA+vXr075jRxYtXMBBhxxKnTpBh2PPvfZh0cIFcYZZEBYvXMDEd96i1wknr2ub+9/P2XOfbgDsd8BBvPX6P2OKLt3ykgjN7DLgReDnwIdm1idj9q/ysc8t5dmnR3Jm31O45frr+O67b+MOp6DMn/clH8+cQdc9996gfeyLL3DwoYfFFFXhGH7vnZx/0WVY0foyqH3HXXh3/OsAjH/tHyxO8BeG5fhfIchXRfh/wP7ufjJwJDDEzC4P5xXGK8/Bj047nefH/oUnRo2mecuW/O6O2+MOqWCsWLGCwVf9gsuvvIb6DdZ3Ah59aDjFdYrpdXzvGKOL34S336Rx02Z02m2PDdoHDLyel194lst++mNWrlhBnZKSmCLcfEnuGufrYEmxuy8HcPc5ZnYk8JyZtaeGRGhm/YH+ADz4IJxzfp7Cy03z5i3WPT75h3254ueXxBhN4VhTXs7gq37BscefwJE9eq5rf+WlF3nrzde59w8PJXogPQofTf0P7731BpPefYvVq1ezsnQ5t98yhKuH3MKtw+4D4MsvPmfiO+NjjjR3RXn8G5vZHGAZUAGscfcDzKwZ8AzQAZgDnObu3+Sy/Xwlwq/MbF93nwLg7svNrDfwCLBXdSu5+3BgOICBs6o8T+HlZvGiRbRo2RKA1/45jl123TXmiOLn7vzq5hvo0HFnzvjxueva331rPE88+gj3PfQnttl22xgjLAznXXgp5114KQAf/HsSzz/9BFcPuYWl33xNk6bNqKys5OnHHuYHfX4Uc6S52wLfdUe5++KM5wOBce5+m5kNDJ9fk8uG85UIzwHWZDa4+xrgHDN7ME/7jNR111zN+5MmsnTpUnof04P/u+hiJk+ayKyZMzGDNju0ZdCQG+IOM3YfTPk3f335JXbZtRPn9usLwIWXXsZdv72N8vLVDLgoKPC77rU3v7z2+jhDLUiv/+NvjH3hWQAO/f5RHHP8STFHlLsYiv4+BENvACOA18gxEZq7RxNSxAx8aYFVhIWoyTYlLC5dHXcYBa9F/boAfLxgWcyRFL5dWzfMKaXNmv9dTsmkc5tGte7PzD4DvgEceNDdh5vZUndvkrHMN+7eNJcYdEK1iEQi14pwg2MDgeHhMFmm77n7PDNrBbxqZjNy21vVlAhFJBK5HhDLPDZQwzLzwv8vNLMXgO7AAjNr4+7zzawNsDCnANCVJSISkXydPmNm9c2s4drHwLHAh8AYYO0RunMJzl3OiSpCEYlEHk+Rag28EG6/DvCUu//VzCYCo8zsAuC/wKm57kCJUEQika806O6fAvtU0b4E6BHFPpQIRSQSST5pXolQRCKR4DyoRCgi0UhwHlQiFJGIJLgkVCIUkUgkNw3qPEIREVWEIhKNBPeMlQhFJBoJzoNKhCISkQSXhEqEIhKJ5KZBJUIRiUiCC0IlQhGJSnIzoRKhiERCFaGIpF6C86ASoYhEQxWhiEiCa0IlQhGJhCpCEUm9BOdBJUIRiUiCM6ESoYhEwhKcCZUIRSQSGiMUkdRLcB5UIhSRiCS4JNQdqkUk9VQRikgkklsPKhGKSEQS3DNWIhSRaCQ4DyoRikhEElwSKhGKSCSSmwaVCEUkIgkuCJUIRSQqyc2ESoQiEglVhCKSegnOg0qEIhINVYQiIgmuCc3d446hSgaFGZjIVs5zzGirKipz+je7TXFR7Bm0YBNhITKz/u4+PO44kkDvVXb0PhUG3X1m0/SPO4AE0XuVHb1PBUCJUERST4lQRFJPiXDTaCwne3qvsqP3qQDoYImIpJ4qQhFJPSXCLJnZcWY208xmm9nAuOMpVGb2iJktNLMP446lkJlZOzP7l5lNN7NpZnZ53DGlmbrGWTCzYmAWcAwwF5gInOHuH8UaWAEys+8Dy4HH3H3PuOMpVGbWBmjj7pPNrCHwPnCyPlPxUEWYne7AbHf/1N1XA08DfWKOqSC5+xvA13HHUejcfb67Tw4fLwOmA23jjSq9lAiz0xb4IuP5XPShlYiYWQdgP+C9mENJLSXC7FR1LaTGFGSzmVkDYDQwwN2/izuetFIizM5coF3G8x2BeTHFIlsJMyshSIJPuvvzcceTZkqE2ZkIdDKzjmZWF+gHjIk5JkkwMzPgYWC6uw+LO560UyLMgruvAS4F/kYwqD3K3afFG1VhMrORwDvAbmY218wuiDumAvU94GzgaDObEk7Hxx1UWun0GRFJPVWEIpJ6SoQiknpKhCKSekqEIpJ6SoQiknpKhClngfFm9oOMttPM7K953GeHtXenMbMDzOyeWpYfnK9YRECnzwhgZnsCzxJc71oMTAGOc/dPNnE7dcJzLmtbrgMwNtu705jZcndvsCmxiGwK/cC74O4fmtlLwDVAfYJbaG2QBM1sOfAgcBTwDdDP3ReZ2WvA2wQnCI8Jnw8DGgCLgfPcfb6Z7Q88AqwAxmds90jgKnfvHV53ey9wAMG13DcBBwLbmtkUYJq7n5WP90DSTYlQ1roJmAysJkhEG6sPTHb3K83seuAGgqttAJq4+xHhtbOvA33CJHk6MBT4CfAn4Ofu/rqZ3V5NDEOAb919LwAza+ruo83sUnffN6LXKfI/lAgFAHcvNbNngOXuXlbFIpXAM+HjJ4DMmwSsbd8N2BN4NbiUlmJgvpk1JkiWr4fLPQ78gP/Vk+A67rUxfZPjyxHZJEqEkqkynLKRObhcGv7fCLqvh2QuaGZNyO62ZZblciKR0lFjyVYR0Dd8fCYZ43wZZgItzewQCG4zZWZd3X0p8K2ZHRYuV904399Z393GzJqGD8vDbrdIXigRSrZKga5m9j5wNHDzxguEP2PQF/iNmf2H4OjzoeHs84H7zOwdYGU1+7gVaGpmH4brHxW2Dwc+MLMno3oxIpl0+oxkRaewyNZMFaGIpJ4qQhFJPVWEIpJ6SoQiknpKhCKSekqEIpJ6SoQiknpKhCKSev8PFxfG1k2+ICUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 360x360 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# The confusion matrix\n",
    "from sklearn.metrics import confusion_matrix\n",
    "import seaborn as sns\n",
    "\n",
    "lstm_val = confusion_matrix(rounded_labels, yhat_classes)\n",
    "f, ax = plt.subplots(figsize=(5,5))\n",
    "sns.heatmap(lstm_val, annot=True, linewidth=0.7, linecolor='cyan', fmt='g', ax=ax, cmap=\"BuPu\")\n",
    "plt.title('LSTM Classification Confusion Matrix')\n",
    "plt.xlabel('Y predict')\n",
    "plt.ylabel('Y test')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4/4 [==============================] - 0s 11ms/step - loss: 1.0998 - acc: 0.7607\n",
      "score: 1.10\n",
      "acc: 0.76\n"
     ]
    }
   ],
   "source": [
    "# Validation\n",
    "validation_size = 610\n",
    "\n",
    "X_validate = X_test[-validation_size:]\n",
    "Y_validate = Y_test[-validation_size:]\n",
    "X_test = X_test[:-validation_size]\n",
    "Y_test = Y_test[:-validation_size]\n",
    "score,acc = model.evaluate(X_test, Y_test, verbose = 1, batch_size = batch_size)\n",
    "\n",
    "print(\"score: %.2f\" % (score))\n",
    "print(\"acc: %.2f\" % (acc))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save the model\n",
    "model.save('Mymodel.h5')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[9.8723638e-01 4.1791354e-04 1.2345733e-02]] 0\n"
     ]
    }
   ],
   "source": [
    "# Test with new and different data set aside from the data to build the model.\n",
    "# Test 1\n",
    "message = ['Congratulations! you have won a $1,000 Walmart gift card']\n",
    "seq = tokenizer.texts_to_sequences(message)\n",
    "\n",
    "padded = pad_sequences(seq, maxlen=X.shape[1], dtype='int32', value=0)\n",
    "\n",
    "pred = model.predict(padded)\n",
    "\n",
    "labels = ['0','1','2']\n",
    "print(pred, labels[np.argmax(pred)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[9.9999762e-01 1.0559489e-06 1.3480803e-06]] 0\n"
     ]
    }
   ],
   "source": [
    "# test 2\n",
    "message = ['such massive asteroid hit will certainly create new business opportunities']\n",
    "seq = tokenizer.texts_to_sequences(message)\n",
    "\n",
    "padded = pad_sequences(seq, maxlen=X.shape[1], dtype='int32', value=0)\n",
    "\n",
    "pred = model.predict(padded)\n",
    "\n",
    "labels = ['0','1','2']\n",
    "print(pred, labels[np.argmax(pred)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[7.2783207e-10 1.0000000e+00 4.0219553e-10]] 1\n"
     ]
    }
   ],
   "source": [
    "# test 4\n",
    "message = ['so does anyone else not open Snapchat anymore ? or is it just me ... ugh this so sad.']\n",
    "seq = tokenizer.texts_to_sequences(message)\n",
    "\n",
    "padded = pad_sequences(seq, maxlen=X.shape[1], dtype='int32', value=0)\n",
    "\n",
    "pred = model.predict(padded)\n",
    "\n",
    "labels = ['0','1','2']\n",
    "print(pred, labels[np.argmax(pred)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[9.9952853e-01 3.7651797e-04 9.5002018e-05]] 0\n"
     ]
    }
   ],
   "source": [
    "# Test 5\n",
    "message = ['@elonmusk had a terrible experience with a very pushy sales guy from tesla Stanford shop while shopping for model x']\n",
    "seq = tokenizer.texts_to_sequences(message)\n",
    "\n",
    "padded = pad_sequences(seq, maxlen=X.shape[1], dtype='int32', value=0)\n",
    "\n",
    "pred = model.predict(padded)\n",
    "\n",
    "labels = ['0','1','2']\n",
    "print(pred, labels[np.argmax(pred)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[6.4313441e-05 9.9993515e-01 4.8162610e-07]] 1\n"
     ]
    }
   ],
   "source": [
    "# test 6\n",
    "message = ['The local electronics industry is expected to remain stable amid layoff concerns surrounding Japanese electronics giants operating in the country, an official says.']\n",
    "seq = tokenizer.texts_to_sequences(message)\n",
    "\n",
    "padded = pad_sequences(seq, maxlen=X.shape[1], dtype='int32', value=0)\n",
    "\n",
    "pred = model.predict(padded)\n",
    "\n",
    "labels = ['0','1','2']\n",
    "print(pred, labels[np.argmax(pred)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[1.7219327e-06 2.2136858e-07 9.9999809e-01]] 2\n"
     ]
    }
   ],
   "source": [
    "# Test 7\n",
    "message = ['The local electronics industry is amid layoff concerns and last year has laid off tens of employees']\n",
    "seq = tokenizer.texts_to_sequences(message)\n",
    "\n",
    "padded = pad_sequences(seq, maxlen=X.shape[1], dtype='int32', value=0)\n",
    "\n",
    "pred = model.predict(padded)\n",
    "\n",
    "labels = ['0','1','2']\n",
    "print(pred, labels[np.argmax(pred)])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**References**\n",
    "\n",
    "    [1] Artificial Inteligence, Machine Learning and Deep Learning by Mercury Learning and Informarion (Book)\n",
    "    [2] Source Information of NLP :https://monkeylearn.com/what-is-text-classification/\n",
    "    [3] Parameter doc2vec and LSTM inspired by : https://dl.acm.org/doi/10.1145/3406601.3406624\n",
    "    [4] (Bing Liu. Sentiment Analysis and Opinion Mining, Morgan & Claypool Publishers, May 2012.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Tips**\n",
    "    There are some changes in `Doc2Vec` so check youre python version and work on it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Learning Continued"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}