nlapi/masterclass/sentiment_analysis_nlapi_v2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img style=\"float: right;\" src=\"https://docs.expert.ai/logo.png\" width=\"150px\">\n",
    "\n",
    "\n",
    "# Sentiment Analysis with expert.ai NLAPI\n",
    "expert.ai Natural Language API v2 introduces a sentiment analysis feature that identifies the sentiment polarity for text document such as positive, negative, overall and its associated intensity.\n",
    "\n",
    "## Installation and Setup\n",
    "First, install expert.ai-nlapi library using pip.\n",
    "* https://pypi.org/project/expertai-nlapi/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 142,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: expertai-nlapi in c:\\python38\\lib\\site-packages (2.1.3)\n",
      "Requirement already satisfied: requests in c:\\users\\abelli\\appdata\\roaming\\python\\python38\\site-packages (from expertai-nlapi) (2.24.0)\n",
      "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in c:\\users\\abelli\\appdata\\roaming\\python\\python38\\site-packages (from requests->expertai-nlapi) (1.25.10)\n",
      "Requirement already satisfied: chardet<4,>=3.0.2 in c:\\users\\abelli\\appdata\\roaming\\python\\python38\\site-packages (from requests->expertai-nlapi) (3.0.4)\n",
      "Requirement already satisfied: idna<3,>=2.5 in c:\\users\\abelli\\appdata\\roaming\\python\\python38\\site-packages (from requests->expertai-nlapi) (2.10)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in c:\\users\\abelli\\appdata\\roaming\\python\\python38\\site-packages (from requests->expertai-nlapi) (2020.6.20)\n"
     ]
    }
   ],
   "source": [
    "!pip install -U expertai-nlapi"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Working with NL API in Python\n",
    "First you have to setup your account credentials; if you don't have them, get them at https://developer.expert.ai/ui/login\n",
    "Set your environment variables with your credentials \n",
    "\n",
    "```bash\n",
    "SET EAI_USERNAME=YOUR_USER\n",
    "SET EAI_PASSWORD=YOUR_PASSWORD\n",
    "```\n",
    "\n",
    "as an alternative you can always add to your notebook the following statements\n",
    "\n",
    "```python\n",
    "import os\n",
    "os.environ[\"EAI_USERNAME\"] = 'YOUR_USER'\n",
    "os.environ[\"EAI_PASSWORD\"] = 'YOUR_PASSWORD'\n",
    "```\n",
    "\n",
    "\n",
    "\n",
    "## Initialize the library\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 143,
   "metadata": {},
   "outputs": [],
   "source": [
    "from expertai.nlapi.cloud.client import ExpertAiClient\n",
    "client = ExpertAiClient()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 144,
   "metadata": {},
   "outputs": [],
   "source": [
    "text = '''Michael Jordan was one of the best basketball players \n",
    "of all time. Scoring was Jordan's stand-out skill, but he still holds a defensive NBA record.''' \n",
    "language= 'en'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As today, v2 provides the sentiment analysis feature in English only\n",
    "\n",
    "## Let's play with it\n",
    "\n",
    "Sentiment is availabble throught the `specific_resource_analysis` with the resource type `sentiment`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 145,
   "metadata": {},
   "outputs": [],
   "source": [
    "document = client.specific_resource_analysis(\n",
    "    body={\"document\": {\"text\": text}}, \n",
    "    params={'language': language, 'resource': 'sentiment'\n",
    "})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sentiment feature takes in a string and returns a score for:\n",
    "* negativity\n",
    "* positivity\n",
    "* overall (computed by normalizing the scores above)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 146,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "10.39"
      ]
     },
     "execution_count": 146,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "document.sentiment.positivity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 147,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 147,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "document.sentiment.negativity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 148,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "10.39"
      ]
     },
     "execution_count": 148,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "document.sentiment.overall"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "During the sentiment analysis process, the terms that contributed to the score generation are reported, with their score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 149,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TERM                 SENTIMENT\n",
      "['but']                      -1\n",
      "basketball player             5\n",
      "scoring                     9.5\n",
      "skill                       3.5\n"
     ]
    }
   ],
   "source": [
    "print (f'{\"TERM\":{20}} {\"SENTIMENT\":{4}}')    \n",
    "\n",
    "for item in document.sentiment.items:\n",
    "    if not item.lemma:\n",
    "        group_lemma=[]\n",
    "        for nested in item.items:\n",
    "            group_lemma.append(nested.lemma)\n",
    "        print (f'{str(group_lemma):{20}} {item.sentiment:{10}}')\n",
    "    else:\n",
    "        print (f'{item.lemma:{20}} {item.sentiment:{10}}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sentiment Analysis on the movie review dataset\n",
    "\n",
    "Let's apply now the  sentiment analysis to a dataset, i.e. the movie reviews dataset: we'll use a subset of the Cornell University Movie Review polarity dataset v2.0 obtained from http://www.cs.cornell.edu/people/pabo/movie-review-data/\n",
    "\n",
    "The dataset contains the text of 2000 movie reviews: 1000 are positive, 1000 are negative, and the text has been preprocessed as a tab-delimited file. For demo purposes, We use a subset of this dataset, limited to 100 reviews per class.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 150,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>label</th>\n",
       "      <th>review</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>neg</td>\n",
       "      <td>how do films like mouse hunt get into theatres...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>neg</td>\n",
       "      <td>some talented actresses are blessed with a dem...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>neg</td>\n",
       "      <td>my first press screening of 1998 and already i...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>neg</td>\n",
       "      <td>to put it bluntly , ed wood would have been pr...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>neg</td>\n",
       "      <td>synopsis : melissa , a mentally-disturbed woma...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  label                                             review\n",
       "0   neg  how do films like mouse hunt get into theatres...\n",
       "1   neg  some talented actresses are blessed with a dem...\n",
       "2   neg  my first press screening of 1998 and already i...\n",
       "3   neg  to put it bluntly , ed wood would have been pr...\n",
       "4   neg  synopsis : melissa , a mentally-disturbed woma..."
      ]
     },
     "execution_count": 150,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "df = pd.read_csv('./moviereviews_100.tsv', sep='\\t')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 151,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(200, 2)"
      ]
     },
     "execution_count": 151,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 152,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pos    100\n",
       "neg    100\n",
       "Name: label, dtype: int64"
      ]
     },
     "execution_count": 152,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['label'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Append to the dataset the sentiment result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we create a function that calls the API and returns the overall value.\n",
    "Please note that the current API allows to process 10 kB size document maximum; for this reason in the function we reduce the text size accordingly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 153,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sentiment_overall(client, text, language):\n",
    "    try:\n",
    "        input_text = str(text)[0:10000]  # limit the input size\n",
    "        document = client.specific_resource_analysis(\n",
    "        body={\"document\": {\"text\": input_text}}, \n",
    "        params={'language':  language, 'resource': 'sentiment'\n",
    "       })\n",
    "        return document.sentiment.overall\n",
    "    except Exception as e: \n",
    "        print(str(e) +\": \" + str(text))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The overall value is then stored in the dataset, creating a new column `overall`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 154,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>label</th>\n",
       "      <th>review</th>\n",
       "      <th>overall</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>neg</td>\n",
       "      <td>how do films like mouse hunt get into theatres...</td>\n",
       "      <td>-12.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>neg</td>\n",
       "      <td>some talented actresses are blessed with a dem...</td>\n",
       "      <td>-9.39</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>neg</td>\n",
       "      <td>my first press screening of 1998 and already i...</td>\n",
       "      <td>-0.80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>neg</td>\n",
       "      <td>to put it bluntly , ed wood would have been pr...</td>\n",
       "      <td>-11.60</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>neg</td>\n",
       "      <td>synopsis : melissa , a mentally-disturbed woma...</td>\n",
       "      <td>-6.69</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  label                                             review  overall\n",
       "0   neg  how do films like mouse hunt get into theatres...   -12.00\n",
       "1   neg  some talented actresses are blessed with a dem...    -9.39\n",
       "2   neg  my first press screening of 1998 and already i...    -0.80\n",
       "3   neg  to put it bluntly , ed wood would have been pr...   -11.60\n",
       "4   neg  synopsis : melissa , a mentally-disturbed woma...    -6.69"
      ]
     },
     "execution_count": 154,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['overall'] = df['review'].apply(lambda review: sentiment_overall(client, review, language))\n",
    "\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we transform the value into `pos` and `neg` to be compared with the original dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 155,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>label</th>\n",
       "      <th>review</th>\n",
       "      <th>overall</th>\n",
       "      <th>overall_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>neg</td>\n",
       "      <td>how do films like mouse hunt get into theatres...</td>\n",
       "      <td>-12.00</td>\n",
       "      <td>neg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>neg</td>\n",
       "      <td>some talented actresses are blessed with a dem...</td>\n",
       "      <td>-9.39</td>\n",
       "      <td>neg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>neg</td>\n",
       "      <td>my first press screening of 1998 and already i...</td>\n",
       "      <td>-0.80</td>\n",
       "      <td>neg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>neg</td>\n",
       "      <td>to put it bluntly , ed wood would have been pr...</td>\n",
       "      <td>-11.60</td>\n",
       "      <td>neg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>neg</td>\n",
       "      <td>synopsis : melissa , a mentally-disturbed woma...</td>\n",
       "      <td>-6.69</td>\n",
       "      <td>neg</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  label                                             review  overall  \\\n",
       "0   neg  how do films like mouse hunt get into theatres...   -12.00   \n",
       "1   neg  some talented actresses are blessed with a dem...    -9.39   \n",
       "2   neg  my first press screening of 1998 and already i...    -0.80   \n",
       "3   neg  to put it bluntly , ed wood would have been pr...   -11.60   \n",
       "4   neg  synopsis : melissa , a mentally-disturbed woma...    -6.69   \n",
       "\n",
       "  overall_score  \n",
       "0           neg  \n",
       "1           neg  \n",
       "2           neg  \n",
       "3           neg  \n",
       "4           neg  "
      ]
     },
     "execution_count": 155,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['overall_score'] = df['overall'].apply(lambda c: 'pos' if c >=0 else 'neg')\n",
    "\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Perform a comparison analysis between the original `label` and `overall_score`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we compare the results with the annotated corpus, computing the precision and recall and then printing the confusion matrix. `sklearn` and `seaborn` packages need to be installed before proceeding."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 156,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import accuracy_score,classification_report,confusion_matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 157,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "         neg       0.70      0.87      0.77       100\n",
      "         pos       0.83      0.62      0.71       100\n",
      "\n",
      "    accuracy                           0.74       200\n",
      "   macro avg       0.76      0.74      0.74       200\n",
      "weighted avg       0.76      0.74      0.74       200\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(classification_report(df['label'],df['overall_score']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 160,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[87 13]\n",
      " [38 62]]\n"
     ]
    }
   ],
   "source": [
    "cf_matrix = confusion_matrix(df['label'],df['overall_score'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 159,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:>"
      ]
     },
     "execution_count": 159,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAVoAAAD4CAYAAACt8i4nAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAq00lEQVR4nO3dd3xUxfrH8c+z2TSSAEkgoUsvoUNAEKQriCggglhR8eIVwf4DFERsFywXu1wiiiAqoGLBhoAU6b13IUBoARJIL7s7vz+yRhDIBpJNNsvz9nVe7J4yZ06EL8OcM2fEGINSSin3sRR3BZRSyttp0CqllJtp0CqllJtp0CqllJtp0CqllJtZ3X2CwObD9LEGdYHEte8XdxWUBwqwIgUt43IyJ33j+wU+X35oi1YppdzM7S1apZQqUuJ57UcNWqWUd7H4FHcNLqBBq5TyLlIk3a6XRYNWKeVdtOtAKaXcTFu0SinlZtqiVUopN9MWrVJKuZk+daCUUm6mXQdKKeVm2nWglFJupi1apZRyMw1apZRyMx+9GaaUUu6lfbRKKeVm2nWglFJu5oEtWs+LfqWUKgix5H9xVZTIkyKyXUS2iciXIhIgIjVEZLWI7BORWSLi56ocDVqllHcRyf+SZzFSGXgMiDbGNAJ8gIHAa8BbxpjaQCIw2FWVNGiVUt7F4pP/xTUrECgiVqAUcAzoAnzt3D4N6OOySld2JUop5aEuo+tARIaIyLpzliF/FWOMOQK8CRwiJ2DPAuuBM8YYm3O3OKCyqyrpzTCllHe5jJthxpgYIObixUgo0BuoAZwBvgJ6XEmVNGiVUt6l8B7v6gYcMMacBBCROUA7oKyIWJ2t2irAEVcFadeBUsq7FN5TB4eANiJSSkQE6ArsABYBtzv3GQR876ogDVqllHcppJthxpjV5Nz02gBsJScvY4CRwFMisg8IBz52VSXtOlBKeZdCHLBgjHkBeOEfq/cDrS+nHA1apZR30SG4SinlZh44BFeDVinlVUSDViml3EuDViml3EwsGrRKKeVW2qJVSik306BVSik306BVSil387yc1aBVSnkXbdEqpZSbWSw6MkwppdxKW7QlRFiZIH6ePByAyPDSOBwOTiamAHD9PW+QbbMX+BzzPnqcoFL+tL/7dQBaRFVj/JN96f6vdwpctnKP5o0bUKdO3dzvb733AZUrV7novm2im7Nq3cYCne/550axbt0aQoJDEIuF58aMpWmz5gUq86rgeTmrQXsxCWdTaTNwAgCjH+5Jalomb3+2MHe7j48Fu91R4PNEhAZzY7soflu+o8BlKffz9w9g9hyXrx4tVE89PYIbuvdgxfJlvPziWL7+dm6Rnr8k0hZtCRbz4j1kZNloVq8KKzfvJykl47wAXvfVc9z22P84dCyBgT1b8eidHfH1tbJ2ayyPj5+Fw2EuKPOt6QsZObj7BUFrsQivPNabDtF18PO1Mnn2Uj7+Zjkiwluj+tOpVV3iTpwh22Zn+vcr+XbBpqL4Eah/SEtN5fHhQ0lKSsJmszHsscfp3KXbefucPBnPiKefJDUlBZvdzpix42jRMpoVy5cx6YP3yMrKomrVqrz0ynhKBQVd8lwto1tx+NAhAKZ/OpXvvv0GgNv63c49991PWloaI55+ghPHj2N3OBjy76H0uKmn+y7eg2nQlnCVI8rS6f7/4nAYRj988d/E9WpEcvuNLej8wERsNgdvPzuAgT1b8cWPay7Yd/WWA9zauQkdouuQkpaZu/7+PtdxNiWd9ve8gZ+vld8/fYoFK3fRIqoq11QKp3m/V4kIC2bjnOeZ/v1Kt12vOl9mZgYDbusNQKUqVXhz4ju89e4HBAcHk5iYwL133kGnzl3P+4P+808/cl279vzr4Uew2+1kZKSTmJjAR5MnMXnKVEqVKsUnU2KYPm0q/x467JLnXrL4d2rXqcuO7dv4/rs5zPhyNhjD3XcOoGWr1hw5fJjy5SN4f1LO9FfJycnu/WF4MB2CW8LNWbDxoi3Tc3VuXY8WUdVYNmMEAIH+vpxMSLnk/hOmzGPUQz0Y8+7f/yTt1rY+jepUpm+3nP64MsEB1K5Wnuua1WLO/I0YYzhxOpmla/cUwlWp/Ppn10F2djbvvj2RDevXYhEL8fEnOH3qFOXKl8/dp1Gjxrww5jlsNhudu3SjfoMGrFu7iP1/7uP+e+7MLadJs2YXPefE/77OR5MnERoWxriXX2XNqpV06dqNUqVKAdC12w1sWL+Odu2v579vvMZb/32Djp0606JltPt+EB5OW7QlXFr6361Om92O5Zy/OQP8fIGc/8kz5q5m7Hs/5KvMJWv3MO7RXrRuXD13nYjw1GtfsWDlzvP27dG+YQFqrwrbzz/OJTExgS9nz8HX15ebbuhCZlbmefu0jG7FJ9Nn8MeSJYwdPYp7Bz1ASOnStGnbjtfenOjyHH/10f5lzaqL/wumevUazPxqDn/8sYT3332b1te2ybOF7M08MWg974GzEuLg0QSaNagKQLP6VaheORyARWt207dbM8qHBgMQWroU1SqG5lnWhCm/8tSgv/v25q/YyZD+7bFac/731K4WQakAP1Zu2k+frs0QESLCQrg+uo47Lk3lU0pKMmFh4fj6+rJm9SqOHr1wMtSjR48QHl6Ofv0H0Ldff3bu2E6Tps3YtHEDhw4eBCAtLY3Y2AP5OmeLltEs+n0B6enppKWl8fvCBbRoGU18/AkCAgPpdUtvBj0wmF07r94brCKS76WoaIv2Cn23cBN392rN+q9Hs3ZrLHsPxgOwa/9xXvzgR+ZOGoZFhGybnScnzObQscRLljVv2Y7cx8cApn67gmsqhbHyi1GIwKnEFAY8FcO3CzfR6dp6bPxmNHEnzrBp12HOJme4/VrVxfXsdQuPPfoI/frcQlTDRtSoWfOCfdatWcOnUz/GarVSqlQpXhn/GmFhYbz06nhG/d9TZGVnATBs+BNUr17D5TkbRDXk1t63cffA/kDOzbAGDaJYvuwP3vrv61jEgtVqZfTYcYV6rSWJJ7ZoxZi8+xwLKrD5MPee4CoTFOhHanoWYWWC+OOzZ+jywEROnC55Nz4S175f3FVQHijAWvCnYCv9e06+M+fo/2675PlEpB4w65xVNYGxwHTn+upALDDAGHPplhTaoi1x5rz7CGVCAvHz9WH8R7+WyJBVyp0KawiuMWY30AxARHyAI8C3wChgoTFmgoiMcn4fmVdZGrQljI4cUypvbuo66Ar8aYw5KCK9gU7O9dOAxbgIWr0ZppTyLpL/RUSGiMi6c5Yhlyh1IPCl83OkMeaY8/NxINJVlbRF6ybD7+7M/X2vwxjD9n1HGfLCDH6aNIzgoAAAIsJCWLctlgFPfVTMNVVFaeyYZ1m6ZDFhYeHM+f5HAN5/920WL1qIRSyEhofz8qvjiYhw+WdXXcLltGiNMTFAjIvy/IBbgWcvcrwREZd9wtqidYNK5csw9M6OtLv7daL7/wcfi4X+3VvSbfDbtBk4gTYDJ7B6ywG++31zcVdVFbHefW5j0uQp5627/8GH+Prbucye8z0dOnZi8qQPiql23sENj3fdBGwwxpxwfj8hIhWd56oIxLsqwGXQikh9ERkpIu86l5Ei0iC/NbxaWX18CPT3xcfHQmCAH8dOns3dFhIUQMdWdZm7aEsx1lAVh5bRrShdpsx564KDg3M/Z6Sne+TjSSWJG4L2Tv7uNgD4ARjk/DwIcPmmoTy7DkRkpPMkM4G/ButXAb4UkZnGmAn5renV5OjJs7w9fSF7fnmZ9MwsFq7cxcJVu3K339K5CYvX7CY5VZ+BVTnee+ct5v7wHcHBIUyZOr24q1OiFea7DkQkCLgBePic1ROA2SIyGDgIDHBVjqsW7WCglTFmgjFmhnOZALR2brtU5XI7mG2ntruqg9cpGxJIr06NadDrBWreOJqgQD8G9myVu31Aj5bM/nV9MdZQeZrhjz/JbwuXcHOvW5j5xYzirk6JVpgtWmNMqjEm3Bhz9px1p40xXY0xdYwx3YwxCa7KcRW0DqDSRdZXdG67VOVijDHRxphoa7mrb3x+l2vrE3v0NKcSU7DZHHz3+2baNM0Z9RNeNojohtX55Y9txVxL5Yl63nwLC+b/VtzVKNFK4hDcJ4CFIrIXOOxcVw2oDVydb6zIh8PHE2jduAaBAb6kZ2TTuXU9NuzIeZdo327N+eWPbWRm2Yq5lspTHDwYyzXXVAdg0aKF1Khx4VBelX+e2MWdZ9AaY34VkbrkdBVUdq4+Aqw1xhR8PhcvtXbbQb5dsJGVX4zEZneweVccH3+zHID+3Vvy5lRtsVytRj7zFOvWruHMmURu6NKBRx4dzrKlS4mNPYDFIlSsWJkxL7xY3NUs0TzxZqK+60AVC33XgbqYwnjXQb2R8/KdObtf614kqawDFpRSXsUDG7QatEop72LRqWxKNotFWP75CI7Gn6Xf4/9j0gt30SKqGoKw71A8/xr7GanpWecdU61iGJvmjGGP8321a7bG8tirMwFo3qAqMS/eS6C/L/OWb+fp178G4JXHenNjuyi27Injoec/A2Bgz1aUKxvE+18sLroLVpclKSmJF8eOYd++PYgIL778n/OmB1+7ZjVPDB+aO0V5l2435M6CsPyPpbw24VUcdgd9+/Vn8L9yhtw/O+Jp9u7dQ4eOnXnsiacAiPnfh9SuU5cuXbuhLqQt2hJu2F2d2X3gBCHO9xWMeHNO7qCD156+jUcGduTNqfMvOG5/3Knc6cvP9e5zd/Doy1+wZmss373/CDe2i2LV5v00a1CV1neM58Oxd9GwdiX+PHyS+25tw63DdGimJ3t9/Ks5c3e9/S7ZWVmkZ1w4IKV5y2je/3Dyeevsdjv/efUlJn80lcjISO6643Y6de6C3W7DPyCAr7+dy8MPPUBycjIZGels3bKFIf8eWlSXVeJ44s0wfddBPlWOKEuP9g2Z+u2K3HXnjuwK8Pflcm4sVihXmpCgANZsjQXgix/XcEunJjgcBl+rDwClAvzIttl54r6uTJq5BJvtko8uq2KWnJzM+vVr6dvvdgB8/fwoXbp0vo7dtnULVateQ5WqVfH186NHz5tZvGghVqsvmRkZOBwObDYbPhYLH773LkOHDXfnpZR4IvlfiooGbT698X/9GP3OdxfMgjt53D3ELvgP9apH8uHMJRc9tnrlcFZ+OZLfpjxOu+a1AKgUUZYj8Wdy9zly4gyVIsqSkpbJvGXbWTVzFMdPnSUpJZ1Wjaozd7G+F8GTHYmLIzQ0jLGjn2VAvz6MGzuatLS0C/bbsmkT/fveytCHH2Lfvr0AxJ84QYWKFXL3iYiM5MSJE9SsVYvQ0DAG3t6XDp06c+jQIRzGQYOoq28Q0OWwWCz5XoqKdh3kw03XNyI+IZmNOw9zfcvzJ0R8eNwMLBZh4sj+3H5jSz77YdV524+fSqLuTWNJOJtK8wZVmT1xCC1ufzXP802ctoCJ0xYA8OHYu3h50o/c37ct3do0YOveI7w2ZV7hXqAqMLvdxq6dOxg1+nmaNGnKa+Nf4ZMpMQx77IncfRpENeTX+b9TKiiIP5Yu4cnhjzL3l7yfqR7x7Ojcz8OH/pvnx73IR5MnsWf3Ltq0bUe//i6H2V91PLDnQFu0+dG2WU16dWzMrp9eZPqEB+jUqi6fvHJf7naHw/DVvPX06drsgmOzsm0knE0FYOPOw+yPO0WdayI4Gn+GyhFlc/erHFmWo+e0cAGa1quCCOyJjee2bi24Z+Qn1KxSnlrVyrvjMlUBREZWIDKyAk2aNAXghht7XDATbXBwMKWCggC4vkNHbDYbiYkJRERGcvzY8dz94k+cIDLy/PfRLvp9AVENG5KWlsbhw4d4Y+I7zP9tHunp6W6+spLHE4fgatDmw9j3fqB2j+epf/ML3DdqKovX7uHBMdOpWbVc7j69OjZhT+yJC44tFxqc+7hJ9crh1K5WngNxpzh+Konk1AxaN64OwF29WvPjkvO7B8YO7cVLH/6Er9UHH5+cMhzGQakAPzddqbpS5cqXJ7JCBWIP7Adg9aqV1KxV67x9Tp08mduPv3XLFhwOB2XLhtKwUWMOHYolLu4w2VlZ/PrzT3Ts3CX3uOzsbGZMn8b9Dz5EZkZmbkA4HHays7OL6ApLDk/so9WugyskIkx56V5CggIRga17jvDYf3ImzLy5Y2NaRFXj5Uk/0b5FbZ5/5GaybXYcDsPwV2eSmJTTd/f4+NnEvHgPgf6+/LZ8B/OW/d0CuqVTEzbsOJT7Htstu4+wdvZzbNt7hK17jhT9BSuXRj33PM+OfIbs7GyqVKnKS6+MZ/asnNeYDrjjTub/No/Zs77E6uODf0AAr705ERHBarXy7OixPDLkIRwOO3369qN27b+7qGZ9+Tm39u5LYGAgdevVIyM9g359bqH99R3yfcPtauKJTx3oEFxVLHQIrrqYwhiCG/3KonxnzroxnXUIrlJKXS4dGaaUUm7miV0HGrRKKa/igTmrQauU8i7aolVKKTfzwJzVoFVKeRdPvBmmAxaUUl6lMEeGiUhZEflaRHaJyE4RaSsiYSIyX0T2On8NdVWOBq1SyqsU8hDcd4BfjTH1gabATmAUsNAYUwdY6PyeJw1apZRXKawhuCJSBugAfAxgjMkyxpwBegPTnLtNA/q4qpMGrVLKq1xOi1ZEhojIunOWIecUVQM4CUwVkY0iMkVEgoBIY8wx5z7Hgch/1uGf9GaYUsqrXM5TB8aYGCDmEputQAtguDFmtYi8wz+6CYwxRkRcDvnVFq1SyqtYLJLvxYU4IM4Ys9r5/WtygveEiFQEcP4a77JOBbgepZTyOBaRfC95McYcBw6LSD3nqq7ADuAHYJBz3SDge1d10q4DpZRXKeQBC8OBz0XED9gPPEBOA3W2iAwGDgIup7nQoFVKeZXCHIJrjNkERF9kU9fLKUeDVinlVTxwYJgGrVLKu3jiEFwNWqWUV5GCT9JQ6DRolVJexQMbtBq0Sinvou+jVUopN/PAnNWgVUp5F1cDEYqDBq1SyqvoUwdKKeVmHtig1aBVSnkX7TpQSik387yY1aBVSnkZfbxLKaXczAPvhWnQKqW8iz51oJRSbqZdB0op5WYe2KDVoFVKeRdt0SqllJt5Xsxq0CqlvIyPB/YdaNAqpbyKdh0opZSbFWbOikgskAzYAZsxJlpEwoBZQHUgFhhgjEnMqxxL4VVJKaWKn0Uk30s+dTbGNDPG/DUb7ihgoTGmDrDQ+T3vOl3ZpSillGcSyf9yhXoD05yfpwF9XB3g9q6D6VOfc/cpVAnU+b9Li7sKygOtHNmhwGVcTh+tiAwBhpyzKsYYE3POdwP8JiIGmOzcFmmMOebcfhyIdHUe7aNVSnkVn8sIWmdwxuSxS3tjzBERiQDmi8iufxxvnCGcJ+06UEp5FYvkf3HFGHPE+Ws88C3QGjghIhUBnL/Gu6xTQS5IKaU8TWEFrYgEiUjIX5+BG4FtwA/AIOdug4DvXdVJuw6UUl6lEJ+jjQS+dZZnBb4wxvwqImuB2SIyGDgIDHBVkAatUsqrFNbAMGPMfqDpRdafBrpeTlkatEopr+KBA8M0aJVS3sXqgUmrQauU8ioemLMatEop76LTjSullJt5YM5q0CqlvIsHvo5Wg1Yp5V30xd9KKeVmHpizGrRKKe8iHjhrmAatUsqraItWKaXcTINWKaXcTCdnVEopN/PxwJe/atAqpbyKjgxTSik30z5apZRyMw9s0GrQKqW8i0Wfo1VKKffSFq1SSrmZ1QM7aTVolVJexRNbtB74xJlSSl05i0i+l/wQER8R2SgiPzq/1xCR1SKyT0RmiYifyzoV8JqUUsqjiOR/yafHgZ3nfH8NeMsYUxtIBAa7KkCDVinlVSyXsbgiIlWAm4Epzu8CdAG+du4yDeiTnzoppZTXuJyuAxEZIiLrzlmG/KO4t4ERgMP5PRw4Y4yxOb/HAZVd1UlvhimlvMrlDME1xsQAMRfbJiK9gHhjzHoR6VSQOmnQKqW8SiE+dNAOuFVEegIBQGngHaCsiFidrdoqwBFXBWnXgVLKqxTWzTBjzLPGmCrGmOrAQOB3Y8zdwCLgdudug4DvXdVJg1Yp5VUkp+81X8sVGgk8JSL7yOmz/djVAdp1oJTyKu5oPRpjFgOLnZ/3A60v53gNWqWUV9H30SqllJvpVDZKKeVmnnjjSYNWKeVVtEVbAjw/sCuR1Wrkfr/7mVcIjahw0X1fuu8mxk7/pUDn++bDCezbsp6n3/scq68fqUlnmfTcwzzz/swClavco3SAlfcGNgEgPMgPhzEkpmUDMHj6RmwOU+BzfHBnE8KD/MiyO0jPsvPqL3s4lJBe4HKvFp4Xsxq0F/D182PY61OK9JwWi4X1i37h2ht7F+l51eVLyrAx6NMNAAxudw3p2Xa+WBOXu91HwF7wrGXcj7vYdTyF3k0rMKxTTUbM2V7wQq8SPtqiLXkyM9L5/I3RpKek4LDb6HbHgzRo1f68fZITTzPz7ZfITE/FYbdz60NPUr1BE/ZuXsvvX32KLTubsMhK3DZ0JP4BgRec47qe/Vjx89dEd+11wbY/fpjJtpWLsdmyiWrVnq4DHgBg0TfT2fzHAkqVLkOZ8Agq16xL+1vucM8PQeVpTM+6ZNkc1I0MZsuRJFIz7ecF8IwHW/LM19s4npRJ96gIBrSshNXHwo5jybzx217yagRvOnyWO6JzhtIP61SDNjXDAJi64hALd50kPMiPl3s3IMjPBx+L8MZve9kcl+T2a/ZkHpizGrT/lJ2VxfsjHgIgNKIiA58cx11Pv0xAqSBSk84yecxQ6ke3O68faPOyhdRp2opOt92Dw2EnOzOT1KSzLJ4zgwfGvIlfQCBLv/+S5T/Opsvtgy44Z5lykVxTrxGblv5G/ZbX5a7fu3ktp4/H8e//TMIYw+dvjObAjs34+vmzffVSHn19Cg67jQ9HDaFyzbru/+GoS4oI8WfIjE04TE5L92KuCQ+kW4PyDPl8M3aH4ZkbatM9KoJftsdfstz2tcP582QqneqWo05kMPdNXU+ZQF8+GdScTYfPcmNUeVYfSGDaysNYBAJ8fdx1iSWGeGDngQbtP/yz68BuszF/5hRid25BREhKOEXK2URCyobl7lO5Vj2+/d/r2O02olq1p2L12hzYsZKTR2KJGTs8t5yqdaMued4Ofe7m8zfGUK9Fm9x1+7asY9+WdXww8l8AZGWkc/p4HFnp6TSIboevnx/gR72WbQv5p6Au1++7T+XZMgVodU0o9SKD+eS+5gD4Wy25/bv/NK5XfTJtDo6dzWDigj+5s1Vl5u+Ix2EgMS2bjYfO0qBiMDuPpTC6Z12sFgtL955ib3xqYV9aiaMt2hJo87IFpCadYej4yfhYrbw5bCC2rKzz9qkR1ZSHxr3D7o2r+ObDCbS7uT+BwSHUahzNHY8/n6/zlKtYhYrVa7Ft5eK/VxpDh9530fqGW8/bd8VPX6M8S3q2Pfez3WHOa1P5W/9+4OiXbSeYtDTWZXl/9dG6sinuLI98sZnraoUxpmc9Zq6Ny7OFfDXwxFlwPfGRM4+SkZZCcOlQfKxW9m/byJmTJy7YJ/HkcYLLhtKqay+iu9zM0QN7qVonikO7t3H6eM6LfbIy0jl19HCe5+rY9x6W/Tgr93vtpq3YsPgXMjNy7jgnJZwk5Wwi1eo1Ytf6lWRnZZGZkc7uDasK8YpVQR07m0G9CsEA1I0MpmKZAADWHTxD53rlCS3lC+Q8wVChtH++ytwUd5auDcpjESgb6EuzqmXYcSyZCqX9SUjN4ofNx5m75Xjuea9mbphhocC0RetC0/Y3MOP153jvmQepVLMe5StVu2CfAzs2seyHWfhYrfgFBNLv0WcJKl2W24aOZPY7L2Oz5fzzsNsdD1KuUtVLniuyag0qVa/L0dg9ANRp2oqTRw4SM+ZRAPwCArl92HNUqV2f+tHX8f6IwQSXCSWyag38SwW54erVlVi85xQ3NYrk88Et2XE0mcOJOX9Rxp5OY/Ifsbw9oDEWAZvD8Ob8fRxPynRZ5pI9p2lcqTTTH2gJwAeLD5CQmk3PRpHc1boKdochLcvOSz/tduu1lQSeOARXjCmEZ1Hy8NWmo+49wVUqMyMd/4BAsjIzmDLucfr862kqlaAbYhPn7SvuKigPtHJkhwKn5MJdp/KdOV3rlyuSVNYWbQn1fcybxMcdxJadRfMO3UtUyCrlTvrUgSo0Ax7L3002pa42HthzoEHrLtlZWUwZ9zj27CwcDjsNr+1I1wEP8OfW9fw6YzLGOHL6c4eOIryCy7ndlJcI9vfh2ZvqUqtcEAZ49efddKpbjva1w8m2OzhyJoNXft5NSqbdZVnq4rRFexWx+vry4NiJ+AcEYrfZ+OiF4dRtdi0/fPw2dz/zChFVrmH1vO9YPOcz+g0dVdzVVUXkya61WbU/kdHf7cRqEQJ8LayJPcOkJQewGxjasQb3tanGh0sOFHdVSyyL5+WsPt7lLiKSO9zWbrdht9lBcv62zUzPeag8Iy2VkNDw4qymKkJBfj40q1qGuVuOAzlPHaRk2lkTm5j7foTtR5OICMnfI1/q4i5nuvGioi1aN3I47Hw46mESjh/h2u59qFonij4PP8P0Cc/i6+eHf2AQD7/yQXFXUxWRSmUDOJOWxZiedakTEcyu48m8tfBPMrIdufv0alKBBTtPFmMtSz4PbNBeeYtWRB7IY9sQEVknIusWfDPjSk9R4lksPgx7fQr/N+kr4vbt4sShA6z46WvuGzWeEZO+okWnHvwy/cPirqYqIj4WoW6FEOZsPMagTzeQnu3gvjZ/P1c9qG1V7A7DvB1X98iugvLEFm1Bug5evNQGY0yMMSbaGBPdrd89BTiFdwgMCqZGw2bs2bSaY4f+pGqdnHceNL6uM4f26OvvrhbxyZmcTM5kx7FkABbtPkndyJyRXD0bRdKuVjgvzN1VnFX0CnIZS1HJM2hFZMsllq1AZBHVsURKTTpDemrOWPXsrEz+3Lqe8pWvITMtJXco7r4t6yhf+cKRZso7JaRmcyIpk2phOX330deEEnsqjTY1Qrnn2iqM+GY7mTaHi1KUS4WUtCISICJrRGSziGwXkRed62uIyGoR2Scis0TEz1WVXPXRRgLdgcSLXMoKV4VfzZITT/PNhxNwOBwYh4NGbTtRv2Vb+gx5hi8nvoCIEBAcwm3/HlHcVVVFaOKCfYzrVR9fH+HImQxe/XkPnwxqjq+PhXfuaAzk3BB7/TcdOXelCrFLIBPoYoxJERFfYJmI/AI8BbxljJkpIv8DBgOT8ioozyG4IvIxMNUYs+wi274wxtzlqqY6BFddjA7BVRdTGENw1+4/m+/MaVWzTL7OJyKlgGXAI8BPQAVjjE1E2gLjjDHd8zo+z64DY8zgi4Wsc5vLkFVKqSJ3GV0H5964dy5DzitKxEdENgHxwHzgT+CMMcbm3CUOcDniSB/vUkp5lcsZGWaMiQFi8thuB5qJSFngW6D+ldRJg1Yp5VXc8dSWMeaMiCwC2gJlRcTqbNVWAY64Ol6DNh/mTHqN3RtWEVS6LI/9dyoAx2L38cOUt8jKSKds+Qr0Hz6agIu8E3bPpjX8/On7OBx2Wna5mY59cnpcEuKPMfudl0hLTqJSzbrcPuw5rFZfVv4yh7UL5lK2XCR3/d/LWK2+xO7ayo7VS+k56NEivW6VNz8fYdJdTfG1WvCxCIt2n2LKsoPc3qISd0RXpkpoID3eXcHZdNtFj+/ZKJL72+Y8dfLpykP8vC3npfL1IoN5/uZ6+FstrPgzgbcW/gnkDM9tWzOUvfGpue+d7R4VQdlSvsxa5/LP+lWjsHJWRMoD2c6QDQRuAF4DFgG3AzOBQcD3rsrSIbj50LxjDwY9+9p5676b/CY33vUvhr/5CVGt27Ns7qwLjnM47Mz95B3ue3YCj038lK3LFxIfFwvAb59P5rqe/Xnq3c8JDAph/e8/AzlT5wx742Oq1m3Ivs1rMcaw+JvpdOp3r9uvU12eLLth2Mwt3Dd1A/dN3UCbGqE0rBTClrizDJ+5hWNnMy55bOkAKw+2q8ZDn21k8PSNPNiuGiH+Oe2eETfWZvyve+gfs5aqYYG0qRlKkJ8P9SoEc+/UDWTbHdQqVwp/q4VejSP5esPRorrkEkFE8r24UBFYJCJbgLXAfGPMj8BI4CkR2QeEAx+7KkiDNh9qRDUlMLj0eetOHYujeoOmANRqHM321UsvOC5u3y7CIysRFlkJq9WXxtd1Yefa5Rhj2L99Iw3bdASgecfu7Fz71z1Hg8NuIzsrA4uPlU1/zKdus2sp9Y/zK8+Q7hw+a7UIVotgDOyJT3U5a8K1NUJZG3uGpAwbyZk21saeoU3NUMKD/Ajyt7L9aM6ghl+2naBjnXIY5zkgZ6Zbm8NwV+sqfLXhKHZXs0JeZQprKhtjzBZjTHNjTBNjTCNjzEvO9fuNMa2NMbWNMf2NMS6nyNCgvUIRVauzc91yALavWszZ0xcOm0xKOEWZ8Ijc76XDy5OUeIq05CQCSgXj45MzNXTpsPIkJZwC4NrufZk85lHOnornmnqN2LD4V67t3sf9F6SuiEVg2v0t+Hl4W9bEnskd9eVK+RB/TpwTxvHJmZQP8ad8iB/xyf9YH+xHWpadFX8mMO3+FpxKySIl007DiiEs3Xu60K+ppPPEkWHaR3uFbvv3CH789D0WfzOd+i3b4WP1LZRym3e4keYdbgTg96+n0fam29izaTWblv5GmfAIetz7CBaL/v3oKRwGBn26gWB/Hyb0bUjNcqXYfyrNLef6fE0cn6+JA+DZHnWIWXaQW5pU4NoaoeyLT+XTlYfcct4SxwPfKqN/Yq9Q+crVeGD0GwydEEOTdl0Ii6x0wT6lw8qd19JNOn2S0qHlKBVSmoy0FOz2nJc7JyWcpHRYufOOTUo4xZE/dxHVqj3Lf5zNHU+MJaBUMPu3bXDvhakrkpJpZ8OhM7SpGZav/U8mZxJ5zgy4ESH+nEzO5GRy1nmvSYwI8edkyvnT29eNCEJEOJSQRpf65Rjz/U4qhwZQJTSgcC6mhJPL+K+oaNBeoZSzOaOSHQ4Hi+d8Rusbbrlgn8q16nP6+BES4o9hs2WzdcXv1I++DhGhRlRztq9aAsDGJfNoEN3uvGMXzP6Erv1zXpCWnZUFCGIRsjMvfYNFFa2ygb4E++d0//hbLbSqHsrB0/lrza4+kEjr6qGE+FsJ8bfSunooqw8kcjo1i9RMGw0rhQBwU6NIlu49dd6xQ66vTswfsVgtgo+zo9EYCLD6FOLVlVw63XgJNeudlzmwYxNpyWd5/ZH+dOl/P1kZ6az+LeepjqjW19Oi001ATkv0u8lvct+zE/Dx8aHXg48x7T8jcDgctOx0E5FVawDQ/e4hzHrnZRbM+piK1evQskvP3PMdPbAXIHfCxabtuvL+/z1ImfAIrr91YFFeuspDeLAfY2+uhyVnhBG/7zrJ8j8T6N+yEvdcW5WwID8+e6AlK/cnMP7XvdSvEEzfZhUZ/+tekjJsTF1xkE8GNQfgkxUHScrIeQzsjfn7GNMz5/GuVfsTWLn/71eNdKgTzs7jKZxytnL3xqcw48GW7ItPZd/J1KL/IXggT5wzTKcbV8VC33WgLqYw3nWw/UhqvjOnYeUgnW5cKaUulye2aDVolVJexQNzVoNWKeVlPDBpNWiVUl6lKOcCyy8NWqWUV/G8mNWgVUp5Gw9MWg1apZRXKcoRX/mlQauU8ioe2EWrQauU8i4emLMatEop75KPF3oXOQ1apZRX8cCc1aBVSnkXD8xZDVqllJfxwKTVoFVKeRVPfLxLX/ytlPIqhfXibxGpKiKLRGSHiGwXkced68NEZL6I7HX+GuqqThq0SimvYpH8Ly7YgKeNMVFAG+BREYkCRgELjTF1gIXO73nXqWCXpJRSnqZw5sE1xhwzxmxwfk4GdgKVgd7ANOdu04A+rmqkQauU8iqX03UgIkNEZN05y5CLlynVgebAaiDSGHPMuek4EOmqTnozTCnlVS7nVpgxJgaIybM8kWDgG+AJY0zSuQMijDFGRFxOnaMtWqWUVynMWXBFxJeckP3cGDPHufqEiFR0bq8IxLsqR4NWKeVVRCTfi4tyBPgY2GmMmXjOph+AQc7Pg4DvXdVJuw6UUl6lEJ+ibQfcC2wVkU3Odc8BE4DZIjIYOAgMcFWQBq1SyqsU1rsOjDHLuHRud72csjRolVJexRNHhmnQKqW8i+flrAatUsq7eGDOatAqpbyLTjeulFJu5oE5q8/RKqWUu2mLVinlVTyxRatBq5TyKvp4l1JKuZm2aJVSys00aJVSys2060AppdxMW7RKKeVmHpizGrRKKS/jgUmrQauU8iqeOARXjHE53Y0qJCIyxDlHkVK59PeF99MhuEXrojNsqque/r7wchq0SinlZhq0SinlZhq0RUv74dTF6O8LL6c3w5RSys20RauUUm6mQauUUm6mQVtERKSHiOwWkX0iMqq466OKn4h8IiLxIrKtuOui3EuDtgiIiA/wAXATEAXcKSJRxVsr5QE+BXoUdyWU+2nQFo3WwD5jzH5jTBYwE+hdzHVSxcwYsxRIKO56KPfToC0alYHD53yPc65TSl0FNGiVUsrNNGiLxhGg6jnfqzjXKaWuAhq0RWMtUEdEaoiIHzAQ+KGY66SUKiIatEXAGGMDhgHzgJ3AbGPM9uKtlSpuIvIlsBKoJyJxIjK4uOuk3EOH4CqllJtpi1YppdxMg1YppdxMg1YppdxMg1YppdxMg1YppdxMg1YppdxMg1Yppdzs/wE3p3n8ELVPEwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import seaborn as sns\n",
    "group_names = ['True Neg','False Pos','False Neg','True Pos']\n",
    "group_counts = [\"{0:0.0f}\".format(value) for value in\n",
    "                cf_matrix.flatten()]\n",
    "group_percentages = [\"{0:.2%}\".format(value) for value in\n",
    "                     cf_matrix.flatten()/np.sum(cf_matrix)]\n",
    "labels = [f\"{v1}\\n{v2}\\n{v3}\" for v1, v2, v3 in\n",
    "          zip(group_names,group_counts,group_percentages)]\n",
    "labels = np.asarray(labels).reshape(2,2)\n",
    "sns.heatmap(cf_matrix, annot=labels, fmt='', cmap='Blues')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 161,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.745"
      ]
     },
     "execution_count": 161,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "accuracy_score(df['label'],df['overall_score'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}