LogisticRegression_SentimentAnalysis.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "LogisticRegression_SentimentAnalysis.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/smorga41/Logistic-Regression-Sentiment-Analysis/blob/master/LogisticRegression_SentimentAnalysis.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7Q1IMYO7JCal"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bNtLtMnzTaAI"
      },
      "source": [
        "#IMDB reviews sentiment analysis project\n",
        "\n",
        "In this project my aim is to build a machine learning model able to classify movie reviews into *positive* or *negative*, extracting information from the text of the review. This task is known as **Sentiment Analysis**, where you need to identify the sentiment of a text.\n",
        "\n",
        "Data has been downloaded from [Kaggle](https://www.kaggle.com/lakshmi25npathi/imdb-dataset-of-50k-movie-reviews), however, the original data is available [here](http://ai.stanford.edu/~amaas/data/sentiment/). \n",
        "\n",
        "i Have split data into 3 groups:\n",
        "- training data: 30,000 reviews annotated with *positive* and *negative* labels;\n",
        "- validation data: 10,000 reviews annotated with *positive* and *negative* labels;\n",
        "- test data: 10,000 reviews un-labelled for which i will predict labels for (i submitted my predictions for this test data to the competition server where my attempt will be graded). \n",
        "\n",
        "The program will guess whether each review should be labelled as positive or negative. \n",
        "\n",
        "For humans it is quite easy to do. For example one review starts: \"i can't believe how dumb this movie truly is.\" which we would rightly assign as being a negative review, but can we program a computer to guess?\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "r_O1lCWxcQtw"
      },
      "source": [
        "## Modules, Functions and Downloads\n",
        "\n",
        "The following few cells import useful modules, download the datasets and provide a series of useful functions. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "c_VspJfDScrk"
      },
      "source": [
        "#useful python librarys\n",
        "import numpy as np\n",
        "import urllib.request\n",
        "from urllib.request import urlopen\n",
        "import re\n",
        "from zipfile import ZipFile\n",
        "from google.colab import files\n",
        "import warnings\n",
        "warnings.filterwarnings('ignore')\n",
        "import matplotlib.pyplot as plt\n",
        "%matplotlib inline\n",
        "import pandas as pd\n",
        "\n",
        "#classifiers from sklearn. More options here: https://scikit-learn.org/stable/supervised_learning.html#supervised-learning\n",
        "from sklearn.linear_model import LogisticRegression #Logistic regression classifier: https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
        "from sklearn.neighbors import KNeighborsClassifier #K-neareast neighbors classifier: https://scikit-learn.org/stable/modules/neighbors.html#nearest-neighbors-classification\n",
        "from sklearn.dummy import DummyClassifier #Dummy classifier (majority class): https://scikit-learn.org/stable/modules/generated/sklearn.dummy.DummyClassifier.html\n",
        "from sklearn.naive_bayes import GaussianNB #Naive Bayes classifier: https://scikit-learn.org/stable/modules/naive_bayes.html#gaussian-naive-bayes\n",
        "from sklearn.tree import DecisionTreeClassifier #Decision tree classifier: https://scikit-learn.org/stable/modules/tree.html#classification\n",
        "from sklearn.ensemble import RandomForestClassifier #Random Forests classifier: https://scikit-learn.org/stable/modules/ensemble.html#forests-of-randomized-trees\n",
        "from sklearn.neural_network import MLPClassifier #Multi-layer perceptron classifier: https://scikit-learn.org/stable/modules/neural_networks_supervised.html#classification\n",
        "\n",
        "#pre-processing and feature extraction\n",
        "from nltk.corpus import stopwords\n",
        "from sklearn.feature_extraction.text import TfidfVectorizer\n",
        "from sklearn.feature_extraction.text import CountVectorizer\n",
        "\n",
        "#evaluation\n",
        "from sklearn.metrics import accuracy_score, confusion_matrix"
      ],
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WAXAXfd8VLZx"
      },
      "source": [
        "This cell loads the data in the correct splits for the project. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ng1R6cATWmJ6"
      },
      "source": [
        "### DO NOT change this part ###\n",
        "\n",
        "#loading datasets\n",
        "df_test = pd.read_csv('https://github.com/carolscarton/headstart/raw/master/data/hs2020_imdb_test.csv')\n",
        "df_val = pd.read_csv('https://github.com/carolscarton/headstart/raw/master/data/hs2020_imdb_val.csv')\n",
        "df_train = pd.read_csv('https://github.com/carolscarton/headstart/raw/master/data/hs2020_imdb_train.csv')"
      ],
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CwY-ftmxU4ci"
      },
      "source": [
        "Pre-processing function."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Du7z57z8Rpx6"
      },
      "source": [
        ""
      ],
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Glbo_A3FJkDx"
      },
      "source": [
        "########### PRE-PROCESSING ###########\n",
        "#Pre-processing the reviews\n",
        "#normalise words, remove punctuation, remove extra spaces, etc\n",
        "def pre_proc(features):\n",
        "    processed_features = []\n",
        "    for sentence in range(0, len(features)):\n",
        "        # Remove all tags (like <br />)\n",
        "        processed_feature = re.sub(r'<.*?>', ' ', str(features[sentence]))\n",
        "\n",
        "        #Remove all special characters\n",
        "        processed_feature = re.sub(r'[^a-zA-Z0-9]', ' ', processed_feature)\n",
        "\n",
        "        # Substituting multiple spaces with single space\n",
        "        processed_feature = re.sub(r'\\s+', ' ', processed_feature, flags=re.I)\n",
        "\n",
        "        # Removing prefixed 'b'\n",
        "        processed_feature = re.sub(r'^b\\s+', '', processed_feature)\n",
        "        \n",
        "        # Removing everything that has numbers \n",
        "        processed_feature = re.sub(r'\\w*\\d\\w*', '', processed_feature)\n",
        "\n",
        "        # Converting to Lowercase\n",
        "        processed_feature = processed_feature.lower()\n",
        "\n",
        "        processed_features.append(processed_feature)\n",
        "    return processed_features\n"
      ],
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AYpcj1tYVBnY"
      },
      "source": [
        "Feature extraction function."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2TvwziMgUs-c"
      },
      "source": [
        "########### EXTRACT FEATURES ###########\n",
        "# I will use bag-of-words as features for training our classifiers. \n",
        "# In a bag-of-words approach, an algorithm counts the number of times a word appear in a document. \n",
        "# Each word in the entire collection of documents (corpus) became a feature in the feature vector, which results in a sparse vector.\n",
        "# Instead of count the \"number of times\" a word appear in a document, we can also use a binary approach (whether or not a word a appear in a document). \n",
        "# if a word appears in a document it will receive 1 (0 otherwise)\n",
        "def extract_features(proc_train, proc_val, proc_test, binary=False, max_df=1.0, min_df=0.0, ngram_range=(1,1), sw=False):\n",
        "    stop_words=[]\n",
        "    if sw:\n",
        "        f = urlopen(\"https://raw.githubusercontent.com/lionfish0/discover_stem/master/stopwords.txt\").read()\n",
        "        stop_words = list(np.array(f.split(), dtype=str))\n",
        "\n",
        "    cv = CountVectorizer(binary=True, max_df=max_df, min_df=min_df, ngram_range=ngram_range, stop_words=stop_words)\n",
        "    cv.fit(proc_train)\n",
        "\n",
        "    #check the features outputted below\n",
        "    #each possible word in our pre-processed vector became a feature\n",
        "    print(\"** Vocabulary size: %d\" % len(cv.get_feature_names()))\n",
        "    print(\"** Words:\")\n",
        "    #print(cv.get_feature_names()) #causes error with colab bandwidth limits\n",
        "\n",
        "    #apply the model to all data splits\n",
        "    X_train = cv.transform(proc_train)\n",
        "    X_val = cv.transform(proc_val)\n",
        "    X_test = cv.transform(proc_test)\n",
        "    return X_train, X_val, X_test\n",
        "\n"
      ],
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LMmU17STVFzn"
      },
      "source": [
        "Evaluation function.\n",
        "\n",
        "1.   List item\n",
        "2.   List item\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "df3losFzUyYb"
      },
      "source": [
        "########### EVALUATION ###########\n",
        "#a function to print a confusion matrix\n",
        "def plot_confusion_matrix(y_true, y_pred, classes,\n",
        "                          normalize=False,\n",
        "                          title=None,\n",
        "                          cmap=plt.cm.Blues):\n",
        "    \"\"\"\n",
        "    This function prints and plots the confusion matrix.\n",
        "    Normalization can be applied by setting `normalize=True`.\n",
        "    \"\"\"\n",
        "    if not title:\n",
        "        if normalize:\n",
        "            title = 'Normalized confusion matrix'\n",
        "        else:\n",
        "            title = 'Confusion matrix, without normalization'\n",
        "\n",
        "    # Compute confusion matrix\n",
        "    cm = confusion_matrix(y_true, y_pred)\n",
        "    if normalize:\n",
        "        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n",
        "\n",
        "    fig, ax = plt.subplots()\n",
        "    im = ax.imshow(cm, interpolation='nearest', cmap=cmap)\n",
        "    ax.figure.colorbar(im, ax=ax)\n",
        "    # We want to show all ticks...\n",
        "    ax.set(xticks=np.arange(cm.shape[1]),\n",
        "           yticks=np.arange(cm.shape[0]),\n",
        "           # ... and label them with the respective list entries\n",
        "           xticklabels=classes, yticklabels=classes,\n",
        "           title=title,\n",
        "           ylabel='True label',\n",
        "           xlabel='Predicted label')\n",
        "\n",
        "    # Rotate the tick labels and set their alignment.\n",
        "    plt.setp(ax.get_xticklabels(), rotation=45, ha=\"right\",\n",
        "             rotation_mode=\"anchor\")\n",
        "\n",
        "    # Loop over data dimensions and create text annotations.\n",
        "    fmt = '.2f' if normalize else 'd'\n",
        "    thresh = cm.max() * 0.8\n",
        "    for i in range(cm.shape[0]):\n",
        "        for j in range(cm.shape[1]):\n",
        "            ax.text(j, i, format(cm[i, j], fmt),\n",
        "                    ha=\"center\", va=\"center\",\n",
        "                      color=\"white\" if cm[i, j] > thresh else \"black\")\n",
        "    fig.tight_layout()\n",
        "    return ax\n",
        "class_names = ['0', '1', '2', '3', '4']\n",
        "np.set_printoptions(precision=2)\n",
        "\n",
        "#evaluation function\n",
        "def evaluate_classifier(y_test, y_pred):\n",
        "    print(\"*** Accuracy score: %.4f\\n\" % accuracy_score(y_test, y_pred))\n",
        "    ax = plot_confusion_matrix(y_test, y_pred, classes=class_names, normalize=True,\n",
        "                      title='Normalized confusion matrix')\n",
        "    plt.plot()\n"
      ],
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wt4bNxpRVWZY"
      },
      "source": [
        "Code to create a .zip file for the predictions (answers) of the classifier. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "g9azGsXKU1cM"
      },
      "source": [
        "########### CREATE ZIP FILE FOR CODALAB ###########\n",
        "#builds and lets you download the answer.zip file\n",
        "def create_zip (answers):\n",
        "  from zipfile import ZipFile\n",
        "  answertext = '\\n'.join(answers)\n",
        "  with open('answer.txt', 'w') as filehandle:\n",
        "      filehandle.write(answertext)\n",
        "  with ZipFile('answer.zip', 'w') as zipf:\n",
        "      zipf.write('answer.txt')    \n",
        "\n",
        "### only works in Chrome (apparently)\n",
        "  from google.colab import files\n",
        "  files.download('answer.zip') "
      ],
      "execution_count": 7,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oKs9XZcHXi1a"
      },
      "source": [
        "<hr>\n",
        "\n",
        "# The Data\n",
        "\n",
        "To give an example of the dataset, i have printed the validation set with the labels (i.e. if positive or negative). Below, I have also printed the test set that does not contain labels."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HWVkO2WUP9hG",
        "outputId": "7d83b55a-1cb2-4c09-e0fc-26085022dca1",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 265
        }
      },
      "source": [
        "print(df_val)"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "                                                 review sentiment\n",
            "0     First off I want to say that I lean liberal on...  negative\n",
            "1     I was excited to see a sitcom that would hopef...  negative\n",
            "2     When you look at the cover and read stuff abou...  negative\n",
            "3     Like many others, I counted on the appearance ...  negative\n",
            "4     This movie was on t.v the other day, and I did...  negative\n",
            "...                                                 ...       ...\n",
            "9995  I thought this movie did a down right good job...  positive\n",
            "9996  Bad plot, bad dialogue, bad acting, idiotic di...  negative\n",
            "9997  I am a Catholic taught in parochial elementary...  negative\n",
            "9998  I'm going to have to disagree with the previou...  negative\n",
            "9999  No one expects the Star Trek movies to be high...  negative\n",
            "\n",
            "[10000 rows x 2 columns]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4xE3o41lzPB7",
        "outputId": "ea42bd7c-e0a9-43c4-d2a6-efe475f50f44",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 265
        }
      },
      "source": [
        "print(df_test)"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "                                                 review\n",
            "0     One of the other reviewers has mentioned that ...\n",
            "1     A wonderful little production. <br /><br />The...\n",
            "2     I thought this was a wonderful way to spend ti...\n",
            "3     Basically there's a family where a little boy ...\n",
            "4     Petter Mattei's \"Love in the Time of Money\" is...\n",
            "...                                                 ...\n",
            "9995  Fun, entertaining movie about WWII German spy ...\n",
            "9996  Give me a break. How can anyone say that this ...\n",
            "9997  This movie is a bad movie. But after watching ...\n",
            "9998  This is a movie that was probably made to ente...\n",
            "9999  Smashing film about film-making. Shows the int...\n",
            "\n",
            "[10000 rows x 1 columns]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "y-RSC5OHkezC"
      },
      "source": [
        "# Naive Classifier \n",
        "\n",
        "Below is an example of a naive solution involving no ML."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VkSF8r9NSrh0",
        "outputId": "953e2a85-8941-4a64-fe37-64257b9e0bec",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 332
        }
      },
      "source": [
        "\n",
        "naive_classifier = []\n",
        "for t in df_val['review']:\n",
        "  if np.any([substr in t.lower() for substr in ['best','wonderful','good','favourite','great','fun','family']]):\n",
        "    naive_classifier.append('positive')\n",
        "  else:\n",
        "    naive_classifier.append('negative')\n",
        "\n",
        "#evaluate how good the approach is (using the validation set)\n",
        "evaluate_classifier(df_val['sentiment'], naive_classifier)\n",
        "\n",
        "\n",
        "naive_classifier_on_test = []\n",
        "for t in df_test['review']:\n",
        "  if np.any([substr in t.lower() for substr in ['best','wonderful','good','favourite','great']]):\n",
        "    naive_classifier_on_test.append('positive')\n",
        "  else:\n",
        "    naive_classifier_on_test.append('negative')\n",
        "#create_zip(naive_classifier_on_test)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "*** Accuracy score: 0.5684\n",
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAATkAAAEYCAYAAAAnEYFiAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dd5xV5Z3H8c93BhABUVBQaQqKCpYIogZjwYYQ28auSYwxRo0SE6NJNMYSV3d13bVs1E3QuPYao2BZwRJiiQpYQMFGlA5SBCxgaL/9456ByzAz9w733rllvm9f5+Upz3nO79yZ+fE8pzxXEYGZWaWqKnYAZmaF5CRnZhXNSc7MKpqTnJlVNCc5M6toTnJmVtGc5EqEpDGSzkjmvytpdJ7r31ZSSGqRz3ozHFOS/lfSIkljc6hnP0kf5DO2YpHUQ9KXkqqLHUtz0WySnKSpkuZJapu27gxJY4oYVp0i4r6IGFzsOPJgX+BQoFtE7LWhlUTESxGxY/7CKozkd+yQhspExPSIaBcRq5oqruau2SS5RDXws1wrSVooze2z2xDbAFMj4qtiB1IKmrIVbWs1tz/U64ALJW1W10ZJ+0gaJ2lJ8v990raNkXS1pFeApUCvpPt3jqSPJH0h6V8lbSfp75I+l/SwpFbJ/h0kPSlpftJ9e1JSt3riOE3Sy8n8r5LuTc20QtKdybZNJf1J0hxJsyRdVdMNklQt6T8lLZD0MXB4Qx+MpO6S/pLEt1DSzcn6Kkm/lTQtaQnfLWnTZFtNF/gHkqYnx7ok2fYj4HZgYBL379LPK+24IWn7ZP7bkiYnn+UsSRcm6wdJmpm2T5/k57FY0iRJR6Vtu1PSLZKeSup5XdJ29ZxzTfw/lDQj+bmcLWlPSROT+m9OK7+dpBeSz2eBpPtqfpck3QP0AJ5IzvdXafX/SNJ04IW0dS0kdZQ0U9KRSR3tJE2RdGpDPytrpIhoFhMwFTgE+AtwVbLuDGBMMt8RWAR8H2gBnJwsb55sHwNMB3ZOtrcEAhgBtE/W/xN4HugFbApMBn6Q7L85cCzQBtgEeAR4PC2+McAZyfxpwMt1nEN3YDYwNFl+DPgj0BboDIwFzkq2nQ28n+zTEfhrEm+LOuqtBiYANyR1tQb2TbadDkxJzqld8vndk2zbNqnzNmBj4BvJZ9CnrvOo67yS/bdP5ucA+yXzHYD+yfwgYGYy3zKJ5zdAK+Ag4Atgx2T7ncBCYK/k53Qf8GA9vxM18f8hOefBwNfA48nn2RWYBxyQlN+eVPd7I6AT8CJwY+3fsTrqvzv5XDdOW9ciKTMYmJsc7zbgz8X+W6m0qegBNNmJrk1yuwBLkl/S9CT3fWBsrX1eBU5L5scAV9baHsC30pbfAH6dtvxf6X8EtfbdHViUtjyGBpJc8geypn5gyyShbJxW5mTgr8n8C8DZadsGU3+SGwjMr2fb88A5acs7AiuSBFLzB9stbftY4KS6zqOe80pPctOBs4D2tcoMYm2S2y9JClVp2x8Arkjm7wRuT9v2beD9en4GNfF3TVu3EDgxbflR4Of17P8vwFu1f8fqqL9XHetapK37PfAOMIvkH1VP+ZuaW3eViHgXeBK4qNamLsC0WuumkfrXvMaMOqr8NG1+WR3L7QAktZH0x6Tb9zmpVsBmyv4u25+ADyLi2mR5G1KtmjlJt2oxqVZd57TzSY+39rml6w5Mi4iVdWyr/blMI5XgtkxbNzdtfinJOW+AY0klpWmS/iZpYD3xzIiI1bViSv85NTaebH+GW0p6MOlKfw7cC2yRoW6o+/cm3XBS//jeGRELs6jPGqHZJbnE5cCPWfcPYzapxJGuB6l/XWvkMmTLBaRaQXtHRHtg/2S9Mu0o6SJgB+BHaatnkGrJbRERmyVT+4jYOdk+h1TyqtGjgUPMAHqo7gvjtT+XHsBK1k0E2fqKVHcdAElbpW+MiHERcTSpRP048HA98XTXujd+av+cCuXfSP0O7Jr8DL/Huj+/+n4/6v29Sf6RG06qS3tOzfVJy59mmeQiYgrwEHBe2uqngR0knZJcFD4R6Euq1ZcPm5BqFSyW1JFUos1I0tAkzu9ExLK0c5gDjAb+S1L75AbBdpIOSIo8DJwnqZukDqzfck03llRSvEZSW0mtJX0r2fYAcL6knpLakfpDf6ieVl8mE4CdJe0uqTVwRdp5tlLq+cBNI2IF8Dmwuo46XifVOvuVpJaSBgFHAg9uQDyNtQnwJbBEUlfgl7W2f0rq2mVj/IZUEjyd1I2xuxvRurcsNMskl7iS1MVgAJJuwhGkWlwLgV8BR0TEgjwd70ZS19UWAK8Bz2S534mkrh++p7V3WP+QbDuV1MX3yaRukvwZ2DrZdhswilRieZPUDYM6ReqZrSNJXVifDsxMjgtwB3APqe71J6QuzP80y9hrH+dDUp/7c8BHwMu1inwfmJp0Bc8GvltHHcuTWIeS+ixvBU6NiPc3JKZG+h3Qn9Q13adY/zP9d+C3yeWDCzNVJmkP4Bek4l8FXEsq4TX0D5I1kpILn2ZmFak5t+TMrBlwkjOziuYkZ2YVzUnOzCpaSb0wvFnHzWPrrg09zmWl5h+fflHsEKwRVn4xj9XLPs/4bGZjVLffJmLlsswFgVg2f1REDMnn8TMpqSS3ddce3D1iTLHDsEY45voxxQ7BGuHThy/Ie52xchkb7XhCVmW/fvuWbN4QyauSSnJmVo4EJTzymJOcmeVGQFXpvqThJGdmuVNeL/PllZOcmeXI3VUzq3RuyZlZxRJuyZlZJZNbcmZW4Xx31cwql288mFklE+6umlmFc0vOzCqXu6tmVumq3F01s0rld1fNrLK5u2pmlc53V82sorklZ2YVS36ty8wqnW88mFnl8o0HM6t07q6aWcXyeHJmVtncXTWzSufuqplVNN9dNbOKJXdXzazSubtqZpVMTnJmVqlSo587yZlZpVIylSgnOTPLkaiqKt0bD6UbmZmVDUlZTVnUM0TSB5KmSLqoju03SHo7mT6UtDhTnW7JmVnO8nFNTlI1cAtwKDATGCdpZERMrikTEeenlf8p0C9TvW7JmVlu1IipYXsBUyLi44hYDjwIHN1A+ZOBBzJV6pacmeVEZNcVTWwhaXza8vCIGJ7MdwVmpG2bCexd5zGlbYCewAuZDugkZ2Y5a8SNhwURMSAPhzwJ+HNErMpU0EnOzHKWp+fkZgHd05a7JevqchJwbjaV+pqcmeUmf9fkxgG9JfWU1IpUIhu53uGknYAOwKvZhOckZ2Y5y8cjJBGxEhgGjALeAx6OiEmSrpR0VFrRk4AHIyKyic3dVTPLSSNvPDQoIp4Gnq617rJay1c0pk4nOTPLmd9dNbPKJVCVk5yZVTC35MysojnJmVnFyueNh0JwkjOz3JVujnOSM7Mcyd1VM6twpTxoppOcmeWudBtyfq0rV6/+7TmOO2QAxxzYj7v+cMN62x+9/w5OHroP3z1iX358whA+/uj9dbbPnT2DA3btyr23/b6pQm7WBvXpzJhLD+alyw/mnEN711nmiH5deP6Sg3jukgP5/Wl7rFnfpcPG3HfuQF747UE8f8lBdOu4cVOFXfLyNTJwIRS0JSdpCHATUA3cHhHXFPJ4TW3VqlX8xxUXcvNdj9N5qy784DsHst/BQ+nVe6c1ZQ478jiOPeV0AF587mluvPoS/vvOR9dsv/HqSxh4wCFNHntzVCW46oTdOOXmvzNn8TKe/OUBPPvOXD6a+8WaMtt2asu5g3tzzPUvsWTZCjZv12rNthtP7c/vR33IS+/Pp02ralZn9eZk5StmAstGwVpyaUMZDwX6AidL6luo4xXDpAlv0G2bXnTtsS0tW7Vi8BHH8uJz67x2R7tN2q+ZX7Z06Tq/DGNGP0mXbtuskxStcHbftgNTF3zF9IVLWbEqGPnmLAbvttU6ZU7ZZxvuevETlixbAcDCL5cD0HurTaiuEi+9Px+ApctX8fWKjEOZNRvNtSW3ZihjAEk1QxlPbnCvMjL/0zlsuXXXNcudt+rCpAlvrFfukXtu4/47bmHF8hXcem9q5JilX33J3cNv4ua7HuPe291VbQpbbdqa2YuWrVmes2gZ/bbtsE6ZXp3bAfCX8/elukrc8PQHjHlvHr06t+XzZSsYfsaedN+8DS9/MJ9/HzHZrblEs2zJUfdQxl1rF5J0pqTxksYv/mxhAcMpnuO//2Me++vbDPv1Fdxxy3UA3HbTNZz8w3No07ZdkaOzdNXVomfndpxw0ysMu/MNrj1ld9pv3ILqKrHXdptz1WOTOOK6F+mxRVuO/2aPYodbMlSlrKZiKPrd1WR89+EAfXbtV1b/Lnbacms+nbN24NJ5c2fTacut6y0/+IhjufbSCwB4d8IbvPDMCG6+9jK++HwJVVVVtNpoI0449cyCx91czV3yNV06rL1ZsHWHjZm75Ot1ysxZvIy3py5m5epgxsKlfDzvS3p2asecxV8zeeYSpi9cCsCoCXPo37MjD706vUnPoSSV+HNyhWzJNWYo47LUd7f+zJj6D2bNmMqK5csZ/eSj7Hfw0HXKTP/kH2vmX/nrKLpv2wuA2x76P0a8+A4jXnyHk374E077yQVOcAU2Ydpitu3Ulu6bt6FltTiqf1eenTh3nTKjJ8zlm703B6BD21b06tyOaQu/YsK0RbTfuCUdkxsR39qx0zo3LJozAVJ2UzEUsiW3ZihjUsntJOCUAh6vybVo0YJfXn4d5512LKtXr+LI477Hdjv04Y83XE2fXfux/yHf5pF7hjP273+jRYsWtG+/GZdf9z/FDrvZWrU6uPThidx77kCqJR56bTofzv2CCw7fiYnTF/PsO3MZ89489u/TiecvOYjVEVz9+CQWf5W6CXHV45N48Kf7IIl3pi/m/lemFveESkZp311VliMIb1jl0reBG0k9QnJHRFzdUPk+u/aLu0eMKVg8ln/HXD+m2CFYI3z68AUsnzclrxmp9VY7RI9T/zursh9dN/SNPH1bV9YKek2urqGMzazCCKo8aKaZVSrhJGdmFa6EL8k5yZlZ7kr5xoOTnJnlpoiPh2TDSc7McpJ6Tq50s5yTnJnlSL7xYGaVzS05M6tcJX5NziMDm1lOaq7J5WM8OUlDJH0gaYqki+opc4KkyZImSbo/U51uyZlZzvLRkksbaPdQUkOzjZM0MiImp5XpDVwMfCsiFknqnKlet+TMLGd5asmtGWg3IpYDNQPtpvsxcEtELAKIiHmZKnWSM7PcJO+uZjNlkM1AuzsAO0h6RdJryffINMjdVTPLSc14clnaQtL4tOXhycC52WoB9AYGkRqj8kVJu0bE4oZ2MDPLQaPGk1vQwFBL2Qy0OxN4PSJWAJ9I+pBU0htX3wHdXTWznOVpZOA1A+1KakVqoN2Rtco8TqoVh6QtSHVfP26oUrfkzCxn+XgYOCJWShoGjGLtQLuTJF0JjI+Ikcm2wZImA6uAX0ZEg9+A5SRnZjlRHgfNrGug3Yi4LG0+gF8kU1ac5MwsZ36ty8wqWgnnOCc5M8udW3JmVrlK/AV9Jzkzy4lK/HtXneTMLGfVHjTTzCpZCTfknOTMLDeptxlKN8s5yZlZzkq4t1p/kpP0eyDq2x4R5xUkIjMrO+XakhvfwDYzszVKOMfVn+Qi4q70ZUltImJp4UMys3IioLqEs1zGoZYkDUze+H8/Wf6GpFsLHpmZlYcshz4vVpc2m/HkbgQOAxYCRMQEYP9CBmVm5SVP48kVRFZ3VyNiRq0svKow4ZhZuRFQVcLd1WyS3AxJ+wAhqSXwM+C9woZlZuWkhHNcVknubOAmUt+aM5vUyJznFjIoMysf+Rw0sxAyJrmIWAB8twliMbMyVcrd1WzurvaS9ISk+ZLmSRohqVdTBGdm5UFZTsWQzd3V+4GHga2BLsAjwAOFDMrMyku5P0LSJiLuiYiVyXQv0LrQgZlZeUjdXc1uKoaG3l3tmMz+n6SLgAdJvct6IrW+TcfMmrEittKy0dCNhzdIJbWa6M9K2xbAxYUKyszKS1neXY2Ink0ZiJmVp5ruaqnK6o0HSbsAfUm7FhcRdxcqKDMrL+XaXQVA0uXAIFJJ7mlgKPAy4CRnZkDxHg/JRjZ3V48DDgbmRsQPgW8AmxY0KjMrG1LqYeBspmLIpru6LCJWS1opqT0wD+he4LjMrIyUcG81q5bceEmbAbeRuuP6JvBqQaMys7JSVaWspkwkDZH0gaQpyaNrtbeflrx99XYynZGpzmzeXT0nmf2DpGeA9hExMWO0ZtYsiPx0RSVVA7cAhwIzgXGSRkbE5FpFH4qIYdnW29DDwP0b2hYRb2Z7EDOrYPkbEHMvYEpEfAwg6UHgaKB2kmuUhlpy/9XAtgAOyuXAdWnTqppde/ieRjlZ8MqzxQ7BGmHll58XpN5GPEKyhaT0L8kaHhHDk/muwIy0bTOBveuo41hJ+wMfAudHxIw6yqzR0MPAB2YXs5k1d9lc3E8siIgBORzqCeCBiPinpLOAu8jQ4GpEbGZm6xN5G4VkFus+udEtWbdGRCyMiH8mi7cDe2Sq1EnOzHLWoiq7KYNxQG9JPSW1Ak4CRqYXkLR12uJRZPFVDFm91mVmVp/UN3HlfuchIlZKGkbqKxaqgTsiYpKkK4HxETESOE/SUcBK4DPgtEz1ZvNal0gNf94rIq6U1APYKiLGbvjpmFklydcL+hHxNLWGcouIy9LmL6aRIyBl0129FRgInJwsf0HqWRYzM6D8v3d174joL+ktgIhYlPSXzcwq4ntXVyRPIgeApE7A6oJGZWZlpbp0c1xWSe6/gceAzpKuJjUqyW8LGpWZlQ0VcYSRbGTz7up9kt4gNdySgH+JiIy3bc2s+SjhHJfV3dUewFJSTxqvWRcR0wsZmJmVj3If/vwp1n6hTWugJ/ABsHMB4zKzMlH2Nx4iYtf05WR0knPqKW5mzVAJ57jGv/EQEW9KqmtkADNrjgTVJZzlsrkm94u0xSqgPzC7YBGZWVmphK8k3CRtfiWpa3SPFiYcMytHZZvkkoeAN4mIC5soHjMrQ2X5vauSWiSjAnyrKQMys/JSzt3VsaSuv70taSTwCPBVzcaI+EuBYzOzclDEl++zkc01udbAQlJDDNc8LxeAk5yZIaBFCTflGkpynZM7q++yNrnViIJGZWZlpVxbctVAO9ZNbjWc5MwsIarqTBOloaEkNycirmyySMysLKW+yKbYUdSvoSRXwmGbWclQ+d5dPbjJojCzsiWguoSzXENfLv1ZUwZiZuWrrEchMTPLpIRznJOcmeVGlPa31DvJmVlu8vTl0oXiJGdmOSvdFOckZ2Y5EmU+aKaZWSYlnOOc5MwsVyrpa3KlfFPEzMpAzd3VbKaMdUlDJH0gaYqkixood6ykkDQgU51OcmaWM0lZTRnqqAZuAYYCfYGTJfWto9wmwM+A17OJzUnOzHKmLKcM9gKmRMTHEbEceBA4uo5y/wpcC3ydTWxOcmaWEyVfSZjNBGwhaXzadGZaVV2BGWnLM5N1acdSf6B7RDyVbXy+8WBmOWvEjYcFEZHxOlo9x6gCrgdOa8x+bsmZWc7y1F2dBXRPW+6WrKuxCbALMEbSVOCbwMhMNx/ckjOznOXpCZJxQG9JPUklt5OAU2o2RsQSYIu1x9QY4MKIGN9QpW7JmVlOUo+QKKupIRGxEhgGjALeAx6OiEmSrpR01IbG55acmeUsX88CR8TTwNO11l1WT9lB2dTpJGdmOZIHzTSzylXTXS1VTnJmlhv5BX0zq3BOcmZW0eTuqplVKg+aaWYVr4RznJOcmeWulLurfuMhR6NHPcNuO+/Izjttz3X/cc1622+64Xr67daXPfvtxtDBBzNt2rQ12+69+y526dObXfr05t6772rKsJu1Q/fpw4THLuXdEZdz4Q8PXW9796068Mzw83j1gV8z9qGLOWzftUOaXXj6YN4dcTkTHruUQwb2acqwS5aAKmU3FUPBkpykOyTNk/RuoY5RbKtWreLn553LiCf+j7cmTuaRBx/gvcmT1ymze79+vPLaeMa9NZHvHHMcl1z8KwA+++wzrr7qd7z4yuu89PexXH3V71i0aFExTqNZqaoSN150AkcPu5V+x17F8UP2YKdeW61T5tdnDOHRZ99k4MnXcurF/8tNF58IwE69tuL4w/rT/7irOercW7np4hOoKtZfbklR1v8VQyFbcncCQwpYf9GNGzuW7bbbnp69etGqVSuOP/EknnxixDplDhh0IG3atAFgr72/yayZMwF4dvQoDj74UDp27EiHDh04+OBDGT3qmSY/h+Zmz1225R8zFjB11kJWrFzFI6Pe5IhBu61TJiJo37Y1AJu225g585cAcMSg3Xhk1JssX7GSabMX8o8ZC9hzl22b+hRKT/KcXDZTMRTsmlxEvChp20LVXwpmz55Ft25rR4bp2rUbY8fWPyLznf/7Jw4bMnTtvt3T9u3WjdmzZ9W3q+VJl86bMvPTtS3mWZ8uYq9aierqPz7NE7cO4ycnHUCbjTfi8LN/D0DXTpvy+jtT1+47bxFdOm/aFGGXNN9dzSAZGfRMgO49ehQ5msJ54L57efON8Tz7wt+KHYplcMKQAdz7xGvcdM8L7L1bT/501anscdy/FTuskla6Ka4EbjxExPCIGBARAzpt0anY4TRKly5dmTlz7WjNs2bNpGvXruuVe+H557j2mqv582Mj2WijjdbuOyNt35kz6dJl/X0tv2bPW0K3LTusWe66ZQdmJd3RGj/4l4E8OvpNAF6f+AmtW7Vki83aMmv+ErptlbZv5w7Mnrfuvs1WnkbNLISiJ7lyNmDPPZky5SOmfvIJy5cv55GHHuTwI9Yd9urtt95i2Dln8ee/jKRz585r1h86+DCee240ixYtYtGiRTz33GgOHXxYU59CszN+0jS279GJbbpsTssW1Rx/WH+eGjNxnTIz5n7GoL12BGDHnlvSeqOWzF/0JU+Nmcjxh/WnVcsWbNNlc7bv0Ylx704twlmUnlK+8VD07mo5a9GiBTfcdDNHHn4Yq1at4gennU7fnXfmyisuo/8eAzjiyKP4zUW/5Ksvv+S7Jx0PpLrkf35sJB07duTi31zKvgP3BOA3l1xGx44di3k6zcKqVas5/9qHeeLWc6muEneNeI33Pp7LpT85nDcnT+epv73DRdc/xq2XnsxPv3cgEfDjy+4B4L2P5/Lo6Ld469FLWLlqNT+/5mFWr44in1FpKOFLciiiMD8kSQ8Ag0gNV/wpcHlE/KmhffbYY0C88nqDIxlbiemw57Bih2CN8M8PHmb10nl5TUl9du0Xd48Yk1XZvbbb7I0N/SKbDVXIu6snF6puMysdolHf1tXk3F01s9x4PDkzq3QlnOOc5MwsD0o4yznJmVmOivd4SDac5MwsJzWjkJQqJzkzy52TnJlVMndXzayi+RESM6toJZzj/IK+meUo2xFIssiEkoZI+kDSFEkX1bH9bEnvSHpb0suS+tZVTzonOTPLSeruqrKaGqxHqgZuAYYCfYGT60hi90fErhGxO/AfwPWZ4nOSM7Oc5akhtxcwJSI+jojlwIPA0ekFIuLztMW2QMYRRnxNzsxyl5+Lcl2BGWnLM4G91zuUdC7wC6AVcFCmSt2SM7OcNWLQzC0kjU+bzmzssSLilojYDvg18NtM5d2SM7OcNeIRkgUNjCc3C+iettwtWVefB4H/yXRAt+TMLGd5uiY3DugtqaekVsBJwMh1jiP1Tls8HPgoU6VuyZlZTvI1aGZErJQ0DBgFVAN3RMQkSVcC4yNiJDBM0iHACmAR8INM9TrJmVlu8jhoZkQ8DTxda91lafM/a2ydTnJmlrNSfuPBSc7MclfCWc5Jzsxy5EEzzayCedBMM6t8TnJmVsncXTWziuZBM82sopVwjnOSM7Mc5fFh4EJwkjOznOTrta5CcZIzs5yVbopzkjOzPCjhhpyTnJnlzo+QmFllK90c5yRnZrkr4RznJGdmuZHI+HWDxeQkZ2a5K90c5yRnZrkr4RznJGdmuSvh3qqTnJnlyoNmmlkFS73WVewo6uckZ2Y5c5Izs4rm7qqZVS4PtWRmlUz4ERIzq3QlnOWc5MwsZ36ty8wqWummOCc5M8uHEs5yTnJmlrNSfoREEVHsGNaQNB+YVuw4CmALYEGxg7BGqdSf2TYR0SmfFUp6htTnlY0FETEkn8fPpKSSXKWSND4iBhQ7Dsuef2aVo6rYAZiZFZKTnJlVNCe5pjG82AFYo/lnViF8Tc7MKppbcmZW0ZzkzKyiOcmZWUVzkisgSTtKGiippaTqYsdj2fHPqrL4xkOBSDoG+DdgVjKNB+6MiM+LGpjVS9IOEfFhMl8dEauKHZPlzi25ApDUEjgR+FFEHAyMALoDv5bUvqjBWZ0kHQG8Lel+gIhY5RZdZXCSK5z2QO9k/jHgSaAlcIpUwoNvNUOS2gLDgJ8DyyXdC050lcJJrgAiYgVwPXCMpP0iYjXwMvA2sG9Rg7P1RMRXwOnA/cCFQOv0RFfM2Cx3TnKF8xIwGvi+pP0jYlVE3A90Ab5R3NCstoiYHRFfRsQC4Cxg45pEJ6m/pJ2KG6FtKI8nVyAR8bWk+4AALk7+SP4JbAnMKWpw1qCIWCjpLOA6Se8D1cCBRQ7LNpCTXAFFxCJJtwGTSbUOvga+FxGfFjcyyyQiFkiaCAwFDo2ImcWOyTaMHyFpIskF7Eiuz1mJk9QBeBi4ICImFjse23BOcmb1kNQ6Ir4udhyWGyc5M6tovrtqZhXNSc7MKpqTnJlVNCc5M6toTnJlRNIqSW9LelfSI5La5FDXnZKOS+Zvl9S3gbKDJO2zAceYKmm97+Osb32tMl828lhXSLqwsTFa5XOSKy/LImL3iNgFWA6cnb5R0gY93B0RZ0TE5AaKDAIaneTMSoGTXPl6Cdg+aWW9JGkkMFlStaTrJI2TNDF5PQml3CzpA0nPAZ1rKpI0RtKAZH6IpDclTZD0vKRtSSXT85NW5H6SOkl6NDnGOEnfSvbdXNJoSZMk3Q5kHG1F0uOS3kj2ObPWthuS9c9L6pSs207SM8k+L/mdUsvEr3WVoaTFNhR4JlnVH9glIj5JEsWSiNhT0kbAK5JGA/2AHYG+pN6fnQzcUXiV+jAAAAH/SURBVKveTsBtwP5JXR0j4jNJfwC+jIj/TMrdD9wQES9L6gGMAvoAlwMvR8SVkg4HfpTF6ZyeHGNjYJykRyNiIdAWGB8R50u6LKl7GKmvCjw7Ij6StDdwK3DQBnyM1kw4yZWXjSW9ncy/BPyJVDdybER8kqwfDOxWc70N2JTUuHb7Aw8kQwfNlvRCHfV/E3ixpq6I+KyeOA4B+qYNi9deUrvkGMck+z4laVEW53SepO8k892TWBcCq4GHkvX3An9JjrEP8EjasTfK4hjWjDnJlZdlEbF7+orkj/2r9FXATyNiVK1y385jHFXAN2u/8tTYsUAlDSKVMAdGxFJJY4DW9RSP5LiLa38GZg3xNbnKMwr4STIEO5J2SEa+fRE4MblmtzV1Dx30GrC/pJ7Jvh2T9V8Am6SVGw38tGZBUk3SeRE4JVk3FOiQIdZNgUVJgtuJVEuyRhVQ0xo9hVQ3+HPgE0nHJ8eQJI/NZw1ykqs8t5O63vampHeBP5JqsT8GfJRsuxt4tfaOETEfOJNU13ACa7uLTwDfqbnxAJwHDEhubExm7V3e35FKkpNIdVunZ4j1GaCFpPeAa0gl2RpfAXsl53AQcGWy/rvAj5L4JgFHZ/GZWDPmF/TNrKK5JWdmFc1JzswqmpOcmVU0Jzkzq2hOcmZW0ZzkzKyiOcmZWUX7f8Vrr4vqG89dAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 2 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2jbHydbkDJ-l"
      },
      "source": [
        "# K Nearest Neighbours (KNN) Classifier \n",
        "\n",
        "Below is an example of a KNN classifier, however this is not a veery good approach for this type of task and yields an accuracy of only 61%.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tCCdAzJxXGUT",
        "outputId": "1bd6ee52-c81e-45d6-9f08-4f316f91c219",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 52
        }
      },
      "source": [
        "# apply pre_proc() function to all data splits\n",
        "proc_train = pre_proc(df_train['review'])\n",
        "proc_val = pre_proc(df_val['review'])\n",
        "proc_test = pre_proc(df_test['review'])\n",
        "\n",
        "# call extract_features() to extract features\n",
        "# you can play with the parameters of this function do generate different feature vectors:\n",
        "### binary = True --> means that it will create a binary vector (either a word appears in a review or not)\n",
        "###          changing it to False will use frequencies (how many times a word appears in a review)\n",
        "###          it will also normalise it. \n",
        "### max_df = ignore terms that have a frequency higher than the value assigned (1.0 means no term is discarded)\n",
        "### min_df = ignore terms that have a frequency lower than the value assigned (0.0 means no term is discarded)\n",
        "### ngram_range = range of the n_grams. If (1,1) only single words are considered. If (1,2) single words and 2-word groups are considered\n",
        "###               If (1,3) single words, 2-word, and 3-word groups are considered. Be careful because this can create far too many positions in your vector!\n",
        "### sw = if True, it will use a pre-defined stopwords list to filter out some terms.  \n",
        "X_train, X_val, X_test = extract_features(proc_train, proc_val, proc_test, binary=True, max_df=10.0, min_df=0.0, ngram_range=(1,1), sw=False)\n",
        "\n",
        "# extract labels\n",
        "y_train = df_train['sentiment']\n",
        "y_val = df_val['sentiment']\n",
        "y_test = df_val['sentiment']"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "** Vocabulary size: 80566\n",
            "** Words:\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HJxDnzCX6d5v"
      },
      "source": [
        "# example of a sklearn classifier \n",
        "# in the first code block here we also import several others classifiers available in sklearn\n",
        "# they all follow the same structure:\n",
        "## 1-) create an instance of the classifier (e.g. kkn = NeighborsClassifier())\n",
        "## 2-) train the model using the training data (e.g. knn.fit(X_train, y_train))\n",
        "## 3-) generate predictions for the test set (e.g. preds = knn.predict(X_test))\n",
        "## link ot more sklearn classifiers: https://scikit-learn.org/stable/supervised_learning.html\n",
        "knn = KNeighborsClassifier()\n",
        "knn.fit(X_train, y_train)\n",
        "knn_preds_val = knn.predict(X_val)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EP6l0GfkExKA",
        "outputId": "1d124aba-8286-40a6-9ae0-0d043b5891ed",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 581
        }
      },
      "source": [
        "# evaluating the model in the validation set (you do not have gold values for test set here!)\n",
        "print(\"Predicted values for the validation set:\")\n",
        "print(knn_preds_val)\n",
        "evaluate_classifier(y_val, knn_preds_val)\n",
        "print(y_val)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Predicted values for the validation set:\n",
            "['negative' 'negative' 'negative' ... 'negative' 'negative' 'negative']\n",
            "*** Accuracy score: 0.6118\n",
            "\n",
            "0       negative\n",
            "1       negative\n",
            "2       negative\n",
            "3       negative\n",
            "4       negative\n",
            "          ...   \n",
            "9995    positive\n",
            "9996    negative\n",
            "9997    negative\n",
            "9998    negative\n",
            "9999    negative\n",
            "Name: sentiment, Length: 10000, dtype: object\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAATkAAAEYCAYAAAAnEYFiAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dd5xdZZ3H8c93ZlIIpEESIJUkJEDoIfRiVEoQhLXQVRBYQEBcFBFZFjEuroiCrsBqKCJSQpESgQUUNkBQSIEESCASSU9IIwFppsxv/zhnkpvJlDu5984t833zOi9Oee5zfufOzC/Pc8pzFBGYmVWqqmIHYGZWSE5yZlbRnOTMrKI5yZlZRXOSM7OK5iRnZhXNSa5ESBov6ex0/jRJT+W5/h0khaSafNbbzD4l6beSVkqamEM9h0qamc/YikVSf0kfSKoudixtRZtJcpLmSFoqacuMdWdLGl/EsBoUEXdFxJHFjiMPDgGOAPpGxH6bW0lEPB8RO+UvrMJIf8cOb6pMRMyLiK0iYl1rxdXWtZkkl6oGvpVrJWkLpa19d5tjADAnIj4sdiCloDVb0bZBW/tDvRa4RFK3hjZKOkjSJEnvpf8/KGPbeElXS3oB+AgYlHb/zpf0lqR/SPqRpMGS/iLpfUn3SWqffr67pEclLUu7b49K6ttIHGdImpDOX5p2b+qmNZJuT7d1lXSrpMWSFkr6z7pukKRqST+TtFzS28AxTX0xkvpJejCNb4WkG9L1VZKukDQ3bQnfIalruq2uC3y6pHnpvv493XYWcAtwYBr3DzOPK2O/IWnHdP5zkmak3+VCSZek60dKWpDxmV3Sn8cqSdMlHZex7XZJN0p6LK3nJUmDGznmuvi/Lml++nM5T9K+kl5N678ho/xgSc+k389ySXfV/S5J+j3QH/hjeryXZtR/lqR5wDMZ62okbS1pgaTPp3VsJWmWpK819bOyFoqINjEBc4DDgQeB/0zXnQ2MT+e3BlYCXwVqgFPS5W3S7eOBecCu6fZ2QACPAF3S9f8EngYGAV2BGcDp6ee3Ab4EdAI6A/cDD2fENx44O50/A5jQwDH0AxYBR6fLDwG/AbYEegETgXPTbecBb6af2Rr4vzTemgbqrQamAdendXUEDkm3nQnMSo9pq/T7+326bYe0zpuBLYA90+9gl4aOo6HjSj+/Yzq/GDg0ne8ODE/nRwIL0vl2aTyXA+2BzwD/AHZKt98OrAD2S39OdwFjG/mdqIv/1+kxHwl8Ajycfp99gKXAp9LyO5J0vzsAPYHngF/U/x1roP470u91i4x1NWmZI4F30v3dDDxQ7L+VSpuKHkCrHeiGJLcb8F76S5qZ5L4KTKz3mb8CZ6Tz44HR9bYHcHDG8hTgexnLP8/8I6j32b2AlRnL42kiyaV/IOvrB7ZNE8oWGWVOAf4vnX8GOC9j25E0nuQOBJY1su1p4PyM5Z2ANWkCqfuD7ZuxfSJwckPH0chxZSa5ecC5QJd6ZUayIckdmiaFqozt9wBXpfO3A7dkbPsc8GYjP4O6+PtkrFsBnJSx/Afg3xr5/L8Ar9T/HWug/kENrKvJWPcr4DVgIek/qp7yN7W17ioR8TrwKHBZvU29gbn11s0l+de8zvwGqlySMf9xA8tbAUjqJOk3abfvfZJWQDdlf5XtVmBmRFyTLg8gadUsTrtVq0hadb0yjicz3vrHlqkfMDci1jawrf73MpckwW2bse6djPmPSI95M3yJJCnNlfSspAMbiWd+RNTWiynz59TSeLL9GW4raWzalX4fuBPo0Uzd0PDvTaYxJP/43h4RK7Koz1qgzSW51A+Af2XjP4xFJIkjU3+Sf13r5DJky3dIWkH7R0QX4LB0vZr7oKTLgKHAWRmr55O05HpERLd06hIRu6bbF5Mkrzr9m9jFfKC/Gj4xXv976Q+sZeNEkK0PSbrrAEjaLnNjREyKiONJEvXDwH2NxNNPG1/4qf9zKpQfk/wO7J7+DL/Cxj+/xn4/Gv29Sf+RG0PSpT2/7vyk5U+bTHIRMQu4F7goY/XjwFBJp6YnhU8ChpG0+vKhM0mrYJWkrUkSbbMkHZ3G+YWI+DjjGBYDTwE/l9QlvUAwWNKn0iL3ARdJ6iupO5u2XDNNJEmKP5G0paSOkg5Ot90DXCxpoKStSP7Q722k1decacCukvaS1BG4KuM42yu5P7BrRKwB3gdqG6jjJZLW2aWS2kkaCXweGLsZ8bRUZ+AD4D1JfYDv1tu+hOTcZUtcTpIEzyS5MHZHC1r3loU2meRSo0lOBgOQdhOOJWlxrQAuBY6NiOV52t8vSM6rLQdeBJ7I8nMnkZw/fEMbrrD+Ot32NZKT7zNILpI8AGyfbrsZeJIksbxMcsGgQZHcs/V5khPr84AF6X4BbgN+T9K9nk1yYv6bWcZefz9/I/ne/wy8BUyoV+SrwJy0K3gecFoDdaxOYz2a5Lu8CfhaRLy5OTG10A+B4STndB9j0+/0v4Ar0tMHlzRXmaR9gG+TxL8OuIYk4TX1D5K1kNITn2ZmFaktt+TMrA1wkjOziuYkZ2YVzUnOzCpaST0wrJotQu07FzsMa4G9d2nq9jsrNXPnzmH58uXN3pvZEtVdBkSs/bj5gkB8vOzJiBiVz/03p7SSXPvOdNjpxGKHYS3wwks3NF/ISsbB+4/Ie52x9uOs/24/mXpjNk+I5FVJJTkzK0eCEh55zEnOzHIjoKp0H9JwkjOz3Cmvp/nyyknOzHLk7qqZVTq35MysYgm35MysksktOTOrcL66amaVyxcezKySCXdXzazCuSVnZpXL3VUzq3RV7q6aWaXys6tmVtncXTWzSuerq2ZW0dySM7OKJT/WZWaVzhcezKxy+cKDmVU6d1fNrGJ5PDkzq2zurppZpXN31cwqmq+umlnFUml3V0s3MjMrH3U3BDc3NVuNRkmaKWmWpMsa2H69pKnp9DdJq5qr0y05M8uZ8nBOTlI1cCNwBLAAmCRpXETMqCsTERdnlP8msHdz9bolZ2Y5SUY/V1ZTM/YDZkXE2xGxGhgLHN9E+VOAe5qr1EnOzHKjFkzQQ9LkjOmcjJr6APMzlhek6zbdpTQAGAg801x47q6aWY5EVVXW7aXlETEiDzs9GXggItY1V9BJzsxylo9zcsBCoF/Gct90XUNOBi7IplJ3V80sZ3k6JzcJGCJpoKT2JIlsXAP72hnoDvw1m9ic5MwsNy07J9eoiFgLXAg8CbwB3BcR0yWNlnRcRtGTgbEREdmE5+6qmeVEZNVKy0pEPA48Xm/dlfWWr2pJnU5yZpazFlx4aHVOcmaWs3y15ArBSc7McpPF+bZicpIzs5y5JWdmFSufFx4KwUnOzHLmJGdmlUugKic5M6tgbsmZWUVzkjOziuULD2ZW+Uo3xznJmVmO5O6qmVU4P7tqZpWtdBtyHk8uV0cctAvTHvoPXn/kB1zy9SM22f7T73yRF8dexotjL+PVh69k8XM/Xb/t6m8dz5QH/p1X/nAFP7/0y60Zdpv11JNPsMeuO7Hrzjty7U9/ssn2X15/HXvvMYx9996Do4/8LHPnzgVg2tSpfOqQAxm+567su/ce3H/fva0deknL06CZBVHQlpykUcAvgWrglojY9LeqjFVViV9cdiLHfOMGFi5ZxYS7vsujz77Gm2+/s77MpT9/cP38N07+FHvu1BeAA/YcyIF7DWLfE38MwDO//TaH7jOE56e81boH0YasW7eOf7voAh773z/Rp29fDjlgX4499jh2GTZsfZm99t6bF86dTKdOnRjz6//h379/KXfefS+dOnXi1t/ewY5DhrBo0SIO3n8fjjjyKLp161bEIyoNxUxg2ShYSy7jHYpHA8OAUyQNa/pT5WXf3Xbg7/OXM2fhCtasXcf9T77MsSP3aLT8iaP24b4npgAQAR3at6N9uxo6tK+hpqaape++31qht0mTJk5k8OAdGThoEO3bt+eEk07m0T8+slGZT438NJ06dQJgv/0PYOGCBQAMGTqUHYcMAaB379707NmL5cuWte4BlLBSbskVsrva0ncolp3evbqyYMnK9csLl6ykT8+uDZbtv313BvTehvGTZgLw0quzeW7yW8z+09XMfurH/PkvbzBz9pJWibutWrRoIX37bnhPSp8+fVm4sLH3pMDtv72Vo0Ydvcn6SRMnsnrNagYNHlyQOMtRKSe5QnZXG3qH4v71C6XvXUzevdhuqwKGU1wnHLUPDz89ldraZFj6Qf16sNPAbdnxqCsAeOzX3+TgvwzmhVf+XswwLXXPXXfy8pTJ/OmZZzdav3jxYs76+le5+dbflfQVxdZWys+uFv2nFBFjImJERIxQzRbFDqdFFi19j77bdl+/3Gfb7ixc9l6DZb981D7c98Tk9cvHf3pPJr42hw8/Xs2HH6/myRems/8eAwsec1vWu3cfFizY8O/uwoUL6NNn03cXP/P0n7nmJ1fzwEPj6NChw/r177//Pl887hiuGn01+x9wQKvEXBZU2i25Qia5lrxDsSxNnj6XHfv3ZEDvbWhXU80JRw3nsfGvblJu6A7b0r1LJ16cNnv9uvnvrOTQfXakurqKmpoqDh0+hDdnv7PJZy1/Ruy7L7NmvcWc2bNZvXo19987lmOOPW6jMlNfeYULzz+XBx4cR69evdavX716NSd9+Quc+pWv8cUv+Up4JgFSdlMxFLK7uv4diiTJ7WTg1ALur9WtW1fLxdfcxx9vuoDqKvG7R17kjbff4T++cQwvz5jHY8++BiRd1fufnLLRZx/88yt8at+hTL7vcoLgT395g8efe70Yh9Fm1NTUcP0vb+DzxxzFunXrOP2MMxm2666MvupKhu8zgmM/fxyXX/ZdPvzgA047+QQA+vXvzwMPjeMP99/HhOef490VK7jzjtsBGHPr7ey5115FPKJSUdpXV5Xlqws3r3Lpc8AvSG4huS0irm6qfFWnXtFhpxMLFo/l38pJNxQ7BGuBg/cfwZQpk/OakTpuNzT6f+2/syr71rVHT4mIEfncf3MKep9cQ+9QNLMKo+Se0VLlx7rMLCfCSc7MKlwJn5JzkjOz3JXyhQcnOTPLTRFvD8mGk5yZ5SS5T650s5yTnJnlSCV94aHoj3WZWfnL12NdkkZJmilplqTLGilzoqQZkqZLuru5Ot2SM7Pc5OmcXMbwbEeQDOgxSdK4iJiRUWYI8H3g4IhYKalXw7Vt4JacmeWk7pxcHlpy2QzP9q/AjRGxEiAiljZXqZOcmeWsBQ/o95A0OWM6J6OahoZnqz9MzFBgqKQXJL2Yjj7eJHdXzSxnLbi6ujzHZ1drgCHASJKRjZ6TtHtErGrqA2Zmmy9/z65mMzzbAuCliFgDzJb0N5KkN6mxSt1dNbOc5HE8ufXDs0lqTzI827h6ZR4macUhqQdJ9/Xtpip1S87McpSf8eQiYq2kC4En2TA823RJo4HJETEu3XakpBnAOuC7EbGiqXqd5MwsZ/l64KGh4dki4sqM+QC+nU5ZcZIzs5z5sS4zq1jyoJlmVunckjOzilbCOc5Jzsxy55acmVUuD5ppZpVMJf7eVSc5M8tZta+umlklK+GGnJOcmeUmeS61dLOck5yZ5ayEe6uNJzlJvwKise0RcVFBIjKzslOuLbnJrRaFmZW1Es5xjSe5iPhd5rKkThHxUeFDMrNyIqC6hLNcs4NmSjowHbvpzXR5T0k3FTwyMysPWb7Eplhd2mxGBv4FcBSwAiAipgGHFTIoMysveRoZuCCyuroaEfPrZeF1hQnHzMqNgKoS7q5mk+TmSzoICEntgG8BbxQ2LDMrJyWc47JKcucBvyR5/+EikjHWLyhkUGZWPsp+0MyIWA6c1gqxmFmZKuXuajZXVwdJ+qOkZZKWSnpE0qDWCM7MyoOynIohm6urdwP3AdsDvYH7gXsKGZSZlZdyv4WkU0T8PiLWptOdQMdCB2Zm5SG5uprdVAxNPbu6dTr7v5IuA8aSPMt6EvXei2hmbVgRW2nZaOrCwxSSpFYX/bkZ2wL4fqGCMrPyUpZXVyNiYGsGYmblqa67WqqyeuJB0m7AMDLOxUXEHYUKyszKS7l2VwGQ9ANgJEmSexw4GpgAOMmZGVC820Oykc3V1S8DnwXeiYivA3sCXQsalZmVDSm5GTibqRiy6a5+HBG1ktZK6gIsBfoVOC4zKyMl3FvNqiU3WVI34GaSK64vA38taFRmVlaqqpTV1BxJoyTNlDQrvXWt/vYz0qevpqbT2c3Vmc2zq+ens7+W9ATQJSJebTZaM2sTRH66opKqgRuBI4AFwCRJ4yJiRr2i90bEhdnW29TNwMOb2hYRL2e7EzOrYPkbEHM/YFZEvA0gaSxwPFA/ybVIUy25nzexLYDP5LLjhnTpuQ2HnPe1fFdrBXTzi7OLHYK1wLIP/1mQeltwC0kPSZkvyRoTEWPS+T7A/IxtC4D9G6jjS5IOA/4GXBwR8xsos15TNwN/OruYzayty+bkfmp5RIzIYVd/BO6JiH9KOhf4Hc00uFoQm5nZpkTeRiFZyMZ3bvRN160XESsioq45eguwT3OVOsmZWc5qqrKbmjEJGCJpoKT2wMnAuMwCkrbPWDyOLF7FkNVjXWZmjUnexJX7lYeIWCvpQpJXLFQDt0XEdEmjgckRMQ64SNJxwFrgXeCM5urN5rEukQx/PigiRkvqD2wXERM3/3DMrJLk6wH9iHicekO5RcSVGfPfp4UjIGXTXb0JOBA4JV3+B8m9LGZmQPm/d3X/iBgu6RWAiFiZ9pfNzCrivatr0juRA0BST6C2oFGZWVmpLt0cl1WS+2/gIaCXpKtJRiW5oqBRmVnZUBFHGMlGNs+u3iVpCslwSwL+JSKavWxrZm1HCee4rK6u9gc+IrnTeP26iJhXyMDMrHyU+/Dnj7HhhTYdgYHATGDXAsZlZmWi7C88RMTumcvp6CTnN1LczNqgEs5xLX/iISJeltTQyABm1hYJqks4y2VzTu7bGYtVwHBgUcEiMrOyUgmvJOycMb+W5BzdHwoTjpmVo7JNculNwJ0j4pJWisfMylBZvndVUk06KsDBrRmQmZWXcu6uTiQ5/zZV0jjgfuDDuo0R8WCBYzOzclDEh++zkc05uY7ACpIhhuvulwvASc7MEFBTwk25ppJcr/TK6utsSG51oqBRmVlZKdeWXDWwFRsntzpOcmaWElUNponS0FSSWxwRo1stEjMrS8mLbIodReOaSnIlHLaZlQyV79XVz7ZaFGZWtgRUl3CWa+rl0u+2ZiBmVr7KehQSM7PmlHCOc5Izs9yI0n5LvZOcmeUmTy+XLhQnOTPLWemmOCc5M8uRKPNBM83MmlPCOc5JzsxyJZ+TM7PK5aurZlbxSrklV8oJ2MzKhLKcmq1HGiVppqRZki5rotyXJIWkEc3V6ZacmeVEeXolYfpOmRuBI4AFwCRJ4yJiRr1ynYFvAS9lU69bcmaWM0lZTc3YD5gVEW9HxGpgLHB8A+V+BFwDfJJNbE5yZpazPHVX+wDzM5YXpOs27EcaDvSLiMeyjc3dVTPLWQt6qz0kTc5YHhMRY7Lbh6qA64AzWhKbk5yZ5SS5hSTrLLc8Ihq7WLAQ6Jex3DddV6czsBswPu36bgeMk3RcRGQmzo04yZlZzvJ0B8kkYIikgSTJ7WTg1LqNEfEe0GPDPjUeuKSpBAdOcmaWM+Vl0Mz0ZfYXAk+SvEjrtoiYLmk0MDkixm1OvU5yZpaTFnZXmxQRjwOP11t3ZSNlR2ZTp5OcmeVGfkDfzCqck5yZVTSV8LCZTnJmlhMPmmlmFa+Ec5yTnJnlzt3VCrZ0+l+Zcd91RG0t/Q4+jh1Hnb7R9vl/eZQ3H/wVHbv1BGDAyBPof0jyzPFj3ziQLn0GA9Bx6+3Y9/yftW7wbdSMl57lwV+Opra2lgOPPZEjvvKNBstNHf+/3PYfF3DJzQ/Tf+c9WLF4AT/+yhH06j8IgB123YuTLrm6NUMvSQKqSjfHFS7JSboNOBZYGhG7FWo/xRS165h+z7Xs/61f0bF7Lyb81xlsu8ehdO49aKNy2+9zOLud8t1NPl/dvgOHXnFna4VrQO26ddx/3Q+44Po76NZzO372r//CbgcfzvYDh2xU7pOPPuDZB25nwLC9Nlrfo88AvvfbrJ8NbyNU0i25Qo5CcjswqoD1F92qOTPo1KsvnXr2oaqmHb33PYIlrz5X7LCsCXPfmEbPPgPo0bs/Ne3aM/yzx/LahD9tUu6xW67j8FPPpV37DkWIssyk98llMxVDwZJcRDwHvFuo+kvBJyuXskX3bdcvd+zWi09WLtuk3Duv/B/P/eg0pvzmMj5+d8n69bVrVjPhx6fzwjVn8s7UZ1sl5rZu1bJ36NZr+/XL3Xpuz3vLl2xUZv7M11m1dDG7HvSZTT6/YvF8rjnzWH554cn8fdrEgsdbDuqurmYzFUPRz8lJOgc4B2CLrbcrcjT5t+0eh9J73yOpbteeuc89yLTf/ZADLr4JgM9c/TAdu/fio2ULefH6C+jcZzBb9uxb5IjbttraWh664WpOu/zaTbZ12aYnP3xgAlt27c68ma9xy+Xn8f07nmCLLTsXIdLSUrqd1RIYNDMixkTEiIgY0X6rbsUOp0U6du/Fxys3tAI+WbWUjt17blSm/VZdqW7XHoD+hxzPe3Pf3OjzAJ169mGbocN5f97MVoi6bevWcztWLV28fnnVssV07bGhNf7Pjz5g8ey/8auLTuGqEw5lzoxXGHPZOcx781Xate/All27A9B/p93p0bs/y+bPbvVjKEn5eslDARQ9yZWzrgN24cOl8/lo+SJq165h0aQ/se0eh21U5pP3lq+fXzLtebbafgcA1nz4PuvWrAZg9QerePfv09hq+4GtFntb1X/nPVi2YA4rFs1n7ZrVvPz0o+x+yOHrt2+xVRf+69EpXHX/81x1//PsMGxvzvnJGPrvvAf/WLmC2nXrAFi+aB7LFsxhm979i3UoJUVZ/lcMRe+ulrOq6hp2O+kSJv73RURtLX0P+jydew9i5rjf0G3ALmy752HMeeZelrz6PKqqpt2WXdjz9GRAhX+8M4fX7/pJcjY2gsGjTt/kqqzlX3VNDV+++Cpu+s7p1NbWcsAxJ7D9wKE8dsv19N95940SXn1/nzaRx2/9BdU1NUhVnHjJf7Jll/LqfRRKKd8MrIgoTMXSPcBIkkHulgA/iIhbm/pMtwG7xCGX/64g8VhhHLVrz+YLWcm49uzjmPfma3lNSbvsvnfc8cj4rMruN7jblCZGBi6IgrXkIuKUQtVtZqVDlPbLpd1dNbPceDw5M6t0JZzjnOTMLA9KOMs5yZlZjkr72VUnOTPLSZsdhcTM2hAnOTOrZO6umllF8y0kZlbRSjjHOcmZWY6KOMJINpzkzCwnydXV0s1yTnJmlrPSTXFOcmaWDyWc5ZzkzCxnpXwLiUcGNrOc5ettXZJGSZopaZakyxrYfp6k1yRNlTRB0rDm6nSSM7Oc5eMVD5KqgRuBo4FhwCkNJLG7I2L3iNgL+ClwXXOxOcmZWU7qBs3MZmrGfsCsiHg7IlYDY4HjMwtExPsZi1sCzQ5t7nNyZpab/A2a2QeYn7G8ANh/k91JFwDfBtoDm74ctx635MwsZy3orvaQNDljOqel+4qIGyNiMPA94IrmyrslZ2a5y74lt7yJF9ksBPplLPdN1zVmLPA/ze3QLTkzy1G2b11tNhNOAoZIGiipPXAyMG6jPUlDMhaPAd5qrlK35MwsJ/kaNDMi1kq6EHgSqAZui4jpkkYDkyNiHHChpMOBNcBK4PTm6nWSM7Pc5ele4Ih4HHi83rorM+a/1dI6neTMLGel/MSDk5yZ5ayEByFxkjOz3JVwjnOSM7Mc5e9m4IJwkjOznNQ91lWqnOTMLGelm+Kc5MwsD0q4IeckZ2a58y0kZlbZSjfHOcmZWe5KOMc5yZlZbiS/ktDMKl3p5jgnOTPLXQnnOCc5M8tdCfdWneTMLFdZDYhZNE5yZpaT5LGuYkfROCc5M8uZk5yZVTR3V82scnmoJTOrZBnvVC1JTnJmlrsSznJOcmaWMz/WZWYVrXRTnJOcmeVDCWc5Jzkzy1kp30KiiCh2DOtJWgbMLXYcBdADWF7sIKxFKvVnNiAieuazQklPkHxf2VgeEaPyuf/mlFSSq1SSJkfEiGLHYdnzz6xyVBU7ADOzQnKSM7OK5iTXOsYUOwBrMf/MKoTPyZlZRXNLzswqmpOcmVU0Jzkzq2hOcgUkaSdJB0pqJ6m62PFYdvyzqiy+8FAgkr4I/BhYmE6Tgdsj4v2iBmaNkjQ0Iv6WzldHxLpix2S5c0uuACS1A04CzoqIzwKPAP2A70nqUtTgrEGSjgWmSrobICLWuUVXGZzkCqcLMCSdfwh4FGgHnCqV8OBbbZCkLYELgX8DVku6E5zoKoWTXAFExBrgOuCLkg6NiFpgAjAVOKSowdkmIuJD4EzgbuASoGNmoitmbJY7J7nCeR54CviqpMMiYl1E3A30BvYsbmhWX0QsiogPImI5cC6wRV2ikzRc0s7FjdA2l8eTK5CI+ETSXUAA30//SP4JbAssLmpw1qSIWCHpXOBaSW8C1cCnixyWbSYnuQKKiJWSbgZmkLQOPgG+EhFLihuZNScilkt6FTgaOCIiFhQ7Jts8voWklaQnsCM9P2clTlJ34D7gOxHxarHjsc3nJGfWCEkdI+KTYsdhuXGSM7OK5qurZlbRnOTMrKI5yZlZRXOSM7OK5iRXRiStkzRV0uuS7pfUKYe6bpf05XT+FknDmig7UtJBm7GPOZI2eR9nY+vrlfmghfu6StIlLY3RKp+TXHn5OCL2iojdgNXAeZkbJW3Wzd0RcXZEzGiiyEigxUnOrBQ4yZWv54Ed01bW85LGATMkVUu6VtIkSa+mjyehxA2SZkr6M9CrriJJ4yWNSOdHSXpZ0jRJT0vagSSZXpy2Ig+V1FPSH9J9TJJ0cPrZbSQ9JWm6pFuAZkdbkfSwpCnpZ86pt+36dP3Tknqm6wZLeiL9zPN+ptSa48e6ylDaYjsaeCJdNRzYLSJmp4nivYjYV1IH4AVJTwF7AzsBw0ien50B3Fav3p7AzcBhaV1bR8S7kn4NfBARP0vL3Q1cHxYBlPEAAAHlSURBVBETJPUHngR2AX4ATIiI0ZKOAc7K4nDOTPexBTBJ0h8iYgWwJTA5Ii6WdGVa94Ukrwo8LyLekrQ/cBPwmc34Gq2NcJIrL1tImprOPw/cStKNnBgRs9P1RwJ71J1vA7qSjGt3GHBPOnTQIknPNFD/AcBzdXVFxLuNxHE4MCxjWLwukrZK9/HF9LOPSVqZxTFdJOkL6Xy/NNYVQC1wb7r+TuDBdB8HAfdn7LtDFvuwNsxJrrx8HBF7Za5I/9g/zFwFfDMinqxX7nN5jKMKOKD+I08tHQtU0kiShHlgRHwkaTzQsZHike53Vf3vwKwpPidXeZ4EvpEOwY6koenIt88BJ6Xn7Lan4aGDXgQOkzQw/ezW6fp/AJ0zyj0FfLNuQVJd0nkOODVddzTQvZlYuwIr0wS3M0lLsk4VUNcaPZWkG/w+MFvSCek+JMlj81mTnOQqzy0k59telvQ68BuSFvtDwFvptjuAv9b/YEQsA84h6RpOY0N38Y/AF+ouPAAXASPSCxsz2HCV94ckSXI6Sbd1XjOxPgHUSHoD+AlJkq3zIbBfegyfAUan608Dzkrjmw4cn8V3Ym2YH9A3s4rmlpyZVTQnOTOraE5yZlbRnOTMrKI5yZlZRXOSM7OK5iRnZhXt/wEjNYUZ7+KW6QAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 2 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LJDmxgHxE2wm",
        "outputId": "d72cf86c-fbba-42d2-f19c-19e49e4a836b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 52
        }
      },
      "source": [
        "# predicting the test set\n",
        "#we can't assess how well these have done in this notebook for that we\n",
        "#will have to submit them to the codalab competition system (see below)\n",
        "knn_preds_test = knn.predict(X_test)\n",
        "\n",
        "print(\"Predicted values for the test set:\")\n",
        "print(knn_preds_test)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Predicted values for the test set:\n",
            "['negative' 'positive' 'positive' ... 'negative' 'negative' 'negative']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eIMYAcvw4dv0",
        "outputId": "f766c8f2-f058-4860-a622-d4dc6bd1a8f4",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 16
        }
      },
      "source": [
        "#Convert the 'knn_preds_test' array of 'negative' and\n",
        "#'positive's into an answer.zip file. This will also\n",
        "#try to allow you to download the file. \n",
        "create_zip(knn_preds_test)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "application/javascript": [
              "\n",
              "    async function download(id, filename, size) {\n",
              "      if (!google.colab.kernel.accessAllowed) {\n",
              "        return;\n",
              "      }\n",
              "      const div = document.createElement('div');\n",
              "      const label = document.createElement('label');\n",
              "      label.textContent = `Downloading \"${filename}\": `;\n",
              "      div.appendChild(label);\n",
              "      const progress = document.createElement('progress');\n",
              "      progress.max = size;\n",
              "      div.appendChild(progress);\n",
              "      document.body.appendChild(div);\n",
              "\n",
              "      const buffers = [];\n",
              "      let downloaded = 0;\n",
              "\n",
              "      const channel = await google.colab.kernel.comms.open(id);\n",
              "      // Send a message to notify the kernel that we're ready.\n",
              "      channel.send({})\n",
              "\n",
              "      for await (const message of channel.messages) {\n",
              "        // Send a message to notify the kernel that we're ready.\n",
              "        channel.send({})\n",
              "        if (message.buffers) {\n",
              "          for (const buffer of message.buffers) {\n",
              "            buffers.push(buffer);\n",
              "            downloaded += buffer.byteLength;\n",
              "            progress.value = downloaded;\n",
              "          }\n",
              "        }\n",
              "      }\n",
              "      const blob = new Blob(buffers, {type: 'application/binary'});\n",
              "      const a = document.createElement('a');\n",
              "      a.href = window.URL.createObjectURL(blob);\n",
              "      a.download = filename;\n",
              "      div.appendChild(a);\n",
              "      a.click();\n",
              "      div.remove();\n",
              "    }\n",
              "  "
            ],
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "application/javascript": [
              "download(\"download_24c242df-ecb4-4a08-825e-7dc6c275e279\", \"answer.zip\", 90117)"
            ],
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mKJVuYTxlOTZ"
      },
      "source": [
        "# Logistic Regression Classifier\n",
        "\n",
        "Below is a classifier using a logistic reression algorithm. This is much better than the above solutions with an accuracy of 91% when ngram_Range of 1,2 is used."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DYLI0y0slSfH",
        "outputId": "acb0bc3c-656b-49d9-fb63-98f806499749",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 756
        }
      },
      "source": [
        "\n",
        "############################## preparing data\n",
        "\n",
        "# apply pre_proc() function to all data splits\n",
        "proc_train = pre_proc(df_train['review'])\n",
        "proc_val = pre_proc(df_val['review'])\n",
        "proc_test = pre_proc(df_test['review'])\n",
        "\n",
        "#extract features\n",
        "X_train, X_val, X_test = extract_features(proc_train, proc_val, proc_test, binary=False, max_df=10.0, min_df=0.0, ngram_range=(1,2), sw=False)\n",
        "\n",
        "# extract pre-known sentiment labels (used for training and evaluation)\n",
        "y_train = df_train['sentiment']\n",
        "y_val = df_val['sentiment']\n",
        "\n",
        "############################ finds best parameter c for the logistic regression (based on evaluation set)\n",
        "\n",
        "def optimise_C(X_train, y_train, X_val, y_val, C=[0.01, 0.025, 0.05, 0.25, 0.5, 1.0]):\n",
        "    #optimise the parameter C using the validation data\n",
        "    best_acc = 0.\n",
        "    best_c = 0.\n",
        "    accuracies = []\n",
        "    for c in C:\n",
        "    \n",
        "        lr = LogisticRegression(C=c, multi_class='auto', solver='liblinear')\n",
        "        lr.fit(X_train, y_train)\n",
        "        cur_acc = accuracy_score(y_val, lr.predict(X_val))\n",
        "        print (\"Accuracy for C=%s: %s\" % (c, cur_acc))\n",
        "        accuracies.append(cur_acc)\n",
        "        if cur_acc > best_acc:\n",
        "            best_c = c\n",
        "            best_acc = cur_acc\n",
        "\n",
        "    print (\"*** Best accuracy = %f, best C = %f\" % (best_acc, best_c))\n",
        "    plt.plot(np.array(C).astype('str'), accuracies, 'ro')\n",
        "    return best_c\n",
        "\n",
        "#apply the above function\n",
        "ngram11_C=[0.0745, 0.075, 0.07525, 0.0754, 0.0755, 0.08, 0.09, 0.10, 0.25] #best c values for ngram_range 1,1\n",
        "ngram12_C=[0.07,0.08,0.09, 0.10, 0.11,0.17] #best c values for ngram_range 1,2\n",
        "ngram13_C=[0.3,0.325,0.35,0.375,0.4] #best c values for ngram_range 1,3\n",
        "best_c = optimise_C(X_train, y_train, X_val, y_val, C=[0.07,0.08,0.09, 0.10, 0.11,0.17])\n",
        "\n",
        "################################# validation set evaluation\n",
        "\n",
        "lr = LogisticRegression(C=best_c, multi_class='auto', solver='liblinear')\n",
        "lr.fit(X_train, y_train) #fits model according to training data\n",
        "lr_preds = lr.predict(X_val) #predict class labels \n",
        "evaluate_classifier(y_val, lr_preds)\n",
        "\n",
        "################################# test set predictions\n",
        "\n",
        "lrt = LogisticRegression(C=best_c,multi_class='auto', solver='liblinear')\n",
        "lrt.fit(X_train,y_train)\n",
        "lrt_preds=lrt.predict(X_test)\n",
        "\n",
        "################################create zip\n",
        "#create_zip(lrt_preds)\n"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "** Vocabulary size: 1706851\n",
            "** Words:\n",
            "Accuracy for C=0.07: 0.9011\n",
            "Accuracy for C=0.08: 0.9015\n",
            "Accuracy for C=0.09: 0.9016\n",
            "Accuracy for C=0.1: 0.9016\n",
            "Accuracy for C=0.11: 0.9015\n",
            "Accuracy for C=0.17: 0.9015\n",
            "*** Best accuracy = 0.901600, best C = 0.090000\n",
            "*** Accuracy score: 0.9016\n",
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "application/javascript": [
              "\n",
              "    async function download(id, filename, size) {\n",
              "      if (!google.colab.kernel.accessAllowed) {\n",
              "        return;\n",
              "      }\n",
              "      const div = document.createElement('div');\n",
              "      const label = document.createElement('label');\n",
              "      label.textContent = `Downloading \"${filename}\": `;\n",
              "      div.appendChild(label);\n",
              "      const progress = document.createElement('progress');\n",
              "      progress.max = size;\n",
              "      div.appendChild(progress);\n",
              "      document.body.appendChild(div);\n",
              "\n",
              "      const buffers = [];\n",
              "      let downloaded = 0;\n",
              "\n",
              "      const channel = await google.colab.kernel.comms.open(id);\n",
              "      // Send a message to notify the kernel that we're ready.\n",
              "      channel.send({})\n",
              "\n",
              "      for await (const message of channel.messages) {\n",
              "        // Send a message to notify the kernel that we're ready.\n",
              "        channel.send({})\n",
              "        if (message.buffers) {\n",
              "          for (const buffer of message.buffers) {\n",
              "            buffers.push(buffer);\n",
              "            downloaded += buffer.byteLength;\n",
              "            progress.value = downloaded;\n",
              "          }\n",
              "        }\n",
              "      }\n",
              "      const blob = new Blob(buffers, {type: 'application/binary'});\n",
              "      const a = document.createElement('a');\n",
              "      a.href = window.URL.createObjectURL(blob);\n",
              "      a.download = filename;\n",
              "      div.appendChild(a);\n",
              "      a.click();\n",
              "      div.remove();\n",
              "    }\n",
              "  "
            ],
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "application/javascript": [
              "download(\"download_36fdbbee-4122-4ebf-8e03-a8cf56acc857\", \"answer.zip\", 90117)"
            ],
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYcAAAD4CAYAAAAHHSreAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAATo0lEQVR4nO3db4xddX7f8ffHULedXdQY7Kx2MXjYiki1GrSUG7OPYrppjXejQvDSFJZKXqWt1XZRq6ZUhbgSyKsRKqFRHhSpcVeslqy3zu5KrUyqBDn8UdJUTj0OmJQlBuOy2AZpJzEkJW6WBb59cA/k+v7Gnjv/dzzvl3R0z/me3zn397VH9zPnnrkzqSokSRq0ZrknIEn64WM4SJIahoMkqWE4SJIahoMkqXHpck9gIaxfv77Gx8eXexqStKIcOXLkj6pqw3T7LopwGB8fZ3JycrmnIUkrSpLvnm+fbytJkhqGgySpYThIkhqGgySpYThIkhojhUOS7UmOJTme5N5p9m9K8mSS55M8k2TjwL6dSV7ulp0D9YkkJ5O8Pc35fjbJd5K8kOQbc21OYt8+GB+HNWv6j/v2LfeMFt9q7FkLr6ouuACXAK8AnwTWAkeBzUNjvgXs7NY/A/xqt345cKJ7XNetr+v2fRr4OPD20LmuBZ4dGPejM83xhhtuKKnx9a9XjY1VwV8sY2P9+sVqNfasOQMm6zyvq6NcOWwBjlfViap6B9gP3Do0ZjPwVLf+9MD+m4GDVXWmqt4EDgLbu1A6VFVvTPN8/wR4pBtPVX1vhDlKrd274ezZc2tnz/brF6vV2LMWxSjhcCVwcmD7VFcbdBTY0a3fBlyW5IoRjx32Y8CPJfndJIeSbJ9uUJJdSSaTTE5NTY3Qhlad116bXf1isBp71qJYqBvS9wBbkzwLbAVOA+/N8VyX0n9r6SbgTuA/J/mR4UFVtbeqelXV27Bh2k9/a7W7+urZ1S8Gq7FnLYpRwuE0cNXA9sau9qGqer2qdlTV9cDurvbWKMdO4xRwoKp+UFX/B3iJflhIszMxAWNj59bGxvr1i9Vq7FmLYpRwOAxcm+SaJGuBO4ADgwOSrE/ywbnuAx7t1p8AtiVZl2QdsK2rXch/o3/VQJL19N9mOjHCPKVz3XUX7N0LmzZB0n/cu7dfv1itxp61KFIj/A3pJJ8Dfpn+Ty49WlUTSfbQv9N9IMntwINAAb8NfKmqvt8d+3PAL3Snmqiqr3b1h4AvAJ8AXge+UlUPJAnwH+jfuH6vO2b/hebX6/XKX7wnSbOT5EhV9abdN0o4/LAzHCRp9i4UDn5CWpLUMBwkSQ3DQZLUMBwkSQ3DQZLUMBwkSQ3DQZLUMBwkSQ3DQZLUMBwkSQ3DQZLUMBwkSQ3DQZLUMBwkSQ3DQZLUMBwkSQ3DQZLUMBwkSQ3DQZLUMBwkSQ3DQZLUMBwkSQ3DQZLUMBwkSQ3DQZLUMBwkSQ3DQZLUMBwkSQ3DQZLUMBwkSQ3DQZLUGCkckmxPcizJ8ST3TrN/U5Inkzyf5JkkGwf27UzycrfsHKhPJDmZ5O2hc30xyVSS57rlH8+nQQ3Ytw/Gx2HNmv7jvn3LPSNpYazGr+3F7rmqLrgAlwCvAJ8E1gJHgc1DY74F7OzWPwP8ard+OXCie1zXra/r9n0a+Djw9tC5vgj8x5nmNbjccMMNpRl8/etVY2NV8BfL2Fi/Lq1kq/Fre4F6BibrPK+ro1w5bAGOV9WJqnoH2A/cOjRmM/BUt/70wP6bgYNVdaaq3gQOAtu7UDpUVW+M8PxaCLt3w9mz59bOnu3XpZVsNX5tL0HPo4TDlcDJge1TXW3QUWBHt34bcFmSK0Y8djqf796i+naSq6YbkGRXkskkk1NTUyOccpV77bXZ1aWVYjV+bS9Bzwt1Q/oeYGuSZ4GtwGngvTme63FgvKquo3+l8bXpBlXV3qrqVVVvw4YNc3yqVeTqq2dXl1aK1fi1vQQ9jxIOp4HB7943drUPVdXrVbWjqq4Hdne1t0Y5dlhV/XFVfb/b/Apwwwhz1EwmJmBs7Nza2Fi/Lq1kq/Frewl6HiUcDgPXJrkmyVrgDuDA4IAk65N8cK77gEe79SeAbUnWJVkHbOtq55Xk4wObtwAvjjBHzeSuu2DvXti0CZL+4969/bq0kq3Gr+0l6Dn9G9YzDEo+B/wy/Z9cerSqJpLsoX+n+0CS24EHgQJ+G/jSB9/9J/k54Be6U01U1Ve7+kPAF4BPAK8DX6mqB5I8SD8U3gXOAP+sqv7wQvPr9Xo1OTk5y9YlaXVLcqSqetPuGyUcftgZDpI0excKBz8hLUlqGA6SpIbhIElqGA6SpIbhIElqGA6SpIbhIElqGA6SpIbhIElqGA6SpIbhIElqGA6SpIbhIElqGA6SpIbhIElqGA6SpIbhIElqGA6SpIbhIElqGA6SpIbhIElqGA6SpIbhIElqGA6SpIbhIElqGA6SpIbhIElqGA6SpIbhIElqGA6SpMZI4ZBke5JjSY4nuXea/ZuSPJnk+STPJNk4sG9nkpe7ZedAfSLJySRvn+c5P5+kkvTm0pgkae5mDIcklwCPAJ8FNgN3Jtk8NOxh4LGqug7YAzzYHXs5cD9wI7AFuD/Juu6Yx7vadM95GfAvgd+bbUOSpPkb5cphC3C8qk5U1TvAfuDWoTGbgae69acH9t8MHKyqM1X1JnAQ2A5QVYeq6o3zPOeXgX8P/PnInUiSFswo4XAlcHJg+1RXG3QU2NGt3wZcluSKEY89R5K/BVxVVf99hnG7kkwmmZyampq5C0nSyBbqhvQ9wNYkzwJbgdPAe7M9SZI1wC8B/3qmsVW1t6p6VdXbsGHDbJ9KknQBo4TDaeCqge2NXe1DVfV6Ve2oquuB3V3trVGOHXIZ8DeBZ5K8CnwaOOBNaUlaWqOEw2Hg2iTXJFkL3AEcGByQZH33XT/AfcCj3foTwLYk67ob0du62rSq6k+qan1VjVfVOHAIuKWqJmfVlSRpXmYMh6p6F7ib/ov6i8A3q+qFJHuS3NINuwk4luQl4GPARHfsGfo3lw93y56uRpKHkpwCxpKcSvLAgnYmSZqzVNVyz2Heer1eTU56cSFJs5HkSFVN+7a9n5CWJDUMB0lSw3CQJDUMB0lSw3CQJDUMB0lSw3CQJDUMB0lSw3CQJDUMB0lSw3CQJDUMB0lSw3CQJDUMB0lSw3CQJDUMB0lSw3CQJDUMB0lSw3CQJDUMB0lSw3CQJDUMB0lSw3CQJDUMB0lSw3CQJDUMB0lSw3CQJDUMB0lSw3CQJDUMB0lSw3CQJDVGCock25McS3I8yb3T7N+U5Mkkzyd5JsnGgX07k7zcLTsH6hNJTiZ5e+hc/zTJHyR5Lsn/SLJ5Pg1KkmZvxnBIcgnwCPBZYDNw5zQv2A8Dj1XVdcAe4MHu2MuB+4EbgS3A/UnWdcc83tWGfaOqfryqPgU8BPzSrLuSJM3LKFcOW4DjVXWiqt4B9gO3Do3ZDDzVrT89sP9m4GBVnamqN4GDwHaAqjpUVW8MP1lV/enA5keAGrUZSdLCGCUcrgRODmyf6mqDjgI7uvXbgMuSXDHisY0kX0ryCv0rh39xnjG7kkwmmZyamhqhDUnSqBbqhvQ9wNYkzwJbgdPAe3M9WVU9UlV/Hfi3wL87z5i9VdWrqt6GDRvm+lSSpGmMEg6ngasGtjd2tQ9V1etVtaOqrgd2d7W3Rjl2BvuBn5nFeEnSAhglHA4D1ya5Jsla4A7gwOCAJOuTfHCu+4BHu/UngG1J1nU3ord1tfNKcu3A5k8DL48wR0nSApoxHKrqXeBu+i/qLwLfrKoXkuxJcks37CbgWJKXgI8BE92xZ4Av0w+Yw8CerkaSh5KcAsaSnEryQHeuu5O8kOQ54OeBD3/8VZK0NFK18n8YqNfr1eTk5HJPQ5JWlCRHqqo33T4/IS1JahgOkqSG4SBJahgOkqSG4SBJahgOkqSG4SBJahgOkqSG4SBJahgOkqSG4SBJahgOkqSG4SBJahgOkqSG4SBJahgOkqSG4SBJahgOkqSG4SBJahgOkqSG4SBJahgOkqSG4SBJahgOkqSG4SBJahgOkqSG4SBJahgOkqSG4SBJahgOkqTGSOGQZHuSY0mOJ7l3mv2bkjyZ5PkkzyTZOLBvZ5KXu2XnQH0iyckkbw+d6+eTfKc715NJNs2nQUnS7M0YDkkuAR4BPgtsBu5Msnlo2MPAY1V1HbAHeLA79nLgfuBGYAtwf5J13TGPd7VhzwK97lzfBh6abVOSpPkZ5cphC3C8qk5U1TvAfuDWoTGbgae69acH9t8MHKyqM1X1JnAQ2A5QVYeq6o3hJ6uqp6vqbLd5CNg4PEaStLhGCYcrgZMD26e62qCjwI5u/TbgsiRXjHjshfwj4Dem25FkV5LJJJNTU1OzOKUkaSYLdUP6HmBrkmeBrcBp4L35nDDJPwR6wC9Ot7+q9lZVr6p6GzZsmM9TSZKGXDrCmNPAVQPbG7vah6rqdborhyQfBT5fVW8lOQ3cNHTsMzM9YZK/A+wGtlbV90eYoyRpAY1y5XAYuDbJNUnWAncABwYHJFmf5INz3Qc82q0/AWxLsq67Eb2tq51XkuuBXwFuqarvjd6KJGmhzBgOVfUucDf9F/UXgW9W1QtJ9iS5pRt2E3AsyUvAx4CJ7tgzwJfpB8xhYE9XI8lDSU4BY0lOJXmgO9cvAh8FvpXkuSTnBJEkafGlqpZ7DvPW6/VqcnJyuachSStKkiNV1Ztun5+QliQ1DAdJUsNwkCQ1DAdJUsNwkCQ1DAdJUsNwkCQ1DAdJUsNwkCQ1DAdJUsNwkCQ1DAdJUsNwkCQ1DAdJUsNwkCQ1DAdJUsNwkCQ1DAdJUsNwkCQ1DAdJUsNwkCQ1DAdJUsNwkCQ1DAdJUsNwkCQ1DAdJUsNwkCQ1DAdJUsNwkCQ1DAdJUsNwkCQ1RgqHJNuTHEtyPMm90+zflOTJJM8neSbJxoF9O5O83C07B+oTSU4meXvoXD+Z5PeTvJvk9vk0d0H79sH4OKxZ03/ct2/RnkqSVpoZwyHJJcAjwGeBzcCdSTYPDXsYeKyqrgP2AA92x14O3A/cCGwB7k+yrjvm8a427DXgi8A3ZtvMyPbtg1274Lvfhar+465dBoQkdUa5ctgCHK+qE1X1DrAfuHVozGbgqW796YH9NwMHq+pMVb0JHAS2A1TVoap6Y/jJqurVqnoeeH/W3Yxq9244e/bc2tmz/bokaaRwuBI4ObB9qqsNOgrs6NZvAy5LcsWIx85Jkl1JJpNMTk1Nze7g116bXV2SVpmFuiF9D7A1ybPAVuA08N4CnXtaVbW3qnpV1duwYcPsDr766tnVJWmVGSUcTgNXDWxv7GofqqrXq2pHVV0P7O5qb41y7LKYmICxsXNrY2P9uiRppHA4DFyb5Joka4E7gAODA5KsT/LBue4DHu3WnwC2JVnX3Yje1tWW1113wd69sGkTJP3HvXv7dUnSzOFQVe8Cd9N/UX8R+GZVvZBkT5JbumE3AceSvAR8DJjojj0DfJl+wBwG9nQ1kjyU5BQwluRUkge6+k909b8P/EqSFxas20F33QWvvgrvv99/NBgk6UOpquWew7z1er2anJxc7mlI0oqS5EhV9abb5yekJUkNw0GS1DAcJEkNw0GS1LgobkgnmQK+O8fD1wN/tIDTWQnseXWw59VhPj1vqqppP0V8UYTDfCSZPN/d+ouVPa8O9rw6LFbPvq0kSWoYDpKkhuEAe5d7AsvAnlcHe14dFqXnVX/PQZLU8spBktQwHCRJjYs6HJJsT3IsyfEk906z/y8n+bVu/+8lGe/qdyV5bmB5P8mnlnr+czGPnv9Skq8l+YMkLya5b6nnPlfz6Hltkq92PR9NctMST31BjND/Tyb5/STvJrl9OeY4X/PpMclvJnkrya8v3Yznb649J/nbQ69ff57kZ2Y9gaq6KBfgEuAV4JPAWvp/ynTz0Jh/Dvynbv0O4NemOc+PA68sdz+L3TPwBWB/tz4GvAqML3dPi9zzl4Cvdus/ChwB1ix3T4vQ/zhwHfAYcPtyz3mpewR+Cvh7wK8vdy9L/f8KXA6cAcZmO4eL+cphC3C8qk5U1TvAfuDWoTG3Al/r1r8N/FSSDI25szt2JZhPzwV8JMmlwF8F3gH+dGmmPS/z6Xkz8BRAVX0PeAtYaR+gmrH/qnq1qp4H3l+OCS6AefVYVU8C/3dJZrpwFur/9XbgN6rq7GwncDGHw5XAyYHtU11t2jHV/6NGfwJcMTTmHwD/ZZHmuNDm0/O3gT8D3gBeAx6u7g8z/ZCbT89HgVuSXJrkGuAGzv2ztivBKP2vdKuhx2EL1fMdzPH169K5HLRaJLkROFtV/3u557IEtgDvAZ8A1gG/k+S3qurE8k5rUT0K/A1gkv7v5vqf9P8NpBUvycfpvy0+pz/NfDFfOZzm3O8CN3a1acd0b6f8NeCPB/bPOXWXyXx6/gLwm1X1g+4tlt9lZbzFMueeq+rdqvpXVfWpqroV+BHgpSWY80Iapf+VbjX0OGwhev5Z4L9W1Q/mMoGLORwOA9cmuSbJWvov9AeGxhwAdnbrtwNPVXcXJ8ka+v+4K+V+A8yv59eAzwAk+QjwaeAPl2TW8zPnnpOMdb2S5O8C71bVd5Zq4gtklP5XutXQ47CF6PlO5vPN7XLflV/kO/6fo/+d4CvA7q62B7ilW/8rwLeA48D/Aj45cOxNwKHl7mGpegY+2tVfAL4D/Jvl7mUJeh4HjgEvAr9F/9cXL3s/i9D/T9B/z/rP6F8lvrDcc17KHoHfAaaA/9eNuXm5+1mCnsfpX2nM+afv/PUZkqTGxfy2kiRpjgwHSVLDcJAkNQwHSVLDcJAkNQwHSVLDcJAkNf4/3vFeCeTMSxQAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAATkAAAEYCAYAAAAnEYFiAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deZxWdd3/8dd7BnFnE7EYwFBRxDVETEsjV0yFu8xErdzKvAsty8rKzGix8u7W7rI7l7xdUnFLxSVxKVz6qTDiCoqSGwymgqilJjJ+fn+cM3jNMHNdZ7iua65l3k8f5+F1zvme7/mca2Y+fL9n+yoiMDOrVw2VDsDMrJyc5MysrjnJmVldc5Izs7rmJGdmdc1JzszqmpNclZA0S9IX089HSLqtxPV/SFJI6lPKegvsU5L+T9JySbOLqGd3SQtKGVulSBoh6V+SGisdS2/Ra5KcpOckvSxp/ZxlX5Q0q4JhdSoiLouIfSsdRwl8DNgHGBYR49e0koi4JyK2Kl1Y5ZH+ju2dr0xEvBARG0REa0/F1dv1miSXagS+VmwlaQult313a2JT4LmIeLPSgVSDnmxF2/t62x/qmcDJkgZ0tlLSbpLmSHo9/f9uOetmSfqppL8BbwGbpd2/r0h6WtI/Jf1Y0uaS/p+kNyRdJalvuv1ASTdJeiXtvt0kaVgXcRwl6d7087fT7k3b9K6ki9J1/SX9QdKLklok/aStGySpUdJ/SVoq6RnggHxfjKThkv6UxrdM0m/T5Q2STpX0fNoSvkRS/3RdWxf4SEkvpPv6frruWOACYNc07h/lHlfOfkPSFunnT0qan36XLZJOTpdPkLQ4Z5ut05/Ha5LmSZqUs+4iSedIujmt5wFJm3dxzG3xHy1pUfpzOV7SzpIeTev/bU75zSX9Jf1+lkq6rO13SdKlwAjgxvR4v51T/7GSXgD+krOsj6RBkhZLOiitYwNJCyV9Id/PyropInrFBDwH7A38CfhJuuyLwKz08yBgOfB5oA9wWDq/Ubp+FvACsE26fi0ggBuAfunyd4A7gc2A/sB84Mh0+42Ag4H1gA2Bq4Hrc+KbBXwx/XwUcG8nxzAcWALsn85fB5wLrA8MAWYDX07XHQ88mW4zCPhrGm+fTuptBB4BzkrrWgf4WLruGGBhekwbpN/fpem6D6V1ng+sC+yQfgdbd3YcnR1Xuv0W6ecXgd3TzwOBsennCcDi9PNaaTzfA/oCewL/BLZK118ELAPGpz+ny4DpXfxOtMX/+/SY9wX+DVyffp9NwMvAx9PyW5B0v9cGNgbuBs7u+DvWSf2XpN/rujnL+qRl9gX+ke7vfOCaSv+t1NtU8QB67EDfT3LbAq+nv6S5Se7zwOwO29wHHJV+ngVM67A+gI/mzD8IfCdn/le5fwQdtt0RWJ4zP4s8SS79A1lVP7BJmlDWzSlzGPDX9PNfgONz1u1L10luV+CVLtbdCXwlZ34r4N00gbT9wQ7LWT8bmNLZcXRxXLlJ7gXgy0C/DmUm8H6S2z1NCg05668ATk8/XwRckLPuk8CTXfwM2uJvylm2DDg0Z/5a4OtdbP8fwEMdf8c6qX+zTpb1yVn2G+AxoIX0H1VPpZt6W3eViHgcuAk4pcOqocDzHZY9T/KveZtFnVT5Us7ntzuZ3wBA0nqSzk27fW+QtAIGKPtVtj8ACyLiF+n8piStmhfTbtVrJK26ITnHkxtvx2PLNRx4PiJWdrKu4/fyPEmC2yRn2T9yPr9Fesxr4GCSpPS8pLsk7dpFPIsi4r0OMeX+nLobT9af4SaSpqdd6TeAPwKDC9QNnf/e5DqP5B/fiyJiWYb6rBt6XZJL/RD4Eu3/MJaQJI5cI0j+dW1TzCtbvknSCtolIvoBe6TLVWhDSacAWwLH5ixeRNKSGxwRA9KpX0Rsk65/kSR5tRmRZxeLgBHq/MR4x+9lBLCS9okgqzdJuusASPpA7sqImBMRk0kS9fXAVV3EM1ztL/x0/DmVy89Ifge2S3+Gn6P9z6+r348uf2/Sf+TOI+nSfqXt/KSVTq9MchGxELgSODFn8S3AlpIOT08KHwqMIWn1lcKGJK2C1yQNIkm0BUnaP43zUxHxds4xvAjcBvxKUr/0AsHmkj6eFrkKOFHSMEkDWb3lmms2SVL8uaT1Ja0j6aPpuiuAkySNlLQByR/6lV20+gp5BNhG0o6S1gFOzznOvkruD+wfEe8CbwDvdVLHAySts29LWkvSBOAgYPoaxNNdGwL/Al6X1AR8q8P6l0jOXXbH90iS4DEkF8Yu6Ubr3jLolUkuNY3kZDAAaTfhQJIW1zLg28CBEbG0RPs7m+S82lLgfuDWjNsdSnL+8Am9f4X19+m6L5CcfJ9PcpHkGuCD6brzgZkkiWUuyQWDTkVyz9ZBJCfWXwAWp/sFuBC4lKR7/SzJifkTMsbecT9PkXzvdwBPA/d2KPJ54Lm0K3g8cEQndaxIY92f5Lv8HfCFiHhyTWLqph8BY0nO6d7M6t/pGcCp6emDkwtVJmkn4Bsk8bcCvyBJePn+QbJuUnri08ysLvXmlpyZ9QJOcmZW15zkzKyuOcmZWdWQNFHSgvTxttUuwEjaVNKd6WN3s9TFo5HttqmmCw/qs25o7X6VDsO6YcfRwwsXsqrxwvPPsXTp0oL3ZnZHY79NI1a+XbggEG+/MjMiJna2Lr115imSR+cWA3OAwyJifk6Zq4GbIuJiSXsCR0fE5/Pts6reiqC1+7H26CmVDsO64d77zq50CNYNH9t155LXGSvfZu2tPpup7L8fPiffEyLjgYUR8QyApOnAZJJbpNqMIbntBpLnsa8vtE93V82sSAI1ZJtgsKTmnOm4nIqaaP8I3GLaP5UEyX2fn04/fwrYUNJG+aKrqpacmdUgAQ2ZH9JYGhHjitjbycBvJR1FcoN6C5D3BaROcmZWPJXkNF8L7Z+3HkaHZ5IjYglpSy59zPDgiHgtX6XurppZkbrVXc1nDjAqfU66LzAFmNFuT9LgnJczfJfkscO8nOTMrHhStimP9KUPU0meuX4CuCoi5kmalvP25wnAAklPkbzu66eFQnN31cyKI7K00jKJiFtI3giUu+y0nM/XkLyIIjMnOTMrUuFWWiU5yZlZ8bJfXe1xTnJmViSVrLtaDk5yZlYc4e6qmdU5t+TMrH65u2pm9a7B3VUzq1fde3a1xznJmVmR3F01s3rnq6tmVtfckjOzupXh4ftKcpIzs+L5woOZ1S9feDCzeufuqpnVrRK+T64cnOTMrEjurppZvavi7mr1pl8zqx0NjdmmAiRNlLRA0kJJp3SyfoSkv0p6SNKjkj5ZMLQ1PCQzs4RKM1qXpEbgHGB/YAxwmKQxHYqdSjLAzYdJRvP6XaHwnOTMrHglGK0LGA8sjIhnImIFMB2Y3KFMAP3Sz/2BJYUq9Tk5Myuasp+TGyypOWf+vIg4L/3cBCzKWbcY2KXD9qcDt0k6AVgf2LvQDp3kzKwoydvPMye5pRExrojdHQZcFBG/krQrcKmkbSPiva42cHfVzIqjbkz5tQDDc+aHpctyHQtcBRAR9wHrAIPzVeokZ2ZFEg0NDZmmAuYAoySNlNSX5MLCjA5lXgD2ApC0NUmSeyVfpe6umlnRutFd7VJErJQ0FZgJNAIXRsQ8SdOA5oiYAXwTOF/SSSQXIY6KiMhXr5OcmRWtFEkOICJuAW7psOy0nM/zgY92p04nOTMrTrbzbRXjJGdmRREqWUuuHJzkzKxoGS4qVIyTnJkVzS05M6tfPidnZvXOLTkzq1u+8GBmdc9Jzszql0ANTnJmVsfckjOzuuYkZ2Z1yxcezKz+VW+Oc5IzsyLJ3VUzq3N+dtXM6lv1NuT8+vNi7bPraB659ns8fv2pnHzU6gMHjfjAQG75368ye/p3mHnuVJqG9F+17ogDd+ax607lsetO5YgDd+7JsHu122beyo7bjma7rUfxX2f+fLX1995zN7vtshP91luL6/50Tbt1f7z0YrYfsyXbj9mSP156cU+FXPUkZZoqoaxJrtBo2LWuoUGcfcohTD7xXD78mTM4ZL+xjB65SbsyZ5w0mctuns34Kb/gZxfMZNrUgwAY2G89vv+liexx5H+z+xd+xfe/NJEBG65bicPoVVpbW/nG16Zy3YxbePCReVx95XSeeGJ+uzLDh4/g3Av+j89OObzd8ldffZUzfjKNWffez11/e4AzfjKN5cuX92T4VSlrgsuS5ArlDElnSXo4nZ6S9FqhOsuW5DKOhl3Tdt5mU/6+6BWea1nGuytbufq2uRw4Ybt2ZUaP/AB3zXkagLvmPM2BH0/W77PraO58YAHL33iL1/75Nnc+sIB9d9u6x4+ht2meM5vNNt+CkZttRt++ffnMZw/lphtvaFdm0w99iO22236180x33D6TPffam0GDBjFw4ED23Gtvbr/t1p4Mv2qVIsllyRkRcVJE7BgROwK/Af5UKLZytuSyjIZd04YO6c/il97/h6Tlpddo2rh/uzKPPb2EyXvuAMDkT2xPvw3WYVD/9Vbf9uXXGDqk/bZWekuWtDBs+LBV801Nw3ixpeOod11s29LCsOHvj5jXNGwYSzJuW+9K1JLrbs44DLiiUKXlTHKdjYbd1LGQpOMkNUtqjpVvlzGcyvjuWdez+9jNue+yb7H7TlvQ8tJrtLbmHVzIrOaoQZmmAjLlDABJmwIjgb8UqrTiFx4i4ryIGBcR49Snts5JLXn5dYZtMmDVfNMmA2h55fV2ZV5c+gZTvnUhux5xJj885yYAXv/X26tvO2QAS15uv62V3tChTSxetHjVfEvLYj7Y1Onf0erbNjWxeNH7f4MtixczNOO2dU3daskNbmvUpNNxa7jXKcA1EdFaqGA5k1yW0bBrWvP8F9hi+MZsOnQQa/Vp5JB9x3LzXY+3K7PRgPVXNdO/dfQ+XDzjfgBuv+9J9v7IaAZsuC4DNlyXvT8ymtvve7LHj6G32Wnczvx94dM89+yzrFixgmuuupIDDpyUadu999mPO++4neXLl7N8+XLuvON29t5nvzJHXP0ESNkmYGlboyadzsupqjs5YwoZuqpQ3vvkVo2GTRLoFODw/JvUltbW9zjpl9dy42//k8bGBi6+4X6eeOYf/OD4/Zk7fxE33/04e+y0BdOmHkREcO9Df+frP78agOVvvMUZF8zk3ku/CcDPzp/J8jfequTh9Ap9+vThV2f/hskHTqS1tZUvHHU0Y8Zsw49/dBpjx47jgIMm8WDzHKZ89tO8tnw5f775Rn467XSaH36cQYMG8Z3vncoeu40H4JTv/4BBgwZV+IiqQcluD8mUMySNBgYC92WKrsDg00WR9EngbN4fDfun+co3rL9JrD16StnisdJbdt/ZlQ7BuuFju+7M3AebS3rD2jof2DJGfOF/MpV9+sz9H4yIcV2t7yxnSJoGNEfEjLTM6cA6EZHptrSyPvHQ2WjYZlZnlNwzWgqd5YyIOK3D/OndqdOPdZlZUUTpklw5OMmZWdGq+CUkTnJmVjy/asnM6pfckjOzOpbcJ1e9Wc5JzsyKJF94MLP65pacmdUvn5Mzs3rmc3JmVveqOMc5yZlZ8dySM7P6VcJnV8vBSc7MitL2Prlq5SRnZkWq3HCDWTjJmVnRqjjHOcmZWfHckjOzuiVfeDCzelfNLbmKD0loZrWvG6N1FahHEyUtkLRQUqdjOEj6rKT5kuZJurxQnW7JmVnRStGSk9QInAPsQzKw9BxJMyJifk6ZUcB3gY9GxHJJQwrV65acmRUnYysuQx4cDyyMiGciYgUwHZjcocyXgHMiYjlARLxcqFInOTMritL75LJMwGBJzTnTcTlVNQGLcuYXp8tybQlsKelvku6XNLFQfO6umlnRGrNfXV2ab9zVDPoAo4AJwDDgbknbRcRrXW3glpyZFa1E3dUWYHjO/LB0Wa7FwIyIeDcingWeIkl6XXKSM7OiJAksc3c1nznAKEkjJfUFpgAzOpS5nqQVh6TBJN3XZ/JV6u6qmRWtFPcCR8RKSVOBmUAjcGFEzJM0DWiOiBnpun0lzQdagW9FxLJ89XaZ5CT9Bog8AZ24BsdhZnWoVDcDR8QtwC0dlp2W8zmAb6RTJvlacs3dDdDMeqcqfuCh6yQXERfnzktaLyLeKn9IZlZLBDRWcZYreOFB0q5p//fJdH4HSb8re2RmVhsyXnSo1POtWa6ung3sBywDiIhHgD3KGZSZ1ZZSPbtaDpmurkbEog5ZuLU84ZhZrRHQUMXd1SxJbpGk3YCQtBbwNeCJ8oZlZrWkinNcpiR3PPBrkmfIlpDcp/LVcgZlZrWj5l+aGRFLgSN6IBYzq1HV3F3NcnV1M0k3SnpF0suSbpC0WU8EZ2a1QRmnSshydfVy4Crgg8BQ4GrginIGZWa1pdZvIVkvIi6NiJXp9EdgnXIHZma1Ibm6mm2qhHzPrg5KP/45fdf6dJJnWQ+lw7NlZtaLVbCVlkW+Cw8PkiS1tui/nLMuSN6zbmZWm1dXI2JkTwZiZrWprbtarTI98SBpW2AMOefiIuKScgVlZrWlVrurAEj6IcmbOMeQnIvbH7gXcJIzM6Byt4dkkeXq6meAvYB/RMTRwA5A/7JGZWY1Q0puBs4yVUKW7urbEfGepJWS+gEv036wCTPr5aq4t5qpJdcsaQBwPskV17nAfWWNysxqSkODMk2FSJooaYGkhemtax3XH5U+ffVwOn2xUJ1Znl39Svrx95JuBfpFxKMFozWzXkGUpisqqRE4B9iHZOjBOZJmRMT8DkWvjIipWevNdzPw2HzrImJu1p2YWR0r3QsxxwMLI+IZAEnTgclAxyTXLflacr/Ksy6APYvZcWc+PHo4f3vg16Wu1spo4M6Z/0G1KvDOghfKUm83biEZLCl3kKzzIuK89HMTsChn3WJgl07qOFjSHiQDS58UEYs6KbNKvpuBP5EtZjPr7boxSv3SiBhXxK5uBK6IiHckfRm4mAINrm7EZma2OlGyt5C00P7OjWHpslUiYllEvJPOXgDsVKhSJzkzK1qfhmxTAXOAUZJGSuoLTAFm5BaQ9MGc2UlkGIoh02NdZmZdSUbiKv7KQ0SslDSVZIiFRuDCiJgnaRrQHBEzgBMlTQJWAq8CRxWqN8tjXSJ5/flmETFN0gjgAxExe80Px8zqSake0I+IW+jwKreIOC3n83fp5huQsnRXfwfsChyWzv+T5F4WMzOg9sdd3SUixkp6CCAilqf9ZTOzuhh39d30TuQAkLQx8F5ZozKzmtJYvTkuU5L7H+A6YIikn5K8leTUskZlZjVDFXzDSBZZnl29TNKDJK9bEvAfEVHwsq2Z9R5VnOMyXV0dAbxFcqfxqmURUZ7nQ8ys5tT6689v5v0BbdYBRgILgG3KGJeZ1Yiav/AQEdvlzqdvJ/lKF8XNrBeq4hzX/SceImKupM7eDGBmvZGgsYqzXJZzct/ImW0AxgJLyhaRmdWUehiScMOczytJztFdW55wzKwW1WySS28C3jAiTu6heMysBtXkuKuS+qRvBfhoTwZkZrWllrurs0nOvz0saQZwNfBm28qI+FOZYzOzWlDBh++zyHJObh1gGckrhtvulwvASc7MENCnipty+ZLckPTK6uO8n9zaRFmjMrOaUqstuUZgA9ontzZOcmaWEg2dponqkC/JvRgR03osEjOrSclANpWOomv53gxcxWGbWdVQcnU1y1SwKmmipAWSFko6JU+5gyWFpILDG+Zrye1VOCQz6+0ENJbgwkN6X+45wD4kA0vPkTQjIuZ3KLch8DXggSz1dtmSi4hX1zxcM+tNGtIXZxaaChgPLIyIZyJiBTAdmNxJuR8DvwD+nSm27hyImVlnujGQzWBJzTnTcTnVNAGLcuYXp8ty9qOxwPCIuDlrbB531cyKIrrVWloaEQXPo3W6H6kB+G8yjLWay0nOzIpTosGlgRZgeM78sHRZmw2BbYFZ6f4+AMyQNCkimruq1EnOzIpWolsx5gCjJI0kSW5TgMPbVkbE68DgVfuUZgEn50tw4CRnZkUSpXlpZvpCkKnATJKHES6MiHmSpgHNETFjTep1kjOzopXqZuCIuAW4pcOy07ooOyFLnU5yZlYk1eb75MzMsujm1dUe5yRnZkVzS87M6lr1pjgnOTMrkmp9SEIzs0LcXTWzula9Kc5JzsxKoIobck5yZlac5BaS6s1yTnJmVjS35MysjmV6IWbFOMmZWVHcXTWz+iZ3V82szjnJmVldk7urZlavSvXSzHJxkjOzolVxjnOSM7PiVXN3tZrfdVcTbpt5K9tvsxXbjN6CM3/589XWv/POO3zu8EPZZvQW7L7bLjz/3HMArFixguOOPZpxO27H+LE7cPdds3o28F5qn9225pHrfsDjN/yQk4/eZ7X1Iz44kFt+fwKzr/wuM8//Gk1DBqxad8Nvv8KLd/+Sa399fE+GXPUENCjbVLAuaaKkBZIWSjqlk/XHS3pM0sOS7pU0plCdZUtyki6U9LKkx8u1j0prbW3l6yd+lRtu/DMPPTqfq6dfwRPz57crc9GFf2DggIHMe3IhJ3ztJL7/ve8AcOEF5wPQ/PBj3HTr7ZzyrW/y3nvv9fgx9CYNDeLsUz7L5Km/48MH/4RDJu7E6M0+0K7MGSd9istuns34Q8/gZ+f9mWknTFq17qxL7uDYUy/p6bBrgDL/l7cWqRE4B9gfGAMc1kkSuzwitouIHYFfkozDmlc5W3IXARPLWH/FzZk9m80334KRm21G3759OeTQKdx04w3tytx04w0c8fkjAfj0wZ9h1l/uJCJ48on5TPjEngAMGTKE/gMG8GBz3pHVrEg7b/sh/r5oKc+1LOPdla1cPXMuB07Yvl2Z0Zt9kLtmLwDgrjlPceCE7VatmzX7Kf755js9GnNNSO+TyzIVMB5YGBHPRMQKYDowObdARLyRM7s+EIUqLVuSi4i7gVfLVX81WLKkhWHD3h8Lt6lpGC0tLauXGZ6U6dOnD/3692fZsmVst/0O3HTTDFauXMlzzz7LQ3MfZPHiRT0af28zdEh/Fr+0fNV8y0vLadq4f7syjz3VwuQ9dwRg8p470G+DdRnUf/0ejbPWtF1dzTIBgyU150zH5VTVBOT+ESxOl7Xfn/RVSX8nacmdWCi+il94SA/yOIDhI0ZUOJqec+TRx/Dkk0/w0V3GMWLTTfnIrrvR2NhY6bB6ve+edR1nfecQPjdpF/42dyEtLy2ntdWnEQrpxmWHpRExrph9RcQ5wDmSDgdOBY7MV77iSS4izgPOA9hpp3EFm57VZOjQpnatr5aWxTQ1Na1eZtEihg0bxsqVK3nj9dfZaKONkMSZvzprVbkJu+/GqFFb9ljsvdGSl19n2CYDV803bTKQlldeb1fmxVdeZ8rJFwCw/rp9+Y+9duT1f73do3HWpNJcXG0BhufMD0uXdWU68L+FKvXV1SKM23lnFi58mueefZYVK1Zw9ZXTOeDASe3KHHDgJC679GIA/nTtNXz8E3siibfeeos333wTgDvvuJ0+ffqw9ZiCF4qsCM3znmeLERuz6dCNWKtPI4fsN5abZz3arsxGA9Zf9Srvbx2zHxffcH8lQq05pbjwAMwBRkkaKakvMAWY0W4/0qic2QOApwtVWvGWXC3r06cPZ/36txx0wH60trZy5FHHMGabbZh2+mmM3WkcBx40iaOOOZZjjvo824zegoEDB3HpZdMBeOXllznogP1oaGhg6NAm/nDRpRU+mvrX2voeJ/3iKm783VdpbBAX33A/TzzzD37wnwcwd/4L3HzXY+wxbhTTTphEBNw7dyFfP+OqVdvf8Yevs+XITdhg3bVZeOuPOf5Hl3PHfU9U8IiqRyluBo6IlZKmAjOBRuDCiJgnaRrQHBEzgKmS9gbeBZZToKsKoIjy9BAlXQFMAAYDLwE/jIg/5Ntmp53Gxd8e8BXGWjJw56mVDsG64Z0FV/HeWy+X9M7drbf7cFxyw6xMZcdvPuDBYs/JdVfZWnIRcVi56jaz6iE8WpeZ1TO/T87M6l0V5zgnOTMrgSrOck5yZlakTLeHVIyTnJkVpe0tJNXKSc7MiuckZ2b1zN1VM6trvoXEzOpaFec4JzkzK5Ko6iznJGdmRUmurlZvlnOSM7OiVW+Kc5Izs1Ko4iznJGdmRfMtJGZW16r4lJyTnJkVr4pznJOcmRWn2l+a6YFszKw4pRtcGkkTJS2QtFDSKZ2s/4ak+ZIelXSnpE0L1ekkZ2ZFU8Ypbx1SI3AOsD8wBjhMUsch7B4CxkXE9sA1JANM5+UkZ2bFK0WWg/HAwoh4JiJWkIyrOjm3QET8NSLeSmfvJxmbNS8nOTMrUtZRVwUwWFJzznRcTkVNwKKc+cXpsq4cC/y5UHS+8GBmRenmSzOXlmJIQkmfA8YBHy9U1knOzIpXmourLcDwnPlh6bL2u0oGl/4+8PGIeKdQpe6umlnRutFdzWcOMErSSEl9gSnAjHb7kT4MnAtMioiXs8TmlpyZFa0Ut8lFxEpJU4GZQCNwYUTMkzQNaI6IGcCZwAbA1em9eS9ExKR89TrJmVnRSnUrcETcAtzSYdlpOZ/37m6dTnJmVpyMN/pWipOcmRWl2h/rcpIzs6JVb4pzkjOzEqjihpyTnJkVzy/NNLP6Vr05zknOzIpXxTnOSc7MiiN5SEIzq3fVm+Oc5MyseFWc45zkzKx4VdxbdZIzs2JlesNIxTjJmVlRkse6Kh1F15zkzKxoTnJmVtfcXTWz+uVXLZlZPcs22mDlOMmZWfGqOMt5IBszK1qDlGkqRNJESQskLZR0Sifr95A0V9JKSZ/JFNsaHI+ZWTvKOOWtQ2oEzgH2B8YAh0ka06HYC8BRwOVZY3N31cyKV5ru6nhgYUQ8AyBpOjAZmN9WICKeS9e9l7VSt+TMrGjdGHd1sKTmnOm4nGqagEU584vTZUWpqpbc3LkPLl13LT1f6TjKYDCwtNJBWLfU689s01JX+NDcB2eu11eDMxZfGhETSx1DPlWV5CJi40rHUA6SmiNiXKXjsOz8M8uuhEmrBRieMz8sXVYUd1fNrFrMAUZJGimpLzAFmFFspU5yZlYVImIlMBWYCTwBXBUR8yRNkzQJQNLOkhYDhwDnSmOoaoQAAASsSURBVJpXqF5FRDnjNkDScRFxXqXjsOz8M6sfTnJmVtfcXTWzuuYkZ2Z1zUnOzOqak1wZSdpK0q6S1kqfy7Ma4J9VffGFhzKR9GngZyQ3M7YAzcBFEfFGRQOzLknaMiKeSj83RkRrpWOy4rklVwaS1gIOBY6NiL2AG0ju5P6OpH4VDc46JelA4GFJlwNERKtbdPXBSa58+gGj0s/XATcBawGHS9X8sujeR9L6JDehfh1YIemP4ERXL5zkyiAi3gX+G/i0pN0j4j3gXuBh4GMVDc5WExFvAseQvKPsZGCd3ERXydiseE5y5XMPcBvweUl7RERrRFwODAV2qGxo1lFELImIf0XEUuDLwLptiU7SWEmjKxuhramqegtJPYmIf0u6DAjgu+kfyTvAJsCLFQ3O8oqIZZK+DJwp6UmgEfhEhcOyNeQkV0YRsVzS+SRvNv0y8G/gcxHxUmUjs0IiYqmkR0lexb1PRCyudEy2ZnwLSQ9JT2BHen7OqpykgcBVwDcj4tFKx2NrzknOrAuS1omIf1c6DiuOk5yZ1TVfXTWzuuYkZ2Z1zUnOzOqak5yZ1TUnuRoiqVXSw5Iel3S1pPWKqOsiSZ9JP18gaUyeshMk7bYG+3hOWn08zq6Wdyjzr27u63RJJ3c3Rqt/TnK15e2I2DEitgVWAMfnrpS0Rjd3R8QXI2J+niITgG4nObNq4CRXu+4BtkhbWfdImgHMl9Qo6UxJcyQ9mj6ehBK/lbRA0h3AkLaKJM2SNC79PFHSXEmPSLpT0odIkulJaStyd0kbS7o23cccSR9Nt91I0m2S5km6ACj4thVJ10t6MN3muA7rzkqX3ylp43TZ5pJuTbe5x8+UWiF+rKsGpS22/YFb00VjgW0j4tk0UbweETtLWhv4m6TbgA8DWwFjSJ6fnQ9c2KHejYHzgT3SugZFxKuSfg/8KyL+Ky13OXBWRNwraQTJOJlbAz8E7o2IaZIOAI7NcDjHpPtYF5gj6dqIWAasDzRHxEmSTkvrngqcBxwfEU9L2gX4HbDnGnyN1ks4ydWWdSU9nH6+B/gDSTdydkQ8my7fF9i+7Xwb0J/kvXZ7AFekrw5aIukvndT/EeDutroi4tUu4tgbGJPzWrx+kjZI9/HpdNubJS3PcEwnSvpU+nl4Gusy4D3gynT5H4E/pfvYDbg6Z99rZ9iH9WJOcrXl7YjYMXdB+sf+Zu4i4ISImNmh3CdLGEcD8JGOjzx1912gkiaQJMxdI+ItSbOAdbooHul+X+v4HZjl43Ny9Wcm8J/pK9iRtGX65tu7gUPTc3YfpPNXB90P7CFpZLrtoHT5P4ENc8rdBpzQNiOpLencDRyeLtsfGFgg1v7A8jTBjSZpSbZpANpao4eTdIPfAJ6VdEi6D0nyu/ksLye5+nMByfm2uZIeB84labFfBzydrrsEuK/jhhHxCnAcSdfwEd7vLt4IfKrtwgNwIjAuvbAxn/ev8v6IJEnOI+m2vlAg1luBPpKeAH5OkmTbvAmMT49hT2BauvwI4Ng0vnnA5AzfifVifkDfzOqaW3JmVtec5MysrjnJmVldc5Izs7rmJGdmdc1JzszqmpOcmdW1/w9ZPrlPxtqvVwAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 2 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QVNgvokAJlPT"
      },
      "source": [
        "# User Input"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-XOUr6JfXCQd",
        "outputId": "67b87b11-05a0-4b85-a6a1-53340c835825",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 551
        }
      },
      "source": [
        "# apply pre_proc() function to all data splits\n",
        "proc_train = pre_proc(df_train['review'])\n",
        "proc_val = pre_proc(df_val['review'])\n",
        "\n",
        "################################# adding user input \n",
        "reviews = [\"In the end, it reduces public-housing dwellers to the same stock roles they occupy in less thoughtful films, limiting their stories rather than allowing them to develop.\",\"Because of the heavy use of tropes, the documentary style and genre subversion are not enough to make Five Star feel urgent or authentic.\" ] # <=-- type a movie review for sentiment analysis here\n",
        "\n",
        "################################# pre-processing of user input  \n",
        "proc_inp = pre_proc(reviews)\n",
        "print (\"Processed data: \", proc_inp)\n",
        "\n",
        "################################ extract features\n",
        "X_train, X_val, X_test = extract_features(proc_train, proc_val, proc_inp, binary=False, max_df=10.0, min_df=0.0, ngram_range=(1,2), sw=False)\n",
        "\n",
        "################################ optimise C\n",
        "best_c = optimise_C(X_train, y_train, X_val, y_val, C=[0.09])\n",
        "\n",
        "################################# validation set evaluation\n",
        "#lr = LogisticRegression(C=0.09, multi_class='auto', solver='liblinear')\n",
        "#lr.fit(X_train, y_train) #fits model according to training data\n",
        "#lr_preds = lr.predict(X_val) #predict class labels \n",
        "#evaluate_classifier(y_val, lr_preds)\n",
        "\n",
        "################################# test set predictions\n",
        "lrt = LogisticRegression(C=0.09,multi_class='auto', solver='liblinear')\n",
        "lrt.fit(X_train,y_train)\n",
        "lrt_preds=lrt.predict(X_test)\n",
        "\n",
        "################################# show results\n",
        "print(\"##########################################################################\")\n",
        "print(\"Processing complete\")\n",
        "print()\n",
        "i=0\n",
        "for review in reviews:\n",
        "  print(\"'\",reviews[i],\"'\")\n",
        "  print(\"^\",lrt_preds[i], \"Sentiment ^\")\n",
        "  print()\n",
        "  i=i+1\n",
        "print(\"##########################################################################\")"
      ],
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Processed data:  ['in the end it reduces public housing dwellers to the same stock roles they occupy in less thoughtful films limiting their stories rather than allowing them to develop ', 'because of the heavy use of tropes the documentary style and genre subversion are not enough to make five star feel urgent or authentic ']\n",
            "** Vocabulary size: 1706851\n",
            "** Words:\n",
            "Accuracy for C=0.09: 0.9016\n",
            "*** Best accuracy = 0.901600, best C = 0.090000\n",
            "##########################################################################\n",
            "Processing complete\n",
            "\n",
            "' In the end, it reduces public-housing dwellers to the same stock roles they occupy in less thoughtful films, limiting their stories rather than allowing them to develop. '\n",
            "^ negative Sentiment ^\n",
            "\n",
            "' Because of the heavy use of tropes, the documentary style and genre subversion are not enough to make Five Star feel urgent or authentic. '\n",
            "^ negative Sentiment ^\n",
            "\n",
            "##########################################################################\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD4CAYAAADiry33AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAANaUlEQVR4nO3dbYiVZ37H8e9PrZuap27jNLRq1FILmZKw2xxsC20TGpKavIgk2wddd5uUsBYW98WSUAxZ2MUlhEK27Yu1BUvdh1RqJVAQNtQueSDQTYvHphpManYqJGoKmWySltYXrubfF+c2PZmdOEccnfHa7wdk7vu6ruNcB/Q7t+eeM6aqkCS1a8Fcb0CSdHEZeklqnKGXpMYZeklqnKGXpMYtmusNTLV06dJatWrVXG9Dki4rBw4ceLuqxqabm3ehX7VqFf1+f663IUmXlSSvf9ScL91IUuMMvSQ1ztBLUuMMvSQ1ztBLUuMMvSQ1ztBLUuMMvSQ1ztBLUuMMvSQ1ztBLUuMMvSQ1ztBLUuMMvSQ1ztBLUuMMvSQ1ztBLUuMMvSQ1ztBLUuMMvSQ1bqTQJ1mX5EiSiSRbp5lfmeSZJIeSPJ9k+ZT5a5IcT/L12dq4JGk0M4Y+yUJgO3AXMA5sTDI+ZdkTwLer6mZgG/D4lPmvAi9c+HYlSedrlCv6tcBEVR2tqlPAbmD9lDXjwLPd8XPD80luAa4H/vHCtytJOl+jhH4ZcGzo/Hg3NuwgcF93fC9wdZLrkiwAvgY8fK5PkGRzkn6S/uTk5Gg7lySNZLZuxj4M3JrkJeBW4ARwBvg88HRVHT/Xg6tqR1X1qqo3NjY2S1uSJAEsGmHNCWDF0PnybuwDVfUm3RV9kquAT1XVe0l+DfiNJJ8HrgIWJ/mfqvqRG7qSpItjlNDvB9YkWc0g8BuATw8vSLIUeKeq3gceAXYCVNWmoTUPAD0jL0mX1owv3VTVaWALsA94FdhTVYeTbEtyT7fsNuBIktcY3Hh97CLtV5J0nlJVc72HD+n1etXv9+d6G5J0WUlyoKp60835zlhJapyhl6TGGXpJapyhl6TGGXpJapyhl6TGGXpJapyhl6TGGXpJapyhl6TGGXpJapyhl6TGGXpJapyhl6TGGXpJapyhl6TGGXpJapyhl6TGGXpJapyhl6TGGXpJapyhl6TGGXpJapyhl6TGGXpJapyhl6TGGXpJapyhl6TGGXpJapyhl6TGGXpJapyhl6TGGXpJapyhl6TGGXpJatxIoU+yLsmRJBNJtk4zvzLJM0kOJXk+yfJu/BNJXkxyuJv7/dl+ApKkc5sx9EkWAtuBu4BxYGOS8SnLngC+XVU3A9uAx7vxk8AfVNUvAeuAP0/yU7O1eUnSzEa5ol8LTFTV0ao6BewG1k9ZMw482x0/d3a+ql6rqu93x28CbwFjs7FxSdJoRgn9MuDY0PnxbmzYQeC+7vhe4Ook1w0vSLIWWAz8x9RPkGRzkn6S/uTk5Kh7lySNYLZuxj4M3JrkJeBW4ARw5uxkkp8FngT+sKren/rgqtpRVb2q6o2NecEvSbNp0QhrTgArhs6Xd2Mf6F6WuQ8gyVXAp6rqve78GuA7wKNV9c+zsWlJ0uhGuaLfD6xJsjrJYmADsHd4QZKlSc7+Xo8AO7vxxcDfM7hR+9TsbVuSNKoZQ19Vp4EtwD7gVWBPVR1Osi3JPd2y24AjSV4Drgce68Z/D/hN4IEk/9b9+sRsPwlJ0kdLVc31Hj6k1+tVv9+f621I0mUlyYGq6k035ztjJalxhl6SGmfoJalxhl6SGmfoJalxhl6SGmfoJalxhl6SGmfoJalxhl6SGmfoJalxhl6SGmfoJalxhl6SGmfoJalxhl6SGmfoJalxhl6SGmfoJalxhl6SGmfoJalxhl6SGmfoJalxhl6SGmfoJalxhl6SGmfoJalxhl6SGmfoJalxhl6SGmfoJalxhl4axa5dsGoVLFgw+Lhr11zvSBrZornegDTv7doFmzfDyZOD89dfH5wDbNo0d/uSRuQVvTSTRx/9/8ifdfLkYFy6DIwU+iTrkhxJMpFk6zTzK5M8k+RQkueTLB+auz/J97tf98/m5qVL4o03zm9cmmdmDH2ShcB24C5gHNiYZHzKsieAb1fVzcA24PHusT8NfBn4FWAt8OUkH5+97UuXwA03nN+4NM+MckW/FpioqqNVdQrYDayfsmYceLY7fm5o/reB71bVO1X1LvBdYN2Fb1u6hB57DJYs+fDYkiWDcekyMErolwHHhs6Pd2PDDgL3dcf3AlcnuW7Ex0rz26ZNsGMHrFwJyeDjjh3eiNVlY7a+6+Zh4OtJHgBeAE4AZ0Z9cJLNwGaAG/znsOajTZsMuy5bo1zRnwBWDJ0v78Y+UFVvVtV9VfVJ4NFu7L1RHtut3VFVvarqjY2NnedTkCSdyyih3w+sSbI6yWJgA7B3eEGSpUnO/l6PADu7433AnUk+3t2EvbMbkyRdIjOGvqpOA1sYBPpVYE9VHU6yLck93bLbgCNJXgOuBx7rHvsO8FUGXyz2A9u6MUnSJZKqmus9fEiv16t+vz/X25Cky0qSA1XVm27Od8ZKUuMMvSQ1ztBLUuMMvSQ1ztBLUuMMvSQ1ztBLUuMMvSQ1ztBLUuMMvSQ1ztBLUuMMvSQ1ztBLUuMMvSQ1ztBLUuMMvSQ1ztBLUuMMvSQ1ztBLUuMMvSQ1ztBLUuMMvSQ1ztBLUuMMvSQ1ztBLUuMMvSQ1ztBLUuMMvSQ1ztBLUuMMvSQ1ztBLUuMMvSQ1ztBLUuMMvSQ1ztBLUuNGCn2SdUmOJJlIsnWa+RuSPJfkpSSHktzdjf9Ekm8leTnJq0keme0nIEk6txlDn2QhsB24CxgHNiYZn7LsS8CeqvoksAH4i278d4GPVdVNwC3AHyVZNTtblySNYpQr+rXARFUdrapTwG5g/ZQ1BVzTHV8LvDk0fmWSRcBPAqeA/77gXUuSRjZK6JcBx4bOj3djw74CfCbJceBp4Avd+FPA/wL/CbwBPFFV70z9BEk2J+kn6U9OTp7fM5AkndNs3YzdCHyzqpYDdwNPJlnA4F8DZ4CfA1YDDyX5+akPrqodVdWrqt7Y2NgsbUmSBKOF/gSwYuh8eTc27EFgD0BVvQhcASwFPg38Q1X9sKreAv4J6F3opiVJoxsl9PuBNUlWJ1nM4Gbr3ilr3gBuB0hyI4PQT3bjv9WNXwn8KvDvs7N1SdIoZgx9VZ0GtgD7gFcZfHfN4STbktzTLXsI+FySg8DfAg9UVTH4bp2rkhxm8AXjG1V16GI8EUnS9DLo8fzR6/Wq3+/P9TYk6bKS5EBVTfvSuO+MlaTGGXpJapyhl6TGGXpJapyhl6TGGXpJapyhl6TGGXpJapyhl6TGGXpJapyhl6TGGXpJapyhl6TGGXpJapyhl6TGGXpJapyhl6TGGXpJapyhl6TGGXpJapyhl6TGGXpJapyhl6TGGXpJapyhl6TGGXpJapyhl6TGGXpJapyhl6TGGXpJapyhl6TGGXpJapyhl6TGGXpJapyhl6TGjRT6JOuSHEkykWTrNPM3JHkuyUtJDiW5e2ju5iQvJjmc5OUkV8zmE5AknduimRYkWQhsB+4AjgP7k+ytqleGln0J2FNVf5lkHHgaWJVkEfA3wGer6mCS64AfzvqzkCR9pFGu6NcCE1V1tKpOAbuB9VPWFHBNd3wt8GZ3fCdwqKoOAlTVD6rqzIVvW5I0qlFCvww4NnR+vBsb9hXgM0mOM7ia/0I3/otAJdmX5F+T/PF0nyDJ5iT9JP3JycnzegKSpHObrZuxG4FvVtVy4G7gySQLGLw09OvApu7jvUlun/rgqtpRVb2q6o2Njc3SliRJMFroTwArhs6Xd2PDHgT2AFTVi8AVwFIGV/8vVNXbVXWSwdX+L1/opiVJoxsl9PuBNUlWJ1kMbAD2TlnzBnA7QJIbGYR+EtgH3JRkSXdj9lbgFSRJl8yM33VTVaeTbGEQ7YXAzqo6nGQb0K+qvcBDwF8l+SKDG7MPVFUB7yb5UwZfLAp4uqq+c7GejCTpR2XQ4/mj1+tVv9+f621I0mUlyYGq6k035ztjJalxhl6SGmfoJalxhl6SGmfoJalxhl6SGmfoJalxhl6SGmfoJalxhl6SGmfoJalxhl6SGjfvfqhZkkng9bneh/QRlgJvz/UmpGmsrKpp/+emeRd6aT5L0v+onxAozVe+dCNJjTP0ktQ4Qy+dnx1zvQHpfPkavSQ1zit6SWqcoZekxhl6CUiyLsmRJBNJtk4z/7Ekf9fN/0uSVd344iTfSPJykoNJbrvEW5dmZOj1Yy/JQmA7cBcwDmxMMj5l2YPAu1X1C8CfAX/SjX8OoKpuAu4AvpbEv1eaV/wDKcFaYKKqjlbVKWA3sH7KmvXAt7rjp4Dbk4TBF4ZnAarqLeA9wDdUaV4x9BIsA44NnR/vxqZdU1Wngf8CrgMOAvckWZRkNXALsOKi71g6D4vmegPSZW4ncCPQZ/Azmr4HnJnTHUlTGHoJTvDhq/Dl3dh0a44nWQRcC/ygBm9E+eLZRUm+B7x2cbcrnR9fupFgP7Amyeoki4ENwN4pa/YC93fHvwM8W1WVZEmSKwGS3AGcrqpXLtXGpVF4Ra8fe1V1OskWYB+wENhZVYeTbAP6VbUX+GvgySQTwDsMvhgA/AywL8n7DK76P3vpn4F0bv4IBElqnC/dSFLjDL0kNc7QS1LjDL0kNc7QS1LjDL0kNc7QS1Lj/g+AsRjEPcc4nAAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    }
  ]
}