EDA+Cleaning.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.3"
    },
    "colab": {
      "name": "EDA+Cleaning.ipynb",
      "provenance": [],
      "collapsed_sections": [
        "PaUUDcfZD9L9",
        "Z1fgNMNBCQ-5",
        "TfjT654BCQ-5",
        "LDjAfi8oFEBq",
        "PnqLlSLhG7BT",
        "dm7_0wyhJ3NB",
        "u-N9UtaoMKQ3",
        "SmSc9qGDM15C",
        "4H6p2aWjOS3g",
        "vyl0Cg3xOsFV",
        "A64q-2qGPW4P",
        "j35YeozRPzxB",
        "XvYqbYqbQLuK",
        "lM4YwN5ERDOx",
        "rkYgeIsOR8jX"
      ],
      "include_colab_link": true
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/ricardoaiworld/DatabaseProject/blob/master/EDA%2BCleaning.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PyKKjt7JSg8J"
      },
      "source": [
        "**CS-GY 9223: Deep Learning Final Project: Solving Natural Language Processing with Disaster Tweets on Kaggle with BERT**\r\n",
        "\r\n",
        "by:\r\n",
        "\r\n",
        "Yan Sun    <ys3928@nyu.edu>\r\n",
        "\r\n",
        "Zhenming Wang <zw2365@nyu.edu>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tghM6QS4UEnt"
      },
      "source": [
        "**Section 1: Exploratory Data Analysis and Data Cleaning**\r\n",
        "\r\n",
        "Reference:https://www.kaggle.com/shahules/basic-eda-cleaning-and-glove"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PaUUDcfZD9L9"
      },
      "source": [
        "# Step 0: Set Up Working Envoronment And Libraries"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "AtBCOwTHCQ-z",
        "outputId": "b82bc1a7-77be-4c14-d754-5a4978629ab4"
      },
      "source": [
        "from google.colab import drive\r\n",
        "drive.mount('/content/gdrive')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Mounted at /content/gdrive\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "jpikAD1oDx-2",
        "outputId": "c6485f3f-9ffa-4952-c7ae-7df5a86ef87b"
      },
      "source": [
        "import os\r\n",
        "os.environ['KAGGLE_CONFIG_DIR'] = \"/content/gdrive/My Drive/Deep Learning Final Project\"                                    \r\n",
        "%cd /content/gdrive/My Drive/Deep Learning Final Project\r\n",
        "\r\n",
        "!pwd"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/content/gdrive/My Drive/Deep Learning Final Project\n",
            "/content/gdrive/My Drive/Deep Learning Final Project\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "bArhTvGQD5Z1",
        "outputId": "7d1f82c3-08ec-4ca9-d267-42d7911627d8"
      },
      "source": [
        "!ls"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            " best_model_state.bin\t\t\t\t sample_submission.csv\n",
            "'CS-GY 9223 Deep Learning Final Project.ipynb'\t test.csv\n",
            " EDA.ipynb\t\t\t\t\t train.csv\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "jjh2Dx1sCQ-4",
        "outputId": "1bfcb062-96d0-40fb-aa16-0491a58d0d16"
      },
      "source": [
        "!python -m pip install wordcloud\n",
        "import gc\n",
        "import re\n",
        "import string\n",
        "import operator\n",
        "from collections import defaultdict\n",
        "\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "pd.set_option('display.max_rows', 500)\n",
        "pd.set_option('display.max_columns', 500)\n",
        "pd.set_option('display.width', 1000)\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "import seaborn as sns\n",
        "\n",
        "import nltk\n",
        "nltk.download('stopwords')\n",
        "from nltk.corpus import stopwords\n",
        "from nltk.util import ngrams\n",
        "from sklearn.feature_extraction.text import CountVectorizer\n",
        "from collections import defaultdict\n",
        "from collections import  Counter\n",
        "plt.style.use('ggplot')\n",
        "stop=set(stopwords.words('english'))\n",
        "from nltk.tokenize import word_tokenize\n",
        "import gensim\n",
        "\n",
        "#from wordcloud import STOPWORDS"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: wordcloud in /usr/local/lib/python3.6/dist-packages (1.5.0)\n",
            "Requirement already satisfied: pillow in /usr/local/lib/python3.6/dist-packages (from wordcloud) (7.0.0)\n",
            "Requirement already satisfied: numpy>=1.6.1 in /usr/local/lib/python3.6/dist-packages (from wordcloud) (1.18.5)\n",
            "[nltk_data] Downloading package stopwords to /root/nltk_data...\n",
            "[nltk_data]   Package stopwords is already up-to-date!\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "o_PRY8vkDiMK"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Z1fgNMNBCQ-5"
      },
      "source": [
        "# Step 1: Exploratory Data Analysis"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TfjT654BCQ-5"
      },
      "source": [
        "## 1.1 Read data from files into pandas dataframe "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dtNcASY0CQ-5"
      },
      "source": [
        "df_train = pd.read_csv('train.csv')\n",
        "df_test = pd.read_csv('test.csv')\n",
        "df_sample_submission = pd.read_csv('sample_submission.csv')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "c29Gl-rKEpaM",
        "outputId": "5dd4382e-6d5c-49cc-a785-becd6ffced69"
      },
      "source": [
        "# Output the size of each dataframe\r\n",
        "df_train.shape, df_test.shape, df_sample_submission.shape"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "((7613, 5), (3263, 4), (3263, 2))"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 198
        },
        "id": "4E6L9ds9EvM6",
        "outputId": "267aa27c-8572-47e4-e9a8-bf55fac45912"
      },
      "source": [
        "# Quick look at the content inside each dataframe\r\n",
        "df_train.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>id</th>\n",
              "      <th>keyword</th>\n",
              "      <th>location</th>\n",
              "      <th>text</th>\n",
              "      <th>target</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Our Deeds are the Reason of this #earthquake M...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>4</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Forest fire near La Ronge Sask. Canada</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>5</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>All residents asked to 'shelter in place' are ...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>6</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>13,000 people receive #wildfires evacuation or...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>7</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Just got sent this photo from Ruby #Alaska as ...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   id keyword location                                               text  target\n",
              "0   1     NaN      NaN  Our Deeds are the Reason of this #earthquake M...       1\n",
              "1   4     NaN      NaN             Forest fire near La Ronge Sask. Canada       1\n",
              "2   5     NaN      NaN  All residents asked to 'shelter in place' are ...       1\n",
              "3   6     NaN      NaN  13,000 people receive #wildfires evacuation or...       1\n",
              "4   7     NaN      NaN  Just got sent this photo from Ruby #Alaska as ...       1"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 198
        },
        "id": "I1Rs2C5KExv9",
        "outputId": "2a56a6ca-6468-4ede-ca0b-024284e52668"
      },
      "source": [
        "df_test.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>id</th>\n",
              "      <th>keyword</th>\n",
              "      <th>location</th>\n",
              "      <th>text</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Just happened a terrible car crash</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Heard about #earthquake is different cities, s...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>3</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>there is a forest fire at spot pond, geese are...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>9</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Apocalypse lighting. #Spokane #wildfires</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>11</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Typhoon Soudelor kills 28 in China and Taiwan</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   id keyword location                                               text\n",
              "0   0     NaN      NaN                 Just happened a terrible car crash\n",
              "1   2     NaN      NaN  Heard about #earthquake is different cities, s...\n",
              "2   3     NaN      NaN  there is a forest fire at spot pond, geese are...\n",
              "3   9     NaN      NaN           Apocalypse lighting. #Spokane #wildfires\n",
              "4  11     NaN      NaN      Typhoon Soudelor kills 28 in China and Taiwan"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 198
        },
        "id": "chF9K5EnEx6M",
        "outputId": "cc8db92c-d1e5-4e94-dd47-047379967e6f"
      },
      "source": [
        "df_sample_submission.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>id</th>\n",
              "      <th>target</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>3</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>9</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>11</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   id  target\n",
              "0   0       0\n",
              "1   2       0\n",
              "2   3       0\n",
              "3   9       0\n",
              "4  11       0"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 198
        },
        "id": "Lh3VN3SoIX2O",
        "outputId": "9def5946-fcd1-494f-d17e-52aeebd1ca66"
      },
      "source": [
        "# Merge df_sample_submission into df_test\r\n",
        "df_test['target'] = df_sample_submission['target']\r\n",
        "df_test.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>id</th>\n",
              "      <th>keyword</th>\n",
              "      <th>location</th>\n",
              "      <th>text</th>\n",
              "      <th>target</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Just happened a terrible car crash</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Heard about #earthquake is different cities, s...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>3</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>there is a forest fire at spot pond, geese are...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>9</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Apocalypse lighting. #Spokane #wildfires</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>11</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Typhoon Soudelor kills 28 in China and Taiwan</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   id keyword location                                               text  target\n",
              "0   0     NaN      NaN                 Just happened a terrible car crash       0\n",
              "1   2     NaN      NaN  Heard about #earthquake is different cities, s...       0\n",
              "2   3     NaN      NaN  there is a forest fire at spot pond, geese are...       0\n",
              "3   9     NaN      NaN           Apocalypse lighting. #Spokane #wildfires       0\n",
              "4  11     NaN      NaN      Typhoon Soudelor kills 28 in China and Taiwan       0"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 22
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LDjAfi8oFEBq"
      },
      "source": [
        "## 1.2 Class Distribution"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GQCAQwvTFLIl"
      },
      "source": [
        "First things fisrt, let us check the class distribution. There are only two classes 0 and 1."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 353
        },
        "id": "R3qc-C76FHGd",
        "outputId": "2a20677c-b27a-4193-c6fc-59b84bec9240"
      },
      "source": [
        "# Class distribution for training set\r\n",
        "x = df_train.target.value_counts()\r\n",
        "sns.barplot(x.index,x)\r\n",
        "plt.gca().set_ylabel('samples')\r\n",
        "plt.title('Training Set Class Distribution')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/seaborn/_decorators.py:43: FutureWarning: Pass the following variables as keyword args: x, y. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.\n",
            "  FutureWarning\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Text(0.5, 1.0, 'Training Set Class Distribution')"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 15
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEICAYAAACuxNj9AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAVeUlEQVR4nO3de7RkZX3m8e8jF9FBaJAeRJq2iZBM0EzUtIK3iYoDjaK41qjB8dI4ZLFMnIyZGCOaKF5wVIzXSbygEBGNiOiKxHE0jEqQFVEajBdAYkdFGrk0dxABG37zx36PKZpzzlvdnjrnNOf7WatW1X737Vd16uyn9rt37UpVIUnSbO630AVIkhY/w0KS1GVYSJK6DAtJUpdhIUnqMiwkSV2GheZckv+bZO1cT7stSXJUknMXuo5RST6Y5HVztKyVSW5Nsl0bPjvJ78/Fstvy7pPvi22ZYSEA2j/+1O3uJD8fGX7hliyrqg6rqlPmetotleS1SX7UnsOGJJ8ac76xNvRJDk1yTpJbkmxM8o9Jnv2rV77lkvy4/c1uSXJjkn9K8rIkv/wfr6qXVdWbx1zW02ebpqp+UlU7V9Vdc1D7G5J8fLPlT+x9oa1jWAiA9o+/c1XtDPwEeNZI2yempkuy/cJVOb72qfTFwNPbc1oNfHkOl/9c4NPAx4AVwJ7A64FnzdU6tsKzqupBwMOAtwGvBk6a65VsK+8BzbGq8ubtHjfgxwwbWYCnABsYNjxXAacCuwGfBzYCN7THK0bmPxv4/fb4KOBc4C/btD8CDtvKafcFzgFuAf4f8NfAx2d4Dn8FvGeW57grw4b0SuAK4HhgO+A3gduBu4BbgRunmTcMgfqqWZZ/FHDuyPB7gcuBm4ELgCePjHscsK6Nuxp4V2vfCfg4cB1wI3A+sGfvb7bZcu8GHtmGPwoc3x7v0f5uNwLXA19j+PB4apvn5+35/xmwCijg6Pa8zxlp237k7/hW4JvteXwO2H30PTRdvcAa4E7gF219357mfXE/4C+Ay4BrGAJ61zZuqo61rbZrgT9f6P+h++LNPQuN4yHA7gyfWI9h+Of9mza8kmHD8lezzH8gcCnDBuoE4KQk2Ypp/5ZhY/Rg4A0Mew4zOQ94SZJXJVk91bc+4qPAJmA/4NHAIQwbp0uAlwFfr2Gvatk0y/4NYB/gjFnWv7nzgUcxvI5/C3w6yU5t3HuB91bVLsDDgdNb+1qGUNuH4Tm/jOG1HktVfZMh6J88zehXtnHLGfaKXjvMUi/mnnuWJ4zM87sMYXroDKt8CfDfgL0YXtv3jVHjF4H/BXyqre+3p5nsqHZ7KvBrwM7c+/32JIa/y8HA65P8Zm/d2jKGhcZxN3BcVd1RVT+vquuq6jNVdVtV3QK8hWFDMpPLqurDNfRvn8KwMdlzS6ZNshJ4LPD6qrqzqs4FzpxphVX1ceCPGDZs/whck+TVAEn2BJ4B/HFV/ayqrgHeDRw55uvx4HZ/5ZjTU1Ufb6/bpqp6J3B/ho0bDJ+q90uyR1XdWlXnjbQ/GNivqu6qqguq6uZx19n8lCGgNvcLhtf2YVX1i6r6WrWP6rN4Q3u9ZgqsU6vqe1X1M+B1wPOnCemt8UKGva0fVtWtwGuAIzfrDntje29+G/g2MF3o6FdgWGgcG6vq9qmBJA9M8qEklyW5maFbYtksG4arph5U1W3t4c5bOO1DgetH2mDo1plRVX2iqp4OLGP4VP7mJIcy7BHtAFzZDgbfCHwI+PezLW/Ede1+rzGnJ8mfJrkkyU1tfbsy7D3B0L3z68D3k5yf5PDWfirwJeC0JD9NckKSHcZdZ7M3QzfT5t4BrAf+IckPkxw7xrJmfb03G38Zw2u8xwzTbomHtuWNLnt77vmB46qRx7cx8/tLW8mw0Dg2/8T5SoZPxQe2rpP/1Npn6lqaC1cCuyd54EjbPuPM2D45fxr4DvBIho3aHcAeVbWs3XapqkdMzdJZ5KVtGf9lnPUneTJD3//zgd1a19ZNtNerqn5QVS9gCKu3A2ck+Xet7jdW1QHAE4DDGbp6xpLksQxhca8zu6rqlqp6ZVX9GvBs4E+SHDw1eoZF9l6X0b/HSoa9l2uBnwG//Lu1DxXLt2C5P2UI+NFlb2I4vqN5YlhoazyIoe/8xiS7A8dNeoVVdRnDQeA3JNkxyeOZ5cyjdvrrM5M8KMn9khwGPAL4RlVdCfwD8M4ku7TxD08y1ZV2NbAiyY4z1FLAnwCvS/LSkWU8KcmJ08zyIIaN20Zg+ySvB3YZqfVFSZZX1d0MB5wB7k7y1CS/1TauNzNsfO/uvVatnsOB0xhOAPjuNNMcnmS/djzoJoYD+lPLvprh2MCWelGSA1qgvwk4o3Un/guwU/t77MBwsPr+I/NdDawaPc13M58E/meSfZPszL8d49i0FTVqKxkW2hrvAR7A8KnxPOCL87TeFwKPZ+gGOh74FMMewnRuZjho+xOGDfAJwB+0Yx0wfELfEbiY4cyrM/i3bqWvABcBVyW5drqFV9UZwO8xHND9KcMG73iGs4A29yWG1+hfGLpQbueeXTZrgIuS3MpwsPvIdlzgIa2um4FLGI69nDrD8wX4+yS3tGX/OfAu4KUzTLs/wxlltwJfB95fVV9t494K/EXrovvTWda3uVMZThy4iuFMrv8BUFU3AX8IfIThzLOfMRxcn/Lpdn9dkgunWe7JbdnnMJwhdzvD8SjNo/SPaUmLU/uS3ferauJ7NtJS556FthlJHtu6i+6XZA1wBPB3C12XtBT4TUxtSx4CfJbhdNINDN1K31rYkqSlwW4oSVKX3VCSpK77ZDfUHnvsUatWrVroMiRpm3LBBRdcW1XLpxt3nwyLVatWsW7duoUuQ5K2KUkum2mc3VCSpC7DQpLUZVhIkroMC0lSl2EhSeoyLCRJXYaFJKnLsJAkdRkWkqSu++Q3uOfC77zqYwtdghahC94x9q+aSvcp7llIkroMC0lSl2EhSeoyLCRJXYaFJKnLsJAkdRkWkqQuw0KS1GVYSJK6DAtJUpdhIUnqMiwkSV0TD4sk2yX5VpLPt+F9k3wjyfokn0qyY2u/fxte38avGlnGa1r7pUkOnXTNkqR7mo89i1cAl4wMvx14d1XtB9wAHN3ajwZuaO3vbtOR5ADgSOARwBrg/Um2m4e6JUnNRMMiyQrgmcBH2nCApwFntElOAZ7THh/RhmnjD27THwGcVlV3VNWPgPXA4yZZtyTpnia9Z/Ee4M+Au9vwg4Ebq2pTG94A7N0e7w1cDtDG39Sm/2X7NPP8UpJjkqxLsm7jxo1z/TwkaUmbWFgkORy4pqoumNQ6RlXViVW1uqpWL1++fD5WKUlLxiR/Ke+JwLOTPAPYCdgFeC+wLMn2be9hBXBFm/4KYB9gQ5LtgV2B60bap4zOI0maBxPbs6iq11TViqpaxXCA+itV9ULgq8Bz22Rrgc+1x2e2Ydr4r1RVtfYj29lS+wL7A9+cVN2SpHtbiN/gfjVwWpLjgW8BJ7X2k4BTk6wHrmcIGKrqoiSnAxcDm4CXV9Vd81+2JC1d8xIWVXU2cHZ7/EOmOZupqm4HnjfD/G8B3jK5CiVJs/Eb3JKkLsNCktRlWEiSugwLSVKXYSFJ6jIsJEldhoUkqcuwkCR1GRaSpC7DQpLUZVhIkroMC0lSl2EhSeoyLCRJXYaFJKnLsJAkdRkWkqQuw0KS1GVYSJK6DAtJUpdhIUnqMiwkSV2GhSSpy7CQJHUZFpKkLsNCktRlWEiSugwLSVKXYSFJ6tp+oQuQtGV+8qbfWugStAitfP13J7p89ywkSV2GhSSpy7CQJHUZFpKkLsNCktRlWEiSugwLSVKXYSFJ6jIsJEldEwuLJDsl+WaSbye5KMkbW/u+Sb6RZH2STyXZsbXfvw2vb+NXjSzrNa390iSHTqpmSdL0JrlncQfwtKr6beBRwJokBwFvB95dVfsBNwBHt+mPBm5o7e9u05HkAOBI4BHAGuD9SbabYN2SpM1MLCxqcGsb3KHdCngacEZrPwV4Tnt8RBumjT84SVr7aVV1R1X9CFgPPG5SdUuS7m2ixyySbJfkn4FrgLOAfwVurKpNbZINwN7t8d7A5QBt/E3Ag0fbp5lndF3HJFmXZN3GjRsn8XQkacmaaFhU1V1V9ShgBcPewH+Y4LpOrKrVVbV6+fLlk1qNJC1J83I2VFXdCHwVeDywLMnUpdFXAFe0x1cA+wC08bsC1422TzOPJGkeTPJsqOVJlrXHDwD+M3AJQ2g8t022Fvhce3xmG6aN/0pVVWs/sp0ttS+wP/DNSdUtSbq3Sf740V7AKe3MpfsBp1fV55NcDJyW5HjgW8BJbfqTgFOTrAeuZzgDiqq6KMnpwMXAJuDlVXXXBOuWJG1mYmFRVd8BHj1N+w+Z5mymqrodeN4My3oL8Ja5rlGSNB6/wS1J6jIsJEldhoUkqcuwkCR1GRaSpC7DQpLUZVhIkroMC0lSl2EhSeoyLCRJXYaFJKlrrLBIckKSXZLskOTLSTYmedGki5MkLQ7j7lkcUlU3A4cDPwb2A141qaIkSYvLuGExdXXaZwKfrqqbJlSPJGkRGvcS5Z9P8n3g58AfJFkO3D65siRJi8lYexZVdSzwBGB1Vf0CuA04YpKFSZIWj3EPcD8Q+EPgA63pocDqSRUlSVpcxj1m8TfAnQx7FwBXAMdPpCJJ0qIzblg8vKpOAH4BUFW3AZlYVZKkRWXcsLgzyQOAAkjycOCOiVUlSVpUxj0b6jjgi8A+ST4BPBE4alJFSZIWl7HCoqrOSnIhcBBD99MrquraiVYmSVo0Zg2LJI/ZrOnKdr8yycqqunAyZUmSFpPensU7ZxlXwNPmsBZJ0iI1a1hU1VPnqxBJ0uI11jGLJDsxfCnvSQx7FF8DPlhVXvJDkpaAcc+G+hhwC/C/2/B/BU4FnjeJoiRJi8u4YfHIqjpgZPirSS6eREGSpMVn3C/lXZjkoKmBJAcC6yZTkiRpsRl3z+J3gH9K8pM2vBK4NMl3gaqq/ziR6iRJi8K4YbFmolVIkha1cb/BfVmS3YB9RufxS3mStDSMe+rsmxmuBfWvtIsJ4pfyJGnJGLcb6vkMlym/c5LFSJIWp3HPhvoesGyShUiSFq9x9yzeCnwryfcY+R2Lqnr2RKqSJC0q44bFKcDbge8Cd0+uHEnSYjRuWNxWVe+baCWSpEVr3GMWX0vy1iSPT/KYqdtsMyTZJ8lXk1yc5KIkr2jtuyc5K8kP2v1urT1J3pdkfZLvjC4/ydo2/Q+SrN3qZytJ2irj7lk8ut0fNNLWO3V2E/DKqrowyYOAC5KcxXAK7per6m1JjgWOBV4NHAbs324HAh8ADkyyO8PPuq5u67wgyZlVdcOYtUuSfkXjfilvi3/XoqqupP2yXlXdkuQSYG/gCOApbbJTgLMZwuII4GNVVcB5SZYl2atNe1ZVXQ/QAmcN8MktrUmStHXG3bMgyTOBRwA7TbVV1ZvGnHcVw97JN4A9W5AAXAXs2R7vDVw+MtuG1jZT++brOAY4BmDlypXjlCVJGtNYxyySfBD4PeCPgDD8jsXDxpx3Z+AzwB9X1c2j49peRE074xaqqhOranVVrV6+fPlcLFKS1Ix7gPsJVfUS4IaqeiPweODXezMl2YEhKD5RVZ9tzVe37iXa/TWt/QqGa09NWdHaZmqXJM2TccNi6udTb0vyUIaD13vNNkOSACcBl1TVu0ZGnQlMndG0FvjcSPtL2llRBwE3te6qLwGHJNmtnTl1SGuTJM2TcY9Z/H2SZcA7gAsZuo4+3JnnicCLge8m+efW9lrgbcDpSY4GLmO47hTAF4BnAOuB24CXAlTV9e1Chue36d40dbBbkjQ/xg2L7wN3VdVnkhwAPAb4u9lmqKpzGY5vTOfgaaYv4OUzLOtk4OQxa5UkzbFxu6Fe105/fRLDdys+wvA9CEnSEjBuWNzV7p8JfLiq/g+w42RKkiQtNuOGxRVJPsRw+uwXktx/C+aVJG3jxt3gP5/hDKRDq+pGYHfgVROrSpK0qIx7uY/bgM+ODP/yUh6SpPs+u5IkSV2GhSSpy7CQJHUZFpKkLsNCktRlWEiSugwLSVKXYSFJ6jIsJEldhoUkqcuwkCR1GRaSpC7DQpLUZVhIkroMC0lSl2EhSeoyLCRJXYaFJKnLsJAkdRkWkqQuw0KS1GVYSJK6DAtJUpdhIUnqMiwkSV2GhSSpy7CQJHUZFpKkLsNCktRlWEiSugwLSVKXYSFJ6jIsJEldhoUkqWtiYZHk5CTXJPneSNvuSc5K8oN2v1trT5L3JVmf5DtJHjMyz9o2/Q+SrJ1UvZKkmU1yz+KjwJrN2o4FvlxV+wNfbsMAhwH7t9sxwAdgCBfgOOBA4HHAcVMBI0maPxMLi6o6B7h+s+YjgFPa41OA54y0f6wG5wHLkuwFHAqcVVXXV9UNwFncO4AkSRM238cs9qyqK9vjq4A92+O9gctHptvQ2mZqv5ckxyRZl2Tdxo0b57ZqSVriFuwAd1UVUHO4vBOranVVrV6+fPlcLVaSxPyHxdWte4l2f01rvwLYZ2S6Fa1tpnZJ0jya77A4E5g6o2kt8LmR9pe0s6IOAm5q3VVfAg5Jsls7sH1Ia5MkzaPtJ7XgJJ8EngLskWQDw1lNbwNOT3I0cBnw/Db5F4BnAOuB24CXAlTV9UneDJzfpntTVW1+0FySNGETC4uqesEMow6eZtoCXj7Dck4GTp7D0iRJW8hvcEuSugwLSVKXYSFJ6jIsJEldhoUkqcuwkCR1GRaSpC7DQpLUZVhIkroMC0lSl2EhSeoyLCRJXYaFJKnLsJAkdRkWkqQuw0KS1GVYSJK6DAtJUpdhIUnqMiwkSV2GhSSpy7CQJHUZFpKkLsNCktRlWEiSugwLSVKXYSFJ6jIsJEldhoUkqcuwkCR1GRaSpC7DQpLUZVhIkroMC0lSl2EhSeoyLCRJXYaFJKnLsJAkdRkWkqSubSYskqxJcmmS9UmOXeh6JGkp2SbCIsl2wF8DhwEHAC9IcsDCViVJS8c2ERbA44D1VfXDqroTOA04YoFrkqQlY/uFLmBMewOXjwxvAA4cnSDJMcAxbfDWJJfOU21LwR7AtQtdxGKQv1y70CXonnxvTjkuc7GUh800YlsJi66qOhE4caHruC9Ksq6qVi90HdLmfG/On22lG+oKYJ+R4RWtTZI0D7aVsDgf2D/Jvkl2BI4EzlzgmiRpydgmuqGqalOS/w58CdgOOLmqLlrgspYSu/e0WPnenCepqoWuQZK0yG0r3VCSpAVkWEiSugwLzcrLrGgxSnJykmuSfG+ha1kqDAvNyMusaBH7KLBmoYtYSgwLzcbLrGhRqqpzgOsXuo6lxLDQbKa7zMreC1SLpAVkWEiSugwLzcbLrEgCDAvNzsusSAIMC82iqjYBU5dZuQQ43cusaDFI8kng68BvJNmQ5OiFrum+zst9SJK63LOQJHUZFpKkLsNCktRlWEiSugwLSVKXYSFJ6jIsJEld/x9a1kfL+nCj1wAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 353
        },
        "id": "4-oEHbopFlD0",
        "outputId": "09e4a188-f491-4c3d-f960-3f9545c7c69e"
      },
      "source": [
        "# Class distribution for test set\r\n",
        "x = df_sample_submission.target.value_counts()\r\n",
        "sns.barplot(x.index,x)\r\n",
        "plt.gca().set_ylabel('samples')\r\n",
        "plt.title('Test Set Class Distribution')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/seaborn/_decorators.py:43: FutureWarning: Pass the following variables as keyword args: x, y. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.\n",
            "  FutureWarning\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Text(0.5, 1.0, 'Test Set Class Distribution')"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 16
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEICAYAAACuxNj9AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAW2ElEQVR4nO3de7hddZ3f8fdHELzgCEiGQhIJI7FTdEbEyEVtx0vlptNgqxSqQ2QYo1OYx+ljnaI+Fm+o442W0dHGMRUvIzJexlRTMaIzah0ugaFCQEpEkAQIkYSbART49o/1O3U3nHPWTjj7nBPyfj3Pfs5av/Vbv/VbG7I/e/3WZaeqkCRpMo+Z6Q5IkmY/w0KS1MuwkCT1MiwkSb0MC0lSL8NCktTLsJCGkOSFSdbNdD8GJXlrkr+awvbuSfJbbfrTSd4zhW1/Isnbp6o9TT/DQtulfbCMvR5Kcu/A/Ku3o72/S/JHPXVOTfLjJHcn2ZBkZZInDdH2UB/0SQ5rbd6RZFOSS5Kcsi37MVXa+3Ff29e7klyW5Iwku4/Vqar3VtWk79lAW731qmqPqrp+Cvr+2iQ/2KrtN1TVux9p25o5hoW2S/tg2aOq9gB+Bvz+QNnnp3p7SX4PeC9wUlU9CfhnwBensP0jge8Afw8cBDwF+GPg2KnaxnY4ve3rfsCbgBOBlUkylRtJsutUtqdHqary5esRvYAbgH/Zph8DnAH8BLgdOB/Yuy17HPC5Vn4HcCmwL3AW8CBwH3AP8NFxtvEfgb+dpA+7Ax+iC64NwCeAxwNPBO4FHmpt3wPsP876PwA+Nkn7LwTWDcyP7ePdwNXAKwaWHUQXOncCPwe+2MoDnA3cBtwFXAk8c4Lt/R3wR1uVPRXYAry8zb8D+Nz2vLdAAacB1wE/HSg7qE1/ur2Hq9o+/j1wQFu2oNXddev+0oX4fW2b9wB3DLT3noH6rwPWApuAFYP/TVrbb2h9uwP4GJCZ/v98Z395ZKGp9ifA8cDvAfsDm+n+sQMsAZ4MzKf75v4G4N6qehvwfbpv0ntU1enjtHsxcHSSdyZ5/uBwTPN+4OnAIXQf1nOB/1xVv6A7Ori5fn3kc/PgikmeABwJfGkb9vMnwD9v+/NO4HNJ9mvL3g18C9gLmAf8RSs/CvgXrZ9PBk6g+3AfSlX9DFjdtru17XlvjwcOBw6eYJOvbvuyD3AF0HvEWFXXtG3/Q9venlvXSfJi4H10+78fcCNw3lbVXg48F/jdVu/ovm1rtAwLTbU3AG+rqnVVdT/dt99XtqGOX9F9kB1UVQ9W1WVVddcwjVbV94F/DRwKfAO4PclHkuzShmWWAv+hqjZV1d10Q1YnDtnnvej+Ldwy7E5W1d9U1c1V9VBVfZHuW/BhbfGvgAPovi3fV1U/GCh/EvDbdN+Ur6mqobfZ3AzsPU759ry372vv170TLP9GVX2v/Xd8G3Bkkvnb2N/xvBpYXlWXt7bf0tpeMFDn/VV1RwvI79J9CdAMMiw01Q4AvtpOEt8BXEM3JLEv8FngAuC8JDcn+UCSxw7bcFX9z6r6fboPy8XAa+mGPuYATwAuG9juN1v5MDbTDVPt11dxTJKTk1wxsL1n0n0DB/gzuiGnS5KsSfKHrf/fAT5Kd6R1W5JlSX5j2G02c+mGbra2Pe/tTcMur6p72nb335bOTmB/uqOJwbZvp9u3MbcOTG8B9piC7eoRMCw01W4Cjq2qPQdej6uq9VX1q6p6Z1UdDDyPbqjh5Lbe0I8/bt/mL6Q7If1MuvMC9wLPGNjmk6s7+d7bdlVtAf4B+DfDbD/JAcAngdOBp7ShlqvoAoKqurWqXldV+wOvB/4yyUFt2TlV9Ry6oZ+nA28edr/bt/rn0A0rbb0P2/Pe9r3n/+8oIskedCF9M/CLVvyEgbr/ZBvavZnuS8VY20+kOypa37OeZpBhoan2CeCs9oFKkjlJFrfpFyX5nSS70J3g/RXdN3roTkr/1kSNJlmc5MQke6VzGN15kYuq6iG6D++zk/xmqz83ydg49wbgKUmePEm//wx4bZI3J3lKa+NZSbYeS4fupHkBG1u9U+hCa6yvr0oyr81ubnUfSvLcJIe3b/y/oDsR/BA9kjyhXQ32NeASYOU4dbb7vZ3EcUlekGQ3unMXF1XVTVW1ke6D/TVtGPAPgacNrLcBmNfWG88XgFOSHNLOPb0XuLiqbtiOPmqaGBaaav+V7uqWbyW5G7iI7iQqdN8+v0T3YXYN3RU2nx1Y75VJNic5Z5x2N9NdQXNdW/9zwAfr15fp/ie6q2suSnIX8G3gnwJU1Y/pPqCub8NGDxtKqaofAi9ur+uTbAKWMc4Hc1VdDXyY7mhkA/A7wP8aqPJc4OIk97T34o3V3b/wG3ShtpluGOZ24IPj7OuYj7b3cAPwX4AvA8e0cNzaI3lvJ/LXwJl0w0/PAV4zsOx1dEdFtwPPAH44sOw7wBrg1iQ/37rRqvo28Pa2P7fQBc2w55c0Q1Lljx9JkibnkYUkqZdhIUnqZVhIknoZFpKkXo/KB4jts88+tWDBgpnuhiTtUC677LKfV9W4N7M+KsNiwYIFrF69eqa7IUk7lCQ3TrTMYShJUi/DQpLUy7CQJPUyLCRJvQwLSVIvw0KS1MuwkCT1MiwkSb0MC0lSr0flHdyzxXPe/JmZ7oKkHcRlHzy5v9IM8shCktTLsJAk9TIsJEm9DAtJUi/DQpLUy7CQJPUyLCRJvQwLSVIvw0KS1MuwkCT1MiwkSb0MC0lSr5GFRZLHJbkkyf9OsibJO1v5gUkuTrI2yReT7NbKd2/za9vyBQNtvaWVX5vk6FH1WZI0vlEeWdwPvLiqngUcAhyT5Ajgz4Gzq+ogYDNwaqt/KrC5lZ/d6pHkYOBE4BnAMcBfJtllhP2WJG1lZGFRnXva7GPbq4AXA19q5ecCx7fpxW2etvwlSdLKz6uq+6vqp8Ba4LBR9VuS9HAjPWeRZJckVwC3AauAnwB3VNUDrco6YG6bngvcBNCW3wk8ZbB8nHUGt7U0yeokqzdu3DiK3ZGkndZIw6KqHqyqQ4B5dEcDvz3CbS2rqkVVtWjOnDmj2owk7ZSm5WqoqroD+C5wJLBnkrFf6JsHrG/T64H5AG35k4HbB8vHWUeSNA1GeTXUnCR7tunHAy8FrqELjVe2akuAr7XpFW2etvw7VVWt/MR2tdSBwELgklH1W5L0cKP8De79gHPblUuPAc6vqq8nuRo4L8l7gH8EPtXqfwr4bJK1wCa6K6CoqjVJzgeuBh4ATquqB0fYb0nSVkYWFlX1I+DZ45RfzzhXM1XVfcCrJmjrLOCsqe6jJGk43sEtSeplWEiSehkWkqRehoUkqZdhIUnqZVhIknoZFpKkXoaFJKmXYSFJ6mVYSJJ6GRaSpF6GhSSpl2EhSeplWEiSehkWkqRehoUkqZdhIUnqZVhIknoZFpKkXoaFJKmXYSFJ6mVYSJJ6GRaSpF4jC4sk85N8N8nVSdYkeWMrf0eS9UmuaK/jBtZ5S5K1Sa5NcvRA+TGtbG2SM0bVZ0nS+HYdYdsPAG+qqsuTPAm4LMmqtuzsqvrQYOUkBwMnAs8A9ge+neTpbfHHgJcC64BLk6yoqqtH2HdJ0oCRhUVV3QLc0qbvTnINMHeSVRYD51XV/cBPk6wFDmvL1lbV9QBJzmt1DQtJmibTcs4iyQLg2cDFrej0JD9KsjzJXq1sLnDTwGrrWtlE5VtvY2mS1UlWb9y4cYr3QJJ2biMPiyR7AF8G/rSq7gI+DjwNOITuyOPDU7GdqlpWVYuqatGcOXOmoklJUjPKcxYkeSxdUHy+qr4CUFUbBpZ/Evh6m10PzB9YfV4rY5JySdI0GOXVUAE+BVxTVR8ZKN9voNorgKva9ArgxCS7JzkQWAhcAlwKLExyYJLd6E6CrxhVvyVJDzfKI4vnA38AXJnkilb2VuCkJIcABdwAvB6gqtYkOZ/uxPUDwGlV9SBAktOBC4BdgOVVtWaE/ZYkbWWUV0P9AMg4i1ZOss5ZwFnjlK+cbD1J0mh5B7ckqZdhIUnqZVhIknoZFpKkXoaFJKmXYSFJ6mVYSJJ6GRaSpF6GhSSpl2EhSeplWEiSehkWkqRehoUkqZdhIUnqZVhIknoZFpKkXoaFJKmXYSFJ6mVYSJJ6GRaSpF6GhSSpl2EhSeplWEiSeo0sLJLMT/LdJFcnWZPkja187ySrklzX/u7VypPknCRrk/woyaEDbS1p9a9LsmRUfZYkjW+URxYPAG+qqoOBI4DTkhwMnAFcWFULgQvbPMCxwML2Wgp8HLpwAc4EDgcOA84cCxhJ0vQYWVhU1S1VdXmbvhu4BpgLLAbObdXOBY5v04uBz1TnImDPJPsBRwOrqmpTVW0GVgHHjKrfkqSHm5ZzFkkWAM8GLgb2rapb2qJbgX3b9FzgpoHV1rWyicq33sbSJKuTrN64ceOU9l+SdnZDhUWSDyT5jSSPTXJhko1JXjPkunsAXwb+tKruGlxWVQXUNvd6HFW1rKoWVdWiOXPmTEWTkqRm2COLo9oH/cuBG4CDgDf3rZTksXRB8fmq+kor3tCGl2h/b2vl64H5A6vPa2UTlUuSpsmwYbFr+/sy4G+q6s6+FZIE+BRwTVV9ZGDRCmDsiqYlwNcGyk9uV0UdAdzZhqsuAI5Kslc7sX1UK5MkTZNd+6sA8PUkPwbuBf44yRzgvp51ng/8AXBlkita2VuB9wPnJzkVuBE4oS1bCRwHrAW2AKcAVNWmJO8GLm313lVVm4bstyRpCgwVFlV1RpIP0H3bfzDJFrqrlyZb5wdAJlj8knHqF3DaBG0tB5YP01dJ0tQb9gT3E4B/T7v3AdgfWDSqTkmSZpdhz1n8d+CXwPPa/HrgPSPpkSRp1hk2LJ5WVR8AfgVQVVuYeIhJkvQoM2xY/DLJ42n3RCR5GnD/yHolSZpVhr0a6kzgm8D8JJ+nu9LptaPqlCRpdhn2aqhVSS6neyBggDdW1c9H2jNJ0qwxaVgMPia8GXum01OTPHXsQYGSpEe3viOLD0+yrIAXT2FfJEmz1KRhUVUvmq6OSJJmr6HOWSR5HN1NeS+gO6L4PvCJqup75Ick6VFg2KuhPgPcDfxFm/93wGeBV42iU5Kk2WXYsHhm+3nUMd9NcvUoOiRJmn2GvSnv8vbYcACSHA6sHk2XJEmzzbBHFs8BfpjkZ23+qcC1Sa6ke2Ds746kd5KkWWHYsDhmpL2QJM1qw97BfWP7lbr5g+t4U54k7RyGvXT23XTPgvoJ7WGCeFOeJO00hh2GOoHuMeW/HGVnJEmz07BXQ10F7DnKjkiSZq9hjyzeB/xjkqsY+B2LqvpXI+mVJGlWGTYszgX+HLgSeGh03ZEkzUbDhsWWqjpnpD2RJM1aw56z+H6S9yU5MsmhY6/JVkiyPMltbehqrOwdSdYnuaK9jhtY9pYka5Ncm+TogfJjWtnaJGds8x5Kkh6xYY8snt3+HjFQ1nfp7KeBj9I9hHDQ2VX1ocGCJAcDJwLPAPYHvp3k6W3xx4CXAuuAS5OsqCqfSyVJ02jYm/K2+Xctqup7SRYMWX0xcF5V3Q/8NMla4LC2bG1VXQ+Q5LxW17CQpGk07JEFSV5G983/cWNlVfWu7djm6UlOpnsQ4ZuqajMwF7hooM66VgZw01blh2/HNiVJj8BQ5yySfAL4t8CfAKH7HYsDtmN7HweeBhxC93vek/1s6zZJsjTJ6iSrN27cOFXNSpIY/gT386rqZGBzVb0TOBJ4es86D1NVG6rqwap6CPgkvx5qWk/33Kkx81rZROXjtb2sqhZV1aI5c+Zsa9ckSZMYNizGfj51S5L9gQeA/bZ1Y0kG13kF3Z3hACuAE5PsnuRAYCFwCXApsDDJgUl2ozsJvmJbtytJemSGPWfxP5LsCXwQuJzuSqhPTrZCki8ALwT2SbIOOBN4YZJD2vo3AK8HqKo1Sc6nO3H9AHBaVT3Y2jkduADYBVheVWu2ZQclSY/csGHxY+DBqvpyu8z1UOBvJ1uhqk4ap/hTk9Q/CzhrnPKVwMoh+ylJGoFhh6HeXlV3J3kB3b0Vf0V3slqStBMYNiwebH9fBnyyqr4B7DaaLkmSZpthw2J9kv9Gd/nsyiS7b8O6kqQd3LAf+CfQnWQ+uqruAPYG3jyyXkmSZpVhH/exBfjKwPwtdDfVSZJ2Ag4lSZJ6GRaSpF6GhSSpl2EhSeplWEiSehkWkqRehoUkqZdhIUnqZVhIknoZFpKkXoaFJKmXYSFJ6mVYSJJ6GRaSpF6GhSSpl2EhSeplWEiSehkWkqReIwuLJMuT3JbkqoGyvZOsSnJd+7tXK0+Sc5KsTfKjJIcOrLOk1b8uyZJR9VeSNLFRHll8Gjhmq7IzgAuraiFwYZsHOBZY2F5LgY9DFy7AmcDhwGHAmWMBI0maPiMLi6r6HrBpq+LFwLlt+lzg+IHyz1TnImDPJPsBRwOrqmpTVW0GVvHwAJIkjdh0n7PYt6puadO3Avu26bnATQP11rWyicofJsnSJKuTrN64cePU9lqSdnIzdoK7qgqoKWxvWVUtqqpFc+bMmapmJUlMf1hsaMNLtL+3tfL1wPyBevNa2UTlkqRpNN1hsQIYu6JpCfC1gfKT21VRRwB3tuGqC4CjkuzVTmwf1cokSdNo11E1nOQLwAuBfZKso7uq6f3A+UlOBW4ETmjVVwLHAWuBLcApAFW1Kcm7gUtbvXdV1dYnzSVJIzaysKiqkyZY9JJx6hZw2gTtLAeWT2HXJEnbyDu4JUm9DAtJUi/DQpLUy7CQJPUyLCRJvQwLSVIvw0KS1MuwkCT1MiwkSb0MC0lSL8NCktTLsJAk9TIsJEm9DAtJUi/DQpLUy7CQJPUyLCRJvQwLSVIvw0KS1MuwkCT1MiwkSb0MC0lSL8NCktRrRsIiyQ1JrkxyRZLVrWzvJKuSXNf+7tXKk+ScJGuT/CjJoTPRZ0namc3kkcWLquqQqlrU5s8ALqyqhcCFbR7gWGBhey0FPj7tPZWkndxsGoZaDJzbps8Fjh8o/0x1LgL2TLLfTHRQknZWMxUWBXwryWVJlrayfavqljZ9K7Bvm54L3DSw7rpW9v9JsjTJ6iSrN27cOKp+S9JOadcZ2u4Lqmp9kt8EViX58eDCqqoktS0NVtUyYBnAokWLtmldSdLkZuTIoqrWt7+3AV8FDgM2jA0vtb+3terrgfkDq89rZZKkaTLtYZHkiUmeNDYNHAVcBawAlrRqS4CvtekVwMntqqgjgDsHhqskSdNgJoah9gW+mmRs+39dVd9McilwfpJTgRuBE1r9lcBxwFpgC3DK9HdZknZu0x4WVXU98Kxxym8HXjJOeQGnTUPXJEkTmE2XzkqSZinDQpLUy7CQJPUyLCRJvQwLSVIvw0KS1MuwkCT1MiwkSb0MC0lSL8NCktTLsJAk9TIsJEm9DAtJUi/DQpLUy7CQJPUyLCRJvQwLSVIvw0KS1MuwkCT1MiwkSb0MC0lSL8NCktTLsJAk9dphwiLJMUmuTbI2yRkz3R9J2pnsEGGRZBfgY8CxwMHASUkOntleSdLOY4cIC+AwYG1VXV9VvwTOAxbPcJ8kaaex60x3YEhzgZsG5tcBhw9WSLIUWNpm70ly7TT1TdpW+wA/n+lOaHbJh5bMdBcADphowY4SFr2qahmwbKb7IfVJsrqqFs10P6RtsaMMQ60H5g/Mz2tlkqRpsKOExaXAwiQHJtkNOBFYMcN9kqSdxg4xDFVVDyQ5HbgA2AVYXlVrZrhb0vZyuFQ7nFTVTPdBkjTL7SjDUJKkGWRYSJJ6GRbSNPGRNdqRec5CmgbtkTX/B3gp3U2llwInVdXVM9oxaUgeWUjTw0fWaIdmWEjTY7xH1sydob5I28ywkCT1Miyk6eEja7RDMyyk6eEja7RD2yEe9yHt6HxkjXZ0XjorSerlMJQkqZdhIUnqZVhIknoZFpKkXoaFJKmXYSFJ6mVYSJJ6/V99Uy281cYXZgAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HZCQTgVCGeln"
      },
      "source": [
        "Observation: Target labels in training set are almost evenly distributed. In test set, there are only 0 labels, meaning that all texts in test.csv are non-disaster tweets."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PnqLlSLhG7BT"
      },
      "source": [
        "## 1.3 Character & Word Count"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 356
        },
        "id": "O-LWCJpVHB7r",
        "outputId": "ea5eb469-fce1-4d10-f672-eb88e4cb020f"
      },
      "source": [
        "# Character count in training set\r\n",
        "fig,(ax1,ax2) = plt.subplots(1,2,figsize=(10,5))\r\n",
        "tweet_len = df_train[df_train['target']==1]['text'].str.len()\r\n",
        "ax1.hist(tweet_len,color='red')\r\n",
        "ax1.set_title('disaster tweets')\r\n",
        "tweet_len = df_train[df_train['target']==0]['text'].str.len()\r\n",
        "ax2.hist(tweet_len,color='green')\r\n",
        "ax2.set_title('Not disaster tweets')\r\n",
        "fig.suptitle('Characters in training tweets')\r\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlYAAAFTCAYAAAD7gEIxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deZxlZX3n8c9Xml2lEVqEBm0QxGAyEdJBjOuIUUEiJIOKMYoGhySDccFE0cxEdBKVxIg6GgiKiksEQzAiIVHCJhpAG0Rl0dCydbcNtOy4gvzmj/MUXoqqrmr63Fpuf96v133VWZ5zznPOrXrqe5+z3FQVkiRJWn8Pm+0KSJIkjQqDlSRJUk8MVpIkST0xWEmSJPXEYCVJktQTg5UkSVJPDFbSLEpydJJPz3Y9ZlqSu5PsMovbvyLJs/suK0kGK2nIkvx+kmUtTKxO8m9Jnj7b9RqU5FVJvjpT26uqh1fVNeu6XJIlSSrJgvXc/pOq6ry+y66LmT7mA9utJLvO9HalDYXBShqiJEcC7wfeBWwHPBb4e+DAIWxrvcLGfN32eHOpLpI2PAYraUiSbAW8Eziiqk6rqh9V1T1V9cWq+vOBopsk+WSSu9ppp6UD6zgqyffbvCuT/O7AvFcl+VqSY5PcAhyd5PFJzklyS5IfJvlMkoUDy+yU5LQka1qZDyX5FeB44KmtV+32VnbTJO9NckOSm5Icn2TzNu/ZSVYmeUuSG4GPJ9k2yRlJbk9ya5ILkkzYxgz2miT5RJIPJ/nXtp8XJ3n8JIf1K+3n7a2uT32Ix+G6JM9tw0cn+dxa3oN1KbtXkm+2ef+U5JQkfzXB/j/omCfZuf18WCvzkSQ3DyzzqSRvaMNbJTmx9YCuSvJXSTYaKPuHSa5KcluSLyV5XJs+dvy+1bb70nV53yRNzT8eaXieCmwGfH6Kci8CTgYWAqcDHxqY933gGcBWwDuATyfZfmD+U4Br6HrD/hoI8G5gB+BXgJ2AowHaP94zgOuBJcBi4OSqugr4Y+DCdopuLIC8B3gC8GRg11b+Lwe2/RjgUcDjgMOBNwErgUWtPm8DpvudWYe0/dsaWN72ZSLPbD8XtrpeuK7HYRJrew+mVTbJJnTv9Sfojstngd+daAUTHfOquha4E9hzYF/vbiEM4FnA+W34E8C9dO/LnsDzgNe0ehxId+x/j+69uKDVhaoaO36/3rZ7Cuv3vkkax2AlDc82wA+r6t4pyn21qs6sql8AnwJ+fWxGVf1TVf2gqu5r/wSvBvYeWPYHVfX/qureqvpJVS2vqrOq6mdVtQZ4H90/ZNpyOwB/3nrPflpVE17jkyR0YemNVXVrVd1FdzrzkIFi9wFvb9v6CXAPsD3wuNYzd0FN/8tIP19VX2/H6jN0YW5drMtxmMik78E6lN0HWAB8sO3/acDX13E/zgeeleQxbfzUNr4z8Ei6nqbtgP2BN7T38WbgWH753vwx8O6quqodz3cBTx7rtZrA+rxvksYxWEnDcwuwbaa+5ufGgeEfA5uNLZPklUkua6dpbgd+Fdh2oPyKwRUl2S7Jye300J3ApwfK7wRcP42gB13vxRbAJQPb/vc2fcyaqvrpwPjf0vU2fTnJNUmOmsZ2xow/Bg9fh2Vh3Y7DdLa/2Vret8nK7gCsGhdKHlCvaTgfeDZdb9VXgPPoAuGzgAuq6j66HsKNgdUD780/AI9u63gc8IGBebfS9eAtnmSb6/O+SRrHYCUNz4XAz4CDHsrCrYfhI8BrgW3aKbrL6f5Jjhnfs/CuNu3XquqRwB8MlF8BPHaSwDB+PT8EfgI8qZ2mWlhVW1XVwydbpqruqqo3VdUudKfLjkyy73T3d5om60lZl+MwLKuBxa23b8xOayk/0b6cT3fq99lt+KvA03jgacAVdL9X2w68N4+sqicNzP+jgXkLq2rzqvrPCSsxM++btMEwWElDUlV30F2T9OEkByXZIsnGSfZL8jfTWMWWdP981wAkeTVdj9XaPAK4G7gjyWJg8CL5r9P9839Pki2TbJbkaW3eTcCO7TohWs/IR4Bjkzy6bX9xkudPtuEkByTZtQWLO4Bf0J0u7NOats6pnoG1tuMwLBfS7fNrkyxo1zrtvZbyDzjmAFV1NV2g/QPg/Kq6s5X7H7RgVVWrgS8Df5fkkUke1i7WHzvVeTzw1iRPgvsvdH/xuO3ef/xm6H2TNhgGK2mIqurvgCOB/00XClbQ9UD9yzSWvRL4O7p/2DcBvwZ8bYrF3gHsRfcP8l+B0wbW9wvgd+gueL6B7oLll7bZ5wBXADcm+WGb9ha6U0QXtdNp/wHsvpZt79bK3N3q/PdVde5U+7kuqurHdBenf62d6tpnkqKTHodhqaqf010wfhhwO104OoOud2kiEx1z6ALULVW1YmA8wKUDZV4JbAJcCdxGdy3W9q0enweOAU5u79vlwH4Dyx4NnNSO30uYgfdN2pDEaxQlaTiSXAwcX1Ufn+26SJoZ9lhJUk+SPCvJY9qpwEOB/0Z30b+kDYRPKJak/uwOfI7u+rhrgIPbNVGSNhCeCpQkSeqJpwIlSZJ6YrCSJEnqicFKkiSpJwYrSZKknhisJEmSemKwkiRJ6onBaoQl+USSv2rDz0jyvdmukyQNS5Lrkjy3Db8tyUdnu07a8BisNhBVdUFVre173tbLYIiby+ucxjafnWTlTG5TGkUt5NycZMuBaa9Jct40lz8vyWse6var6l1V9ZCXn8pgiJvL65zGNo9O8umZ3OaoM1hpTkjitwBIo2cj4PWzXYm5Jh3//44o39gRkmTPJJcmuSvJKcBmA/Me0BOT5C1JVrWy30uyb5u+d5IL033z/eokH0qySZuXJMe2T6F3JvlOkl9NcjjwcuDNSe5O8sVWfock/5xkTZJrk7xuYPtHJzk1yaeT3Am8aty+PGidSV49tu5W5uok/zQwviLJk9vwE5OcleTWtn8vGSi3aZL3JrkhyU1Jjk+yeftk/W/ADm2bd7d92DvJsrbPNyV5Xw9vl7Qh+Fvgz5IsnGhmkt9K8o0kd7Sfv9Wm/zXwDOBD7e/wQ5Ms/4ok1ye5JclfjJt3f09Mks1aW3NLa9u+kWS7Nu/VSa5qbeE1Sf5oYB3bJjmjLXNrkguSPCzJp4DHAl9s9XtzK79Pkv9s5b+V5NkD6zovyV8n+RrwY2CXcfV90DqTnJTkTW3+4iSV5Ig2/vhWp4e18QOSXNa2/Z9J/tvAuidsi5O8AHgb8NK2zW+16a9qx+KuVv7lk7/FepCq8jUCL2AT4HrgjcDGwMHAPcBftfnPBla24d2BFcAObXwJ8Pg2/BvAPnTfI7kEuAp4Q5v3fOASYCEQ4FeA7du8T4xtq40/rJX9y1a3Xei+O+35bf7RrX4HtbKbT7BP49e5C3B7K79D29+VA/Nua/O2bPv36rYfewI/BPZoZY8FTgceBTwC+CLw7vHHaWC7FwKvaMMPB/aZ7ffbl6+5/gKuA54LnDbQDr0GOK8NP6r9zb6i/Z2+rI1v0+afB7xmLevfA7gbeCawKfA+4F7guW3+0cCn2/Aftb/zLeh60X4DeGSb90Lg8a1NexZd6NmrzXs3cDxdm7oxXdjL4P4N1GcxcAuwf2uHfruNLxrYnxuAJ7X93XiyYzYw/ofAF9vw7wPfB04ZmPeFNrwncDPwlLZ/h7Z1bcr02uJPD2xzS+BOYPc2vj3wpNn+fZpPL3usRsc+dH/476+qe6rqVOAbk5T9Bd0f3B5JNq6q66rq+wBVdUlVXVRV91bVdcA/0DU20AWhRwBPpGtcrqrJv2D2N+kalHdW1c+r6hrgI8AhA2UurKp/qar7quonU+1gW8ddwJPpGtMvAT9I8sRWxwuq6j7gAOC6qvp4249vAv8MvDhJgMOBN1bVrVV1F/CucfUa7x5g1yTbVtXdVXXRVHWVdL+/BP40yaJx018IXF1Vn2p/p58Fvgv8zjTXezBwRlV9pap+Bvwf4L5Jyt4DbAPsWlW/aO3cnQBV9a9V9f3qnA98mS5AjS23PfC41q5eUC1tTOAPgDOr6szWpp0FLKMLWmM+UVVXtP29Zxr7eD7w9NYr9Uzgb4CntXnPavOha9P+oaoubvt3EvAzuv8L02mLx7sP+NUkm1fV6qq6Yhp1VWOwGh07AKvG/dFfP1HBqloOvIHuk8rNSU5OsgNAkie0ru8b2ym6dwHbtuXOAT4EfLgtd0KSR05Sn8fRnVK7fexF1+W83UCZFQ9hP8+n61V6Zhs+j66BGWxkHgc8Zdy2Xw48BlhE96n1koF5/96mT+Yw4AnAd9sphAMeQr2lDVJVXQ6cARw1btZYr/Og6+l6fqZjBwbakKr6EV0P0UQ+RfdB7OQkP0jyN0k2BkiyX5KL2mm12+mC0LZtub8FlgNfbqfGxu/DoMfRfXgbbHeeThfMxqxTm9c+8P6I7sPkM+iO4w+S7M6D27w3jdv2TnTHaDpt8eA2fwS8FPhjYHWSf20fXjVNBqvRsRpY3Hpkxjx2ssJV9Y9V9XS6P7oCjmmzjqP71LhbVT2S7g8wA8t9sKp+g64b/gnAn4/NGreJFcC1VbVw4PWIqhr89DbZJ7+1zR8LVs9ow+fz4GC1Ajh/3LYfXlV/QndK8Cd0Xdtj87aqqodPts2qurqqXgY8mu44nZqBO50kTentwP/kgaHpB3Ttz6DHAqva8FTtw2q68ABAki3oeqUepPU2vaOq9gB+i65X+5VJNqXrzX4vsF1VLQTOpLV5VXVXVb2pqnYBXgQcmXY96gT1WwF8aly7s2VVvWewKlPs02Rt3sHAJlW1qo0fCmwNXDaw7b8et+0tWi/gVG3xRG3el6rqt+lC4Xfperg0TQar0XEh3fUFr0uycZLfA/aeqGCS3ZM8pzUqP6ULGmNd6I+gO79+d/uU8icDy/1mkqe0T3o/asuOLXcTD7wY8+vAXekukt88yUbpLnT/zXXYp/HrhK5R+e9012StBC4AXkDXoH6zlTkDeEK6C1s3bq/fTPIr7VThR4Bjkzy67dfiJM8f2OY2SbYa2O8/SLKoLXt7mzzZKQdJ47Re8lOA1w1MPpPu7/T3kyxI8lK6D2xntPkT/f0POhU4IMnT091g804m+Z+W5L8n+bUkG9G1b/fQ/Q1vQndZxBrg3iT7Ac8bWO6AJLu2D6x30F1GMVmb92ngd5I8v7V3m6W7aWjHKQ7PoMnavNcCX2nj57Xxr1bVL9q0jwB/3NrnJNkyyQuTPIKp2+KbgCX55UXw2yU5sH14/BnddWy2d+vAYDUiqurnwO/R3V13K11X7mmTFN8UeA9d782NdD0xb23z/ozuIsm76P5YTxlY7pFt2m10Xfa30HWVA5xId83W7Un+pf3BH0DXhX1t29ZHga2Yvgess+3nf9H9oV/Qxu+kuxDza2ONTLtu6nl01xD8oO3jMW2/Ad5C171/UTvd+R90F/RTVd8FPgtc07a7A11wuyLJ3cAHgEOmc02YpAd4J92F0QBU1S10bcSb6NqSNwMHVNUPW5EPAAcnuS3JB8evrF33cwTwj3S9V7cBkz2D7jF0QexOuhtyzqfrXbqLLux9ri3/+3Q3tozZja59uJvuw+vfV9W5bd67gf/d2ok/q6oVwIF0vfxr6HqK/px1+z/7gHW2aefTfeAdC1ZfpbucYWycqlpG1yP4obYfy2l3Wk+jLR67s/qWJJe2+h5J13beSnc24P4P2Jra2N0NkiRJWk/2WEmSJPXEYCVJktQTg5UkSVJPDFaSJEk9MVhJkiT1ZMFsVwBg2223rSVLlsx2NSTNoEsuueSHVbW2J97PC7Zf0oZnbe3XnAhWS5YsYdmyZbNdDUkzKMmEX7k039h+SRuetbVfngqUJEnqicFKkiSpJwYrSZKknhisJEmSemKwkiRJ6onBSpIkqScGK0mSpJ4YrCRJknpisJIkSeqJwUqSJKknBitJkqSezInvCpQ0IBneuquGt25JG7y8Y3jtV719frRf9lhJkiT1xGAlSZLUE4OVJElSTwxWkiRJPTFYSZIk9cRgJUmS1BODlSRJUk8MVpIkST0xWEmSJPXEYCVJktQTg5UkSVJPDFaSJEk9MVhJkiT1xGAlSZLUE4OVJElSTwxWkiRJPTFYSZIk9cRgJUmS1BODlSRJUk8MVpIkST0xWEmSJPXEYCVJktQTg5UkSVJPDFaSJEk9MVhJkiT1xGAlSZLUE4OVpHkvyceS3Jzk8oFpj0pyVpKr28+t2/Qk+WCS5Um+nWSvgWUObeWvTnLobOyLpPnNYCVpFHwCeMG4aUcBZ1fVbsDZbRxgP2C39jocOA66IAa8HXgKsDfw9rEwJknTZbCSNO9V1VeAW8dNPhA4qQ2fBBw0MP2T1bkIWJhke+D5wFlVdWtV3QacxYPDmiStlcFK0qjarqpWt+Ebge3a8GJgxUC5lW3aZNMladoMVpJGXlUVUH2tL8nhSZYlWbZmzZq+VitpBBisJI2qm9opPtrPm9v0VcBOA+V2bNMmm/4gVXVCVS2tqqWLFi3qveKS5i+DlaRRdTowdmffocAXBqa/st0duA9wRztl+CXgeUm2bhetP69Nk6Rpm1awSvLGJFckuTzJZ5NslmTnJBe3W5ZPSbJJK7tpG1/e5i8Z5g5IUpLPAhcCuydZmeQw4D3Abye5GnhuGwc4E7gGWA58BPhfAFV1K/B/gW+01zvbNEmatgVTFUiyGHgdsEdV/STJ54BDgP2BY6vq5CTHA4fR3bZ8GHBbVe2a5BDgGOClQ9sDSRu8qnrZJLP2naBsAUdMsp6PAR/rsWqSNjDTPRW4ANg8yQJgC2A18Bzg1DZ//K3MY7c4nwrsmyT9VFeSJGnumjJYVdUq4L3ADXSB6g7gEuD2qrq3FRu8Lfn+W5bb/DuAbfqttiRJ0twzZbBqF3EeCOwM7ABsSQ8PzfN2ZUmSNGqmcyrwucC1VbWmqu4BTgOeRve04rFrtAZvS77/luU2fyvglvEr9XZlSZI0aqYTrG4A9kmyRbtWal/gSuBc4OBWZvytzGO3OB8MnNMuFpUkSRpp07nG6mK6i9AvBb7TljkBeAtwZJLldNdQndgWORHYpk0/kl9+8akkSdJIm/JxCwBV9Xa6b30fdA3dN8CPL/tT4MXrXzVJkqT5xSevS5Ik9cRgJUmS1BODlSRJUk8MVpIkST0xWEmSJPXEYCVJktQTg5UkSVJPDFaSJEk9MVhJkiT1xGAlSZLUE4OVJElSTwxWkiRJPTFYSZIk9cRgJUmS1BODlSRJUk8MVpIkST0xWEmSJPXEYCVJktQTg5UkSVJPDFaSJEk9MVhJkiT1xGAlSZLUE4OVJElSTwxWkiRJPTFYSZIk9cRgJUmS1BODlSRJUk8MVpIkST0xWEmSJPXEYCVJktQTg5WkkZbkjUmuSHJ5ks8m2SzJzkkuTrI8ySlJNmllN23jy9v8JbNbe0nzjcFK0shKshh4HbC0qn4V2Ag4BDgGOLaqdgVuAw5rixwG3NamH9vKSdK0GawkjboFwOZJFgBbAKuB5wCntvknAQe14QPbOG3+vkkyg3WVNM8ZrCSNrKpaBbwXuIEuUN0BXALcXlX3tmIrgcVteDGwoi17byu/zUzWWdL8ZrCSNLKSbE3XC7UzsAOwJfCCHtZ7eJJlSZatWbNmfVcnaYQYrCSNsucC11bVmqq6BzgNeBqwsJ0aBNgRWNWGVwE7AbT5WwG3jF9pVZ1QVUuraumiRYuGvQ+S5hGDlaRRdgOwT5It2rVS+wJXAucCB7cyhwJfaMOnt3Ha/HOqqmawvpLmOYOVpJFVVRfTXYR+KfAdujbvBOAtwJFJltNdQ3ViW+REYJs2/UjgqBmvtKR5bcHURSRp/qqqtwNvHzf5GmDvCcr+FHjxTNRL0miyx0qSJKknBitJkqSeGKwkSZJ6YrCSJEnqicFKkiSpJwYrSZKknhisJEmSejKtYJVkYZJTk3w3yVVJnprkUUnOSnJ1+7l1K5skH0yyPMm3k+w13F2QJEmaG6bbY/UB4N+r6onArwNX0T2R+Oyq2g04m18+oXg/YLf2Ohw4rtcaS5IkzVFTBqskWwHPpH3lQ1X9vKpup/vG+JNasZOAg9rwgcAnq3MR3Zedbt97zSVJkuaY6fRY7QysAT6e5JtJPppkS2C7qlrdytwIbNeGFwMrBpZf2aZJkiSNtOkEqwXAXsBxVbUn8CPGfTFp+/b3dfoG+CSHJ1mWZNmaNWvWZVFJkqQ5aTrBaiWwsn1LPHTfFL8XcNPYKb728+Y2fxWw08DyO7ZpD1BVJ1TV0qpaumjRoodaf0mSpDljymBVVTcCK5Ls3ibtC1wJnA4c2qYdCnyhDZ8OvLLdHbgPcMfAKUNJkqSRtWCa5f4U+EySTYBrgFfThbLPJTkMuB54SSt7JrA/sBz4cSsrSZI08qYVrKrqMmDpBLP2naBsAUesZ70kSZLmHZ+8LkmS1BODlSRJUk8MVpIkST0xWEmSJPXEYCVJktQTg5UkSVJPDFaSJEk9MVhJkiT1xGAlSZLUE4OVJElSTwxWkiRJPTFYSZIk9cRgJUmS1BODlSRJUk8MVpIkST0xWEmSJPXEYCVJktQTg5UkSVJPDFaSJEk9MVhJkiT1xGAlaaQlWZjk1CTfTXJVkqcmeVSSs5Jc3X5u3comyQeTLE/y7SR7zXb9Jc0vBitJo+4DwL9X1ROBXweuAo4Czq6q3YCz2zjAfsBu7XU4cNzMV1fSfGawkjSykmwFPBM4EaCqfl5VtwMHAie1YicBB7XhA4FPVuciYGGS7We42pLmMYOVpFG2M7AG+HiSbyb5aJItge2qanUrcyOwXRteDKwYWH5lmyZJ02KwkjTKFgB7AcdV1Z7Aj/jlaT8AqqqAWpeVJjk8ybIky9asWdNbZSXNfwYrSaNsJbCyqi5u46fSBa2bxk7xtZ83t/mrgJ0Glt+xTXuAqjqhqpZW1dJFixYNrfKS5h+DlaSRVVU3AiuS7N4m7QtcCZwOHNqmHQp8oQ2fDryy3R24D3DHwClDSZrSgtmugCQN2Z8Cn0myCXAN8Gq6D5WfS3IYcD3wklb2TGB/YDnw41ZWkqbNYCVppFXVZcDSCWbtO0HZAo4YeqUkjSxPBUqSJPXEYCVJktQTTwVKkrQByTsy21UYafZYSZIk9cRgJUmS1BODlSRJUk8MVpIkST0xWEmSJPXEuwI1uzLEu1Nqnb5XV5Kk9WaPlSRJUk8MVpIkST3xVKAkSXOMD/Gcv+yxkiRJ6onBSpIkqSeeCpQeimHezShJmrcMVtKGxMdbSNJQeSpQkiSpJ9PusUqyEbAMWFVVByTZGTgZ2Aa4BHhFVf08yabAJ4HfAG4BXlpV1/Vec0mSZpF37mki69Jj9XrgqoHxY4Bjq2pX4DbgsDb9MOC2Nv3YVk6SJGnkTStYJdkReCHw0TYe4DnAqa3IScBBbfjANk6bv28rL0mSNNKm22P1fuDNwH1tfBvg9qq6t42vBBa34cXACoA2/45WXpIkaaRNGaySHADcXFWX9LnhJIcnWZZk2Zo1a/pctSRJ0qyYTo/V04AXJbmO7mL15wAfABYmGbv4fUdgVRteBewE0OZvRXcR+wNU1QlVtbSqli5atGi9dkKSJGkumDJYVdVbq2rHqloCHAKcU1UvB84FDm7FDgW+0IZPb+O0+edU+YAbSZI0+tbnOVZvAY5MspzuGqoT2/QTgW3a9COBo9avipIkSfPDOj15varOA85rw9cAe09Q5qfAi3uomyRJ68VnTWmm+eR1SZKknhisJEmSeuKXMGt0+VxaSdIMs8dKkiSpJwYrSZKknhisJEmSemKwkiRJ6onBSpIkqScGK0mSpJ4YrCRJknpisJIkSeqJwUrSyEuyUZJvJjmjje+c5OIky5OckmSTNn3TNr68zV8ym/WWNP8YrCRtCF4PXDUwfgxwbFXtCtwGHNamHwbc1qYf28pJ0rQZrCSNtCQ7Ai8EPtrGAzwHOLUVOQk4qA0f2MZp8/dt5SVpWgxWkkbd+4E3A/e18W2A26vq3ja+EljchhcDKwDa/DtaeUmaFoOVpJGV5ADg5qq6pOf1Hp5kWZJla9as6XPVkuY5g5WkUfY04EVJrgNOpjsF+AFgYZIFrcyOwKo2vArYCaDN3wq4ZfxKq+qEqlpaVUsXLVo03D2QNK8YrCSNrKp6a1XtWFVLgEOAc6rq5cC5wMGt2KHAF9rw6W2cNv+cqqoZrLKkec5gJWlD9BbgyCTL6a6hOrFNPxHYpk0/EjhqluonaZ5aMHURSZr/quo84Lw2fA2w9wRlfgq8eEYrJvIOb7zU6LDHSpIkqScGK0mSpJ54KlBT8/mIkiRNiz1WkiRJPTFYSZIk9cRgJUmS1BOvsZIkTclHIkjTY4+VJElSTwxWkiRJPTFYSZIk9cRgJUmS1BODlSRJUk8MVpIkST0xWEmSJPXEYCVJktQTg5UkSVJPDFaSJEk9MVhJkiT1xGAlSZLUE4OVJElSTwxWkiRJPTFYSZIk9cRgJUmS1BODlSRJUk8MVpIkST0xWEmSJPXEYCVJktSTKYNVkp2SnJvkyiRXJHl9m/6oJGclubr93LpNT5IPJlme5NtJ9hr2TkiSJM0F0+mxuhd4U1XtAewDHJFkD+Ao4Oyq2g04u40D7Afs1l6HA8f1XmtJkqQ5aMpgVVWrq+rSNnwXcBWwGDgQOKkVOwk4qA0fCHyyOhcBC5Ns33vNJUmS5ph1usYqyRJgT+BiYLuqWt1m3Qhs14YXAysGFlvZpo1f1+FJliVZtmbNmnWstiRJ0twz7WCV5OHAPwNvqKo7B+dVVQG1LhuuqhOqamlVLV20aNG6LCpJkjQnLZhOoSQb04Wqz1TVaW3yTUm2r6rV7VTfzW36KmCngcV3bNMkSUOUd2S2qyBt8KZzV2CAE4Grqup9A7NOBw5tw4cCXxiY/sp2d+A+wB0DpwwlSZJG1nR6rJ4GvAL4TpLL2rS3Ae8BPpfkMOB64CVt3pnA/sBy4MfAq3utsSRJ0hw1ZbCqqq8Ck/Uv7ztB+QKOWM96SZIkzTs+eV3SyPIBx5JmmsFK0ijzAceSZpTBStLI8gHHkmaawUrSBqHPBxxL0mQMVpJGXt8POPabIyRNxmAlaaSt7QHHbdu0nwgAAAjsSURBVP46P+DYb46QNBmD1ahIhveS5ikfcCxppk3rK20kaZ7yAceSZpTBStLI8gHHkmaapwIlSZJ6YrCSJEnqicFKkiSpJwYrSZKknhisJEmSemKwkiRJ6onBSpIkqScGK0mSpJ74gNCZ5NfDSJI00uyxkiRJ6onBSpIkqScGK0mSpJ4YrCRJknpisJIkSeqJwUqSJKknBitJkqSeGKwkSZJ6YrCSJEnqicFKkiSpJwYrSZKknhisJEmSeuKXMEuSpDkv78hQ119vr17WY4+VJElST+yxkqQZNOxP3ZJmlz1WkiRJPTFYSZIk9cRTgePFbnpJkvTQ2GMlSZLUE4OVJElSTwxWkiRJPTFYSZIk9cRgJUmS1JP5eVegd+5JkqQ5yB4rSZKknhisJEmSemKwkiRJ6onBSpIkqSdDCVZJXpDke0mWJzlqGNuQpGGxDZP0UPUerJJsBHwY2A/YA3hZkj363o4kDYNtmKT1MYweq72B5VV1TVX9HDgZOHAI25GkYbANk/SQDSNYLQZWDIyvbNMkaT6wDZP0kM3aA0KTHA4c3kbvTvK9IWxmW+CHQ1jvfKsDWI+5VgcYtXqs+4N7H7fe25wlE7RftzA33suJzJXfs/Gs17qbq3Wbq/WCdahbjl6nNmzS9msYwWoVsNPA+I5t2gNU1QnACUPY/v2SLKuqpcPcxnyog/WYe3WwHnPalG3Y+PZrLh/DuVo367Xu5mrd5mq9YHbqNoxTgd8Adkuyc5JNgEOA04ewHUkaBtswSQ9Z7z1WVXVvktcCXwI2Aj5WVVf0vR1JGgbbMEnrYyjXWFXVmcCZw1j3OhrqqcZpmgt1AOsxaC7UAazHnPUQ2rC5fAznat2s17qbq3Wbq/WCWahbqmqmtylJkjSS/EobSZKknoxEsEqyU5Jzk1yZ5Iokr2/TH5XkrCRXt59bz1B9NkryzSRntPGdk1zcvh7jlHZB7DC3vzDJqUm+m+SqJE+djWOR5I3t/bg8yWeTbDYTxyLJx5LcnOTygWkT7n86H2z1+XaSvYZcj79t78u3k3w+ycKBeW9t9fhekucPqw4D896UpJJs28aHdixG2Vz5+pu1tINHJ1mV5LL22n+W6nddku+0Oixr02aljR6o0+4Dx+WyJHcmecNsHbO50nZNs14TtmVJliT5ycCxO36G6zXpezeMNnZCVTXvX8D2wF5t+BHAf9F9FcXfAEe16UcBx8xQfY4E/hE4o41/DjikDR8P/MmQt38S8Jo2vAmwcKaPBd0DFa8FNh84Bq+aiWMBPBPYC7h8YNqE+w/sD/wbEGAf4OIh1+N5wII2fMxAPfYAvgVsCuwMfB/YaBh1aNN3ors4+3pg22Efi1F90V3c/n1gl/a39i1gj1mqy2Tt4NHAn82BY3Xd2O/awLRZaaPX8l7eSPd8olk5ZnOl7ZpmvSZry5aMb29muF4TvnfDamMneo1Ej1VVra6qS9vwXcBVdP/YD6QLGbSfBw27Lkl2BF4IfLSNB3gOcOpM1CPJVnS/bCcCVNXPq+p2ZuFY0N0csXmSBcAWwGpm4FhU1VeAW8dNnmz/DwQ+WZ2LgIVJth9WParqy1V1bxu9iO4ZSWP1OLmqflZV1wLL6b5apfc6NMcCbwYGL7Ic2rEYYXPm62/W0g7OZbPRLk1mX+D7VXX9bFVgrrRd06nXWtqyGbOW9m0iQ2ljJzISwWpQkiXAnsDFwHZVtbrNuhHYbgaq8H66f1j3tfFtgNsHfgGH/fUYOwNrgI+nOx350SRbMsPHoqpWAe8FbqALVHcAlzCzx2LQZPs/m19f8od0nzhntB5JDgRWVdW3xs3yq1zW3Zw8ZuPaQYDXtlM2H5vp020DCvhykkvSPbkeZqeNnswhwGcHxufCMYO52XaNN9iWAezc/v+cn+QZs1Cfid67GTteIxWskjwc+GfgDVV15+C86voCh3oLZJIDgJur6pJhbmcKC+i6Ro+rqj2BH9F1H99vho7F1nSfEHYGdgC2BF4wzG1O10zs/1SS/AVwL/CZGd7uFsDbgL+cye1q5kzQDh4HPB54Mt2HnL+bpao9var2AvYDjkjyzMGZs/l3me5azxcB/9QmzZVj9gBzoe0ab4K2bDXw2Pb/50jgH5M8cgarNOvv3cgEqyQb0zUmn6mq09rkm8a6RtvPm4dcjacBL0pyHd0pgecAH6Droh17ZtiEX/HTo5XAyqoa+6R6Kl3Qmulj8Vzg2qpaU1X3AKfRHZ+ZPBaDJtv/aX0FU5+SvAo4AHh5ayhnsh6Ppwu732q/pzsClyZ5zAzWYZTMqWM2UTtYVTdV1S+q6j7gIwzp9MdUWi82VXUz8PlWj5lulyazH3BpVd3U6jgnjlkzZ9qu8SZqy9qptlva8CV01zI9YabqtJb3bsaO10gEq3Yd04nAVVX1voFZpwOHtuFDgS8Msx5V9daq2rGqltB1K59TVS8HzgUOnol6VNWNwIoku7dJ+wJXMsPHgu4U4D5Jtmjvz1g9ZuxYjDPZ/p8OvLLdYbMPcMdAt3vvkryA7lTxi6rqx+Pqd0iSTZPsDOwGfL3v7VfVd6rq0VW1pP2erqS74PlGZvhYjIg58/U3k7WD4667+V3gQXeIzkDdtkzyiLFhugufL2fm26XJvIyB04Bz4ZgNmBNt13iTtWVJFiXZqA3vQteWXTOD9ZrsvZuRNhYYmbsCn07XPfpt4LL22p/u+qazgauB/wAeNYN1eja/vCtwl/YGLqfrat50yNt+MrCsHY9/AbaejWMBvAP4bvvF/hTd3RhDPxZ0DeRq4B664HDYZPtPd0fNh+k+VX0HWDrkeiynO88/9nt6/ED5v2j1+B6w37DqMG7+dfzyrsChHYtRfrW25r/acfuLWazHZO3gp9r7+W26fy7bz0LddqG7I+tbwBVjx2k22+iBum0J3AJsNTBtVo7ZXGm7plmvCdsy4H+09/gy4FLgd2a4XpO+d8NoYyd6+eR1SZKknozEqUBJkqS5wGAlSZLUE4OVJElSTwxWkiRJPTFYSZIk9cRgJUmS1BODlSRJUk8MVpIkST35/9hMRxBn9JhYAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 720x360 with 2 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 356
        },
        "id": "4AoJ8W2THakc",
        "outputId": "2072464a-29f1-4441-a7c2-7d1ae19669f2"
      },
      "source": [
        "# Word count in training set\r\n",
        "fig,(ax1,ax2) = plt.subplots(1,2,figsize=(10,5))\r\n",
        "tweet_len = df_train[df_train['target']==1]['text'].str.split().map(lambda x: len(x))\r\n",
        "ax1.hist(tweet_len,color='red')\r\n",
        "ax1.set_title('disaster tweets')\r\n",
        "tweet_len=df_train[df_train['target']==0]['text'].str.split().map(lambda x: len(x))\r\n",
        "ax2.hist(tweet_len,color='green')\r\n",
        "ax2.set_title('Not disaster tweets')\r\n",
        "fig.suptitle('Words in training tweet')\r\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlYAAAFTCAYAAAD7gEIxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3de5hlVX3m8e8bbioizaVtubWoIIYkCqRVNKhENAqSNDOPIV6ijYNpk5CJRjNCMjMCSVR0MvHymCGCJDYYBQYvICEXQuQWBWkUL4gOLUK6mwZa7ogXkN/8sVfJoajuOtW9q+pU8f08Tz2199rr7L32qT6r37P22uekqpAkSdLm+7nZboAkSdJ8YbCSJEnqicFKkiSpJwYrSZKknhisJEmSemKwkiRJ6onBStIjJDkhySc24XHXJjl4Gpo07PH/McmyvutK0lQYrKQRl+RPkvzjuLLrN1D2mplt3cOq6heq6uJNeWySSrLXZh7/0Kpa0XfdqUhycJI1fe93iOPemORlM31cSY9msJJG36XAC5NsAZBkF2ArYP9xZXu1ukNLsmXPbZ0Wc6WdkmSwkkbfVXRBar+2/iLgC8B3xpV9t6puTrJrkvOS3JFkVZLfGdtRu8x3TpJPJLkHOCrJ05JckuTeJBcCOw/Uf1yre3uSu5JclWTRRI0cHDVpxzk7yeltv9cmWbKBx42Fwa8luS/Jb42N/CQ5NsktwN8l2SHJ+UnWJ7mzLe8+sJ+Lk7y5LR+V5PIkf9nqfi/JoZtY92lJLm3n8a9J/nqiS6VJtgX+Edi1ncd97W/xwyQ7tzr/PcmDSZ7U1v88yQfb8jatDf+R5NYkf5Pk8QP7PzzJNe3v8MUkz27lZwCLgc+3Y75zoudZ0swwWEkjrqp+AlwJvLgVvRi4DLh8XNlYQDkTWAPsCrwaeE+Slw7scilwDrAA+Hvgk8DVdIHqz4HBuUfLgO2BPYCdgN8Ffjhk03+jtWUBcB7wkQ2c39g5PKeqnlhVZ7X1pwA7Ak8FltP1V3/X1he3dky4z+b5dOFzZ+D9wGlJsgl1Pwl8me78TwDesIHz+AFwKHBzO48nVtXNdMH4Ja3aS4CbgF8ZWL+kLZ8EPJMuLO8F7Aa8CyDJ/sDfAm9p7fgocF6SbarqDcB/AL/ejvn+jTwnkqaZwUqaGy7h4RD1Irpgddm4skuS7EH3n/axVfWjqroG+BjwxoF9famqPldVDwELgecC/7OqflxVlwKfH6j7AN1/5HtV1U+r6uqqumfINl9eVRdU1U+BM4DnTPGcHwKOb+36YVXdXlWfrqr7q+pe4N08HFgmclNVndqOvwLYBZhwtG1DdZMspnt+3lVVP6mqy+lC4lRcArykXc58NvDhtv64tu9LW4hbDvxRVd3Rzu89wNicueXAR6vqyvZ3WAH8GDhwim2RNM0MVtLccClwUJIdgYVVdT3wRbq5VzsCv9jq7AqM/cc85ia60Y8xqweWdwXubKMtg/XHnAH8M3BmkpuTvD/JVkO2+ZaB5fuBx01xrtT6qvrR2EqSJyT5aJKb2mXMS4EFY/PMNnb8qrq/LT5xinXHns/7B+oOPn/DuAQ4GDgA+AZwIV0gPBBYVVW30wXcJwBXt0t9dwH/1MqhG6V7x9i2tn2P1j5JI8RgJc0NX6K7JPc7wL8DtJGjm1vZzVX1vba+Y5LtBh67GFg7sF4Dy+uAHdr8oMH6tGM8UFUnVtW+wAuBw3nk6Nd0qnHr7wD2AZ5fVU/i4dG6DV3e68M6uufzCQNle2yk/vg2QxeA9wH+E3BJVX2L7jk+jIcvA36f7tLmL1TVgvazfVWNBcHVwLsHti2oqidU1ac2clxJs8BgJc0BVfVDYCXwdrpLgGMub2WXtnqr6f4jf2+beP5s4Ghgws+lqqqb2n5PTLJ1koOAXx/bnuRXk/xSGxW6h+7S4EN9nx9wK/D0SepsRxc+7mqjdMdPQzseYeD5OaE9Py9g4PmZwK3ATkm2H9jH/XRz2I7h4SD1Rbr5ape0Og8BpwIfSPJkgCS7JXlFq38q8LtJnp/OtkleNRCgh3n+JM0Ag5U0d1wCPJkuTI25rJUNfszCa4E96UavPks3T+lfN7Lf19FN3r6DLqycPrDtKXQT3e8BrmttOGNzTmIDTgBWtMtcR26gzgeBx9ON7lxBd6lsJrweeAFwO/AXwFl085sepaq+DXwKuKGdy9ilukvo7uz88sD6djzy73YssAq4ol3q/Fe6kS6qaiXdyORHgDtbvaMGHvte4H+0Y/7x5pyspM2TKkeQJWlYSc4Cvl1V0z5iJmnuccRKkjYiyXOTPCPJzyV5Jd3HVXxuttslaTT5acaStHFPAT5D97ETa4Dfq6qvzm6TJI0qLwVKkiT1xEuBkiRJPTFYSZIk9cRgJUmS1BODlSRJUk8MVpIkST0xWEmSJPXEYDWPJfl4kr9oyy9K8p3ZbpMkTZckNyZ5WVv+0yQfm+026bHHYPUYUVWXVdU+07X/wRA3yvsc4pgHJ1kzk8eU5qMWcm5Lsu1A2ZuTXDzk4y9O8uZNPX5VvaeqNvnxkxkMcaO8zyGOeUKSCb+kXZvGYKWRkMRvAZDmny2At852I0ZNOv7/O0/5h51Hkuyf5CtJ7m1fFPu4gW2PGIlJcmySta3ud5Ic0sqfl+RLSe5Ksi7JR5Js3bYlyQfau9B7knwjyS8mWQ68HnhnkvuSfL7V3zXJp5OsT/K9JH84cPwTkpyT5BNJ7gGOGncuj9pnkjeN7bvVuT7J/x1YX51kv7b8rCQXJrmjnd+RA/W2SfKXSf4jya1J/ibJ49s7638Edm3HvK+dw/OSrGznfGuSv+rhzyU9Fvwv4I+TLJhoY5IXJrkqyd3t9wtb+buBFwEfaa/Dj2zg8W9IclOS25P893HbfjYSk+Rxra+5vfVtVyVZ1La9Kcl1rS+8IclbBvaxc5Lz22PuSHJZuu+MPANYDHy+te+drf6BSb7Y6n8tycED+7o4ybuT/DtwP/D0ce191D6TrEjyjrZ9tySV5Ji2/ozWpp9r64cnuaYd+4tJnj2w7wn74nTfffmnwG+1Y36tlR/Vnot7W/3Xb/hPrEepKn/mwQ+wNXAT8EfAVsCrgQeAv2jbDwbWtOV9gNXArm19T+AZbfmXgQPpvkdyT+A64G1t2yuAq4EFQICfB3Zp2z4+dqy2/nOt7rta254O3AC8om0/obXviFb38ROc0/h9Ph24q9XftZ3vmoFtd7Zt27bze1M7j/2B7wP7trofAM4DdgS2Az4PvHf88zRw3C8Bb2jLTwQOnO2/tz/+jPoPcCPwMrrvWRzrh94MXNyWd2yv2Te01+lr2/pObfvFwJs3sv99gfuAFwPbAH8FPAi8rG0/AfhEW35Le50/gW4U7ZeBJ7VtrwKe0fq0l9CFngPatvcCf0PXp25FF/YyeH4D7dkNuB04rPVDL2/rCwfO5z+AX2jnu9WGnrOB9f8CfL4tvw74LnDWwLZz2/L+wG3A89v5LWv72obh+uJPDBxzW+AeYJ+2vgvwC7P972ku/ThiNX8cSPfC/2BVPVBV5wBXbaDuT+lecPsm2aqqbqyq7wJU1dVVdUVVPVhVNwIfpetsoAtC2wHPoutcrquqdRs4xnPpOpQ/q6qfVNUNwKnAawbqfKmqPldVD1XVDyc7wbaPe4H96DrTfwZuTvKs1sbLquoh4HDgxqr6u3YeXwU+DfxmkgDLgT+qqjuq6l7gPePaNd4DwF5Jdq6q+6rqisnaKuln3gX81yQLx5W/Cri+qs5or9NPAd8Gfn3I/b4aOL+qLq2qHwP/E3hoA3UfoPsS7b2q6qetn7sHoKr+oaq+W51LgH+hC1Bjj9sFeGrrVy+rljYm8NvABVV1QevTLgRW0gWtMR+vqmvb+T4wxDleAhzURqVeDLwf+JW27SVtO3R92ker6sp2fiuAH9P9vzBMXzzeQ8AvJnl8Va2rqmuHaKsag9X8sSuwdtyL/qaJKlbVKuBtdO9UbktyZpJdAZI8sw1939Iu0b0H2Lk97t+AjwB/3R53SpInbaA9T6W7pHbX2A/dkPOigTqrN+E8L6EbVXpxW76YroMZ7GSeCjx/3LFfDzwFWEj3rvXqgW3/1Mo35GjgmcC32yWEwzeh3dJjUlV9EzgfOG7cprFR50E30Y38DGNXBvqQqvoB3QjRRM6geyN2ZpKbk7w/yVYASQ5NckW7rHYXXRDauT3ufwGrgH9pl8bGn8Ogp9K9eRvsdw6iC2ZjptTntTe8P6B7M/kiuufx5iT78Og+7x3jjr0H3XM0TF88eMwfAL8F/C6wLsk/tDevGpLBav5YB+zWRmTGLN5Q5ar6ZFUdRPeiK+B9bdPJdO8a966qJ9G9ADPwuA9X1S/TDcM/E/hvY5vGHWI18L2qWjDws11VDb5729A7v41tHwtWL2rLl/DoYLUauGTcsZ9YVb9Hd0nwh3RD22Pbtq+qJ27omFV1fVW9Fngy3fN0TgbudJI0qeOB3+GRoelmuv5n0GJgbVuerH9YRxceAEjyBLpRqUdpo00nVtW+wAvpRrXfmGQbutHsvwQWVdUC4AJan1dV91bVO6rq6cBvAG9Pm486QftWA2eM63e2raqTBpsyyTltqM97NbB1Va1t68uAHYBrBo797nHHfkIbBZysL56oz/vnqno5XSj8Nt0Il4ZksJo/vkQ3v+APk2yV5D8Dz5uoYpJ9kry0dSo/ogsaY0Po29FdX7+vvUv5vYHHPTfJ89s7vR+0x4497lYeORnzy8C96SbJPz7JFukmuj93Cuc0fp/QdSq/Sjcnaw1wGfBKug71q63O+cAz001s3ar9PDfJz7dLhacCH0jy5HZeuyV5xcAxd0qy/cB5/3aShe2xd7XiDV1ykDROGyU/C/jDgeIL6F6nr0uyZZLfonvDdn7bPtHrf9A5wOFJDkp3g82fsYH/05L8apJfSrIFXf/2AN1reGu6aRHrgQeTHAr82sDjDk+yV3vDejfdNIoN9XmfAH49yStaf/e4dDcN7T7J0zNoQ33eHwCXtvWL2/rlVfXTVnYq8Lutf06SbZO8Ksl2TN4X3wrsmYcnwS9KsrS9efwx3Tw2+7spMFjNE1X1E+A/091ddwfdUO5nNlB9G+AkutGbW+hGYv6kbftjukmS99K9WM8aeNyTWtmddEP2t9MNlQOcRjdn664kn2sv+MPphrC/1471MWB7hveIfbbz/H90L/TL2vo9dBMx/32sk2nzpn6Nbg7Bze0c39fOG+BYuuH9K9rlzn+lm9BPVX0b+BRwQzvurnTB7dok9wEfAl4zzJwwSY/wZ3QTowGoqtvp+oh30PUl7wQOr6rvtyofAl6d5M4kHx6/szbv5xjgk3SjV3cCG/oMuqfQBbF76G7IuYRudOleurB3dnv86+hubBmzN13/cB/dm9f/U1VfaNveC/yP1k/8cVWtBpbSjfKvpxsp+m9M7f/ZR+yzlV1C94Z3LFhdTjedYWydqlpJNyL4kXYeq2h3Wg/RF4/dWX17kq+09r6dru+8g+5qwM/eYGtyY3c3SJIkaTM5YiVJktQTg5UkSVJPDFaSJEk9MVhJkiT1xGAlSZLUky1nuwEAO++8c+25556z3QxJM+jqq6/+flVt7BPv5wT7L+mxZ2P910gEqz333JOVK1fOdjMkzaAkE37l0lxj/yU99mys//JSoCRJUk8MVpIkST0xWEmSJPXEYCVJktQTg5UkSVJPDFaSJEk9MVhJkiT1xGAlSZLUE4OVJElSTwxWkiRJPTFYSZIk9WTS7wpMsg9w1kDR04F3Aae38j2BG4Ejq+rOJAE+BBwG3A8cVVVf6bfZmjeS6dt31fTtW9JjXk6cvv6rjrf/mqsmHbGqqu9U1X5VtR/wy3Rh6bPAccBFVbU3cFFbBzgU2Lv9LAdOno6GS5IkjZqpXgo8BPhuVd0ELAVWtPIVwBFteSlwenWuABYk2aWX1kqSJI2wqQar1wCfasuLqmpdW74FWNSWdwNWDzxmTSuTJEma14YOVkm2Bn4D+L/jt1VVAVO6IJxkeZKVSVauX79+Kg+VJEkaSVMZsToU+EpV3drWbx27xNd+39bK1wJ7DDxu91b2CFV1SlUtqaolCxcunHrLJUmSRsxUgtVrefgyIMB5wLK2vAw4d6D8jekcCNw9cMlQkmZMkn2SXDPwc0+StyXZMcmFSa5vv3do9ZPkw0lWJfl6kgNm+xwkzS1DBask2wIvBz4zUHwS8PIk1wMva+sAFwA3AKuAU4Hf7621kjQF3tUsaaZN+jlWAFX1A2CncWW3090lOL5uAcf00jpJ6s/P7mpOshQ4uJWvAC4GjmXgrmbgiiQLkuziqLukYfnJ65IeK7yrWdK0M1hJmve8q1nSTDFYSXos8K5mSTPCYCXpscC7miXNiKEmr0vSXDVwV/NbBopPAs5OcjRwE3BkK7+A7gvkV9HdQfimGWzqY5ZfZqz5xGAlaV7zrmZJM8lLgZIkST1xxEqSNG9N52VGaSKOWEmSJPXEYCVJktQTg5UkSVJPDFaSJEk9MVhJkiT1xGAlSZLUE4OVJElSTwxWkiRJPTFYSZIk9cRgJUmS1BODlSRJUk8MVpIkST0xWEmSJPXEYCVJktSTLWe7AepJMn37rpq+fUuSNI84YiVJktQTg5UkSVJPDFaSJEk9MVhJkiT1xGAlSZLUk6GCVZIFSc5J8u0k1yV5QZIdk1yY5Pr2e4dWN0k+nGRVkq8nOWB6T0GSJGk0DDti9SHgn6rqWcBzgOuA44CLqmpv4KK2DnAosHf7WQ6c3GuLJUmSRtSkwSrJ9sCLgdMAquonVXUXsBRY0aqtAI5oy0uB06tzBbAgyS69t1ySJGnEDDNi9TRgPfB3Sb6a5GNJtgUWVdW6VucWYFFb3g1YPfD4Na3sEZIsT7Iyycr169dv+hlIkiSNiGGC1ZbAAcDJVbU/8AMevuwHQFUVMKWP566qU6pqSVUtWbhw4VQeKkmSNJKGCVZrgDVVdWVbP4cuaN06domv/b6tbV8L7DHw+N1bmSTNOG++kTSTJg1WVXULsDrJPq3oEOBbwHnAsla2DDi3LZ8HvLF1UAcCdw9cMpSkmebNN5JmzLBfwvxfgb9PsjVwA/AmulB2dpKjgZuAI1vdC4DDgFXA/a2uJM24gZtvjoLu5hvgJ0mWAge3aiuAi4FjGbj5BriijXbt4ptDScMaKlhV1TXAkgk2HTJB3QKO2cx2SVIfBm++eQ5wNfBWpn7zzSOCVZLldCNaLF68eNoaL2nu8ZPXJc1n3nwjaUYZrCTNZ958I2lGGawkzVvefCNppg07eV2S5ipvvpE0YwxWkuY1b76RNJMMVpIkjZicmGnbdx0/pXs1NEXOsZIkSeqJwUqSJKknBitJkqSeGKwkSZJ6YrCSJEnqicFKkiSpJwYrSZKknhisJEmSemKwkiRJ6onBSpIkqScGK0mSpJ4YrCRJknpisJIkSeqJwUqSJKknBitJkqSebDnbDZAkjb6cmNlugjQnOGIlSZLUE4OVJElSTwxWkiRJPTFYSZIk9cRgJUmS1JOhglWSG5N8I8k1SVa2sh2TXJjk+vZ7h1aeJB9OsirJ15McMJ0nIG1QMn0/kiRNYCojVr9aVftV1ZK2fhxwUVXtDVzU1gEOBfZuP8uBk/tqrCRJ0ijbnEuBS4EVbXkFcMRA+enVuQJYkGSXzTiOJEnSnDBssCrgX5JcnWR5K1tUVeva8i3Aora8G7B64LFrWpkkSdK8Nuwnrx9UVWuTPBm4MMm3BzdWVSWpqRy4BbTlAIsXL57KQyVpaEluBO4Ffgo8WFVLkuwInAXsCdwIHFlVdyYJ8CHgMOB+4Kiq+spstFvS3DTUiFVVrW2/bwM+CzwPuHXsEl/7fVurvhbYY+Dhu7ey8fs8paqWVNWShQsXbvoZSNLknCMqaUZMGqySbJtku7Fl4NeAbwLnActatWXAuW35POCN7e7AA4G7By4ZStIocI6opGkxzKXARcBnuxFytgQ+WVX/lOQq4OwkRwM3AUe2+hfQDaOvohtKf1PvrZak4Y3NES3go1V1ClOfI+qbQ0lDmTRYVdUNwHMmKL8dOGSC8gKO6aV1krT5nCMqacb4yeuS5jXniEqaSQYrSfOWc0QlzbRhP25BkuYi54hKmlEGK0nzlnNEJc00LwVKkiT1xGAlSZLUE4OVJElSTwxWkiRJPTFYSZIk9cRgJUmS1BODlSRJUk8MVpIkST0xWEmSJPXEYCVJktQTg5UkSVJPDFaSJEk9MVhJkiT1xGAlSZLUE4OVJElSTwxWkiRJPTFYSZIk9cRgJUmS1JMtZ7sBjynJbLdAkiRNI0esJEmSemKwkiRJ6omXAjU5L2FKkjQUR6wkSZJ6MnSwSrJFkq8mOb+tPy3JlUlWJTkrydatfJu2vqpt33N6mi5JkjRapjJi9VbguoH19wEfqKq9gDuBo1v50cCdrfwDrZ4kSdK8N9QcqyS7A68C3g28PUmAlwKva1VWACcAJwNL2zLAOcBHkqSqqr9mS5KkTZETp2/ebB3vf/XDjlh9EHgn8FBb3wm4q6oebOtrgN3a8m7AaoC2/e5WX5IkaV6bNFglORy4raqu7vPASZYnWZlk5fr16/vctSQ9gnNEJc2UYUasfgX4jSQ3AmfSXQL8ELAgydilxN2BtW15LbAHQNu+PXD7+J1W1SlVtaSqlixcuHCzTkKSJuEcUUkzYtJgVVV/UlW7V9WewGuAf6uq1wNfAF7dqi0Dzm3L57V12vZ/c36VpNkyMEf0Y219bI7oOa3KCuCItry0rdO2H9LqS9JQNudzrI6lm8i+im4O1Wmt/DRgp1b+duC4zWuiJG0W54hKmjFT+uT1qroYuLgt3wA8b4I6PwJ+s4e2SdJmGZwjmuTgHve7HFgOsHjx4r52K2ke8JPXJc1nzhGVNKMMVpLmLeeISpppBitJj0XOEZU0LaY0x0qS5irniEqaCY5YSZIk9cRgJUmS1BODlSRJUk8MVpIkST0xWEmSJPXEYCVJktQTg5UkSVJPDFaSJEk9MVhJkiT1xGAlSZLUE4OVJElSTwxWkiRJPTFYSZIk9cRgJUmS1JMtZ7sBkqR+5MTMdhOkxzxHrCRJknpisJIkSeqJwUqSJKknBitJkqSeGKwkSZJ6YrCSJEnqicFKkiSpJwYrSZKknhisJEmSejJpsEryuCRfTvK1JNcmObGVPy3JlUlWJTkrydatfJu2vqpt33N6T0GSJGk0DDNi9WPgpVX1HGA/4JVJDgTeB3ygqvYC7gSObvWPBu5s5R9o9SRJkua9SYNVde5rq1u1nwJeCpzTylcAR7TlpW2dtv2QJH6BlaQZ54i7pJk21ByrJFskuQa4DbgQ+C5wV1U92KqsAXZry7sBqwHa9ruBnfpstCQNyRF3STNqqGBVVT+tqv2A3YHnAc/a3AMnWZ5kZZKV69ev39zdSdKjOOIuaaZN6a7AqroL+ALwAmBBki3bpt2BtW15LbAHQNu+PXD7BPs6paqWVNWShQsXbmLzJWnjHHGXNJOGuStwYZIFbfnxwMuB6+gC1qtbtWXAuW35vLZO2/5vVVV9NlqShuWIu6SZNMyI1S7AF5J8HbgKuLCqzgeOBd6eZBXdO7rTWv3TgJ1a+duB4/pvtiRNjSPukmbClpNVqKqvA/tPUH4D3bu/8eU/An6zl9ZJ0mZIshB4oKruGhhxfx8Pj7ifycQj7l/CEXdJm2DSYCVJc9guwIokW9CN0J9dVecn+RZwZpK/AL7KI0fcz2gj7ncAr5mNRkuauwxWkuYtR9wlzTS/K1CSJKknBitJkqSeGKwkSZJ64hwraVNM54dxexOaJM1ZjlhJkiT1xGAlSZLUE4OVJElSTwxWkiRJPTFYSZIk9cRgJUmS1BODlSRJUk8MVpIkST0xWEmSJPXEYCVJktQTg5UkSVJPDFaSJEk9MVhJkiT1xGAlSZLUE4OVJElSTwxWkiRJPTFYSZIk9cRgJUmS1BODlSRJUk8MVpIkST0xWEmSJPVk0mCVZI8kX0jyrSTXJnlrK98xyYVJrm+/d2jlSfLhJKuSfD3JAdN9EpIkSaNgmBGrB4F3VNW+wIHAMUn2BY4DLqqqvYGL2jrAocDe7Wc5cHLvrZYkSRpBkwarqlpXVV9py/cC1wG7AUuBFa3aCuCItrwUOL06VwALkuzSe8slaRKOuEuaaVOaY5VkT2B/4EpgUVWta5tuARa15d2A1QMPW9PKJGmmOeIuaUZtOWzFJE8EPg28raruSfKzbVVVSWoqB06ynK7jYvHixVN5qCQNpb35W9eW700yOOJ+cKu2ArgYOJaBEXfgiiQLkuwy8CZS0kbkxExeaRPV8VOKGbNmqBGrJFvRhaq/r6rPtOJbxy7xtd+3tfK1wB4DD9+9lT1CVZ1SVUuqasnChQs3tf2SNBRH3CXNhGHuCgxwGnBdVf3VwKbzgGVteRlw7kD5G9tchQOBu323J2k2jR9xH9zWRqemPOKeZGWSlevXr++xpZLmumFGrH4FeAPw0iTXtJ/DgJOAlye5HnhZWwe4ALgBWAWcCvx+/82WpOE44i5pJk06x6qqLgc2dNH0kAnqF3DMZrZLkjbbECPuJ/HoEfc/SHIm8HwccZc0RUNPXpekOWhsxP0bSa5pZX9KF6jOTnI0cBNwZNt2AXAY3Yj7/cCbZra5kuY6g5WkecsRd0kzze8KlCRJ6onBSpIkqSdeChwv0/fhZpIkaX5zxEqSJKknBitJkqSeGKwkSZJ6YrCSJEnqicFKkiSpJwYrSZKknhisJEmSemKwkiRJ6onBSpIkqScGK0mSpJ4YrCRJknpisJIkSeqJwUqSJKknBitJkqSeGKwkSZJ6YrCSJEnqicFKkiSpJwYrSZKknhisJEmSemKwkiRJ6onBSpIkqScGK0mSpJ4YrCRJknoyabBK8rdJbkvyzYGyHZNcmOT69nuHVp4kH06yKsnXkxwwnY2XpMnYh0maScOMWH0ceOW4suOAi6pqb+Citg5wKLB3+1kOnNxPMyVpk30c+zBJM2TSYFVVlwJ3jCteCqxoyyuAIwbKT6/OFcCCJLv01VhJmir7MEkzaVPnWC2qqnVt+RZgUVveDVg9UG9NK2v6KeUAAAbpSURBVHuUJMuTrEyycv369ZvYDEnaJJvdh0nSRLbc3B1UVSWpTXjcKcApAEuWLJny4yWpD5vShyVZTnepkMWLF0/peDkxU6ovaW7Z1BGrW8eGx9vv21r5WmCPgXq7tzJJGiWb1YdV1SlVtaSqlixcuHDaGytp7tjUYHUesKwtLwPOHSh/Y7uz5kDg7oHhdkkaFfZhkqbFpJcCk3wKOBjYOcka4HjgJODsJEcDNwFHtuoXAIcBq4D7gTdNQ5slaWj2YZJm0qTBqqpeu4FNh0xQt4BjNrdRktQX+zBJM8lPXpckSerJZt8VKKlnmca7xsobcCVpOjliJUmS1BODlSRJUk8MVpIkST0xWEmSJPXEYCVJktQTg5UkSVJPDFaSJEk9MVhJkiT1xGAlSZLUE4OVJElSTwxWkiRJPfG7AiVJ0sjLidP4PapAHd/Pd6k6YiVJktQTg5UkSVJP5ualwEzvcKAkSdKmcMRKkiSpJwYrSZKknhisJEmSemKwkiRJ6onBSpIkqScGK0mSpJ4YrCRJknpisJIkSeqJwUqSJKknc/OT1yVtmun81oLq5wtMJWkum5YRqySvTPKdJKuSHDcdx5Ck6WIfJmlT9R6skmwB/DVwKLAv8Nok+/Z9HEmaDvZhkjbHdIxYPQ9YVVU3VNVPgDOBpdNwHEmaDvZhkjbZdASr3YDVA+trWpkkzQX2YZI22axNXk+yHFjeVu9L8p3ZassEdga+P9uNmMRcaCPMjXbaxj4kU23jU6erKdNtkv5rlP9Wo9q2UW0XjG7bbNfUbbRtOWFKN/dssP+ajmC1FthjYH33VvYIVXUKcMo0HH+zJVlZVUtmux0bMxfaCHOjnbaxH3OhjUOatA/bWP81ys/DqLZtVNsFo9s22zV1M9W26bgUeBWwd5KnJdkaeA1w3jQcR5Kmg32YpE3W+4hVVT2Y5A+Afwa2AP62qq7t+ziSNB3swyRtjmmZY1VVFwAXTMe+Z8hIXqIcZy60EeZGO21jP+ZCG4eymX3YKD8Po9q2UW0XjG7bbNfUzUjbUn5asiRJUi/8rkBJkqSeGKzGSXJjkm8kuSbJytluD0CSv01yW5JvDpTtmOTCJNe33zuMYBtPSLK2PZfXJDlsltu4R5IvJPlWkmuTvLWVj8xzuZE2jtpz+bgkX07ytdbOE1v505Jc2b4K5qw2+fsxY1S/CmeU+rVR7c9GtQ8b5X5rVPur2e6fvBQ4TpIbgSVVNTKfw5HkxcB9wOlV9Yut7P3AHVV1UuvAd6iqY0esjScA91XVX85WuwYl2QXYpaq+kmQ74GrgCOAoRuS53Egbj2S0nssA21bVfUm2Ai4H3gq8HfhMVZ2Z5G+Ar1XVybPZ1pmS7qtw/h/wcroPFb0KeG1VfWtWG8Zo9Wuj2p+Nah82yv3WqPZXs90/OWI1B1TVpcAd44qXAiva8gq6f8yzZgNtHClVta6qvtKW7wWuo/tE7ZF5LjfSxpFSnfva6lbtp4CXAue08ln/dznD/CqcIYxqfzaqfdgo91uj2l/Ndv9ksHq0Av4lydXpPl15VC2qqnVt+RZg0Ww2ZiP+IMnX2zD7rF6uHJRkT2B/4EpG9Lkc10YYsecyyRZJrgFuAy4EvgvcVVUPtiqPta+CGeWvwhn1fm0kX4PNyLzuRrnfGrX+ajb7J4PVox1UVQfQfbP9MW14eKRVdz13FK/pngw8A9gPWAf879ltTifJE4FPA2+rqnsGt43KczlBG0fuuayqn1bVfnSfTP484Fmz3CRt2Jzp10blNdiMzOtulPutUeyvZrN/MliNU1Vr2+/bgM/S/UFG0a3t+vbYde7bZrk9j1JVt7Z/3A8BpzICz2W73v5p4O+r6jOteKSey4naOIrP5Ziqugv4AvACYEGSsc/Hm/DrrOaxob7OazbMgX5tpF6DY0bldTfK/dao91ez0T8ZrAYk2bZNwCPJtsCvAd/c+KNmzXnAsra8DDh3FtsyobEXffOfmOXnsk1oPA24rqr+amDTyDyXG2rjCD6XC5MsaMuPp5uwfR1dB/bqVm0k/11Oo5H8Kpw50q+NzGtw0Ci87ka53xrV/mq2+yfvChyQ5Ol07+ag+1T6T1bVu2exSQAk+RRwMN03c98KHA98DjgbWAzcBBxZVbM28XIDbTyYbii4gBuBtwzMCZhxSQ4CLgO+ATzUiv+Ubk7ASDyXG2njaxmt5/LZdJM/t6B7g3Z2Vf1Zew2dCewIfBX47ar68Wy1c6a128o/yMNfhTMK/cdI9Wuj2p+Nah82yv3WqPZXs90/GawkSZJ64qVASZKknhisJEmSemKwkiRJ6onBSpIkqScGK0mSpJ4YrCRJknpisJIkSeqJwUqSJKkn/x9EbrABT2YXsAAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 720x360 with 2 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 356
        },
        "id": "48itF2QxIGHt",
        "outputId": "a4655cef-8517-4603-860b-15ddcd2b12de"
      },
      "source": [
        "# Character count in test set\r\n",
        "fig,(ax1,ax2) = plt.subplots(1,2,figsize=(10,5))\r\n",
        "tweet_len = df_test[df_test['target']==1]['text'].str.len()\r\n",
        "ax1.hist(tweet_len,color='red')\r\n",
        "ax1.set_title('disaster tweets')\r\n",
        "tweet_len = df_test[df_test['target']==0]['text'].str.len()\r\n",
        "ax2.hist(tweet_len,color='green')\r\n",
        "ax2.set_title('Not disaster tweets')\r\n",
        "fig.suptitle('Characters in test tweets')\r\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmEAAAFTCAYAAABxr9twAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3de5xdVX3//9dbbioi4RJTCGhUQI32J2JErJdaUQSLhm/rBWprtPhN9YutFm9Yvz8F6wW1lepPq41ijXgBS7VGpUWKgpcKGhAviEhAMIkBQriLKMjn98deA4dhJpkwk9lzktfz8TiP2Xuttff+7D1z1nzO2uuck6pCkiRJ0+s+fQcgSZK0JTIJkyRJ6oFJmCRJUg9MwiRJknpgEiZJktQDkzBJkqQemIRJm7Ekxyb5VN9xTLckNyd5WN9xSNL6mIRJQy7JnyVZ3hKPNUn+M8lT+o5rUJKXJvnWdB2vqh5QVZdt7HZJ5iWpJFtPNoYkn0jy9g20qSR7TfZYG2MicUmaHiZh0hBLcjTwT8A7gTnAg4F/BhZugmNNOjEZxmNL0qZiEiYNqSQ7Am8Djqqqz1fVr6rqtqr6UlW9fqDptkk+meSmJBcmWTCwj2OSXNrqfpLkfw3UvTTJt5OckGQdcGyShyf5WpJ1Sa5J8ukkswa22TPJ55OsbW0+mORRwEeAJ7XRuutb2+2S/EOSXyS5KslHktyv1T09yaokb0xyJfCvSXZN8uUk1ye5Nsk3k4zZhw2OMLWRnw8l+Uo7z3OTPHycy/qN9vP6FuuT2j7+MslFSa5LcnqSh7TytOtzdZIbk/woyWOSLAZeDLyh7edLY8Q4cqwftDYvSnJ2kj9t9U9u5/HHbf3AJBcMbD9mTK3ukUnOaNfp4iQvbOVjxtWu8+p2fS5OcuA410fSFDIJk4bXk4D7Al/YQLvnAScDs4BlwAcH6i4FngrsCBwHfCrJbgP1TwQuoxtlewcQ4F3A7sCjgD2BYwGSbAV8GbgCmAfMBU6uqouAVwDfabcJR5K244F9gH2BvVr7twwc+/eAnYGHAIuB1wKrgNktnr8DJvq9a4e389sJWNHOZSxPaz9ntVi/k2RhO9aftGN/E/hsa3dQ22Yfumv4QmBdVS0BPg28p+3nuaMPVFUjx3psa3MKcDbw9Fb+h3TX/mkD62cDrC+mJNsDZwCfAR7Uzv2fk8wfK64kjwBeBTyhqnYAng1cvp5rKWmKmIRJw2sX4Jqqun0D7b5VVadV1e+Ak4DHjlRU1b9V1S+r6o6WBFwC7D+w7S+r6v+rqtur6tdVtaKqzqiq31TVWuB9dMkBbbvdgde3Ublbq2rMeWBJQpdY/W1VXVtVN9HdUj18oNkdwFvbsX4N3AbsBjykjfh9syb+5bdfqKrvtmv1abrEb6JeAbyrqi5q278T2LeNPN0G7AA8Ekhrs2Yj9j3a2dx1PZ9Gl/COrN+ZhG0gpkOBy6vqX9vv7fvAvwMvGOeYvwO2A+Yn2aaqLq+qSydxDpImyCRMGl7rgF0nMF/qyoHlW4D7jmyT5CVJLmi3+K4HHgPsOtB+5eCOksxJcnK7dXUj8KmB9nsCV0wgKYRu9Ob+wHkDx/6vVj5ibVXdOrD+XrpRrK8muSzJMRM4zojR1+ABG7HtQ4D3D8R5Ld2I4Nyq+hrdyOKHgKuTLEnywI3Y92jfAfZJMocuUfwksGeSXemS3JFbmOPG1OqeOFLX6l9MN7J4D1W1AngN3Yjm1e33u/skzkHSBJmEScPrO8BvgMPuzcZt1OSjdLeidmm3CX9M9898xOiRpne2st+vqgcCfz7QfiXw4HGSwtH7uQb4NfDoqprVHjtW1QPG26aqbqqq11bVw+husR69CeYujTWythL4q4E4Z1XV/arqf1pcH6iqxwPz6W5Lvn49+1r/watuAc4DXg38uKp+C/wPcDRwaVVdM4GYVgJnj6p7QFW9cry4quozVfUUugSugHdvbOySNp5JmDSkquoGujlUH0pyWJL7J9kmySFJ3jOBXWxP9w93LUCSl9GNhK3PDsDNwA1J5nJXwgHwXWANcHyS7ZPcN8mTW91VwB5Jtm2x30GXAJ6Q5EHt+HOTPHu8Ayc5NMle7VbmDXS30e6YwHlujLVtn4OfMfYR4E1JHt3i2DHJC9ryE5I8Mck2wK+AWwdiumrUfsYyVpuz6RLjkVuPZ41aX29MdPPy9knyF+3vYZsW56PGOmaSRyR5RpLtWvy/Zuqvq6QxmIRJQ6yq/pFulOT/0iUQK+n+Yf/HBLb9CfCPdCNqVwG/D3x7A5sdB+xHlwR9Bfj8wP5+BzyXbpL9L+gm0b+oVX8NuBC4MsnIaM4b6W4vntNubf438Ij1HHvv1ubmFvM/V9XXN3SeG6ONRL0D+Ha7lXdAVX2BbmTo5Bbnj4FD2iYPpEsmr6N7Q8I6utumACfSzbO6Psl4v49jgaWtzQtb2dl0ye43xllnfTG1+XUH0c2v+yXdrdh30837Giuu7ejeJHFNa/sg4E0TvmiS7rVMfF6rJEmSpoojYZIkST0wCZMkSeqBSZgkSVIPTMIkSZJ6YBImSZLUA5MwSZKkHpiESZIk9cAkTJIkqQcmYZIkST0wCRMAST6R5O1t+alJLu47JknaVJJcnuSZbfnvknys75i05TEJ0z1U1Teran3f4TcpgwnfTN7nBI759CSrpvOY0uaoJURXJ9l+oOzlSc6a4PZnJXn5vT1+Vb2zqu719hsymPDN5H1O4JjHJvnUdB5zc2cSpqGTZOu+Y5A05bYCXt13EDNNOv6v3kz5i91CJXlckvOT3JTkFOC+A3V3G+FJ8sYkq1vbi5Mc2Mr3T/KdJNcnWZPkg0m2bXVJckJ7dXtjkh8leUySxcCLgTckuTnJl1r73ZP8e5K1SX6e5G8Gjn9sklOTfCrJjcBLR53LPfaZ5GUj+25tLknybwPrK5Ps25YfmeSMJNe283vhQLvtkvxDkl8kuSrJR5Lcr71i/09g93bMm9s57J9keTvnq5K8bwp+XdKW4L3A65LMGqsyyR8k+V6SG9rPP2jl7wCeCnywPQ8/OM72f5HkiiTrkrx5VN2dIzxJ7tv6mnWtb/tekjmt7mVJLmp94WVJ/mpgH7sm+XLb5tok30xynyQnAQ8GvtTie0Nrf0CS/2ntf5Dk6QP7OivJO5J8G7gFeNioeO+xzyRLk7y21c9NUkmOausPbzHdp60fmuSCduz/SfL/DOx7zL44ycHA3wEvasf8QSt/absWN7X2Lx7/V6x7qCofW9gD2Ba4AvhbYBvg+cBtwNtb/dOBVW35EcBKYPe2Pg94eFt+PHAAsHUrvwh4Tat7NnAeMAsI8Chgt1b3iZFjtfX7tLZvabE9DLgMeHarP7bFd1hre78xzmn0Ph8GXN/a797Od9VA3XWtbvt2fi9r5/E44Bpgfmt7ArAM2BnYAfgS8K7R12nguN8B/qItPwA4oO/ftw8fM/0BXA48E/j8QD/0cuCstrxze87+RXueHtHWd2n1ZwEvX8/+5wM3A08DtgPeB9wOPLPVHwt8qi3/VXue359udO7xwANb3R8DD2992h/SJUj7tbp3AR+h61O3oUsMM3h+A/HMBdYBz2n90LPa+uyB8/kF8Oh2vtuMd80G1v8S+FJb/jPgUuCUgbovtuXHAVcDT2znt6jtazsm1hd/auCY2wM3Ao9o67sBj+7772mYHo6EbZkOoOsk/qmqbquqU4HvjdP2d3RPzvlJtqmqy6vqUoCqOq+qzqmq26vqcuBf6Dom6JKmHYBH0nVEF1XVmnGO8QS6zudtVfXbqroM+Chw+ECb71TVf1TVHVX16w2dYNvHTcC+dB3v6cAvkzyyxfjNqroDOBS4vKr+tZ3H94F/B16QJMBi4G+r6tqqugl456i4RrsN2CvJrlV1c1Wds6FYJd3pLcBfJ5k9qvyPgUuq6qT2PP0s8FPguRPc7/OBL1fVN6rqN8D/C9wxTtvbgF2Avarqd62fuxGgqr5SVZdW52zgq3TJ1sh2uwEPaf3qN6tlJmP4c+C0qjqt9WlnAMvpkrIRn6iqC9v53jaBczwbeEob7Xoa8B7gya3uD1s9dH3av1TVue38lgK/ofu/MJG+eLQ7gMckuV9VramqCycQqxqTsC3T7sDqUR3EFWM1rKoVwGvoXgFdneTkJLsDJNmnDb9f2W4TvhPYtW33NeCDwIfadkuSPHCceB5Cd1vv+pEH3bD3nIE2K+/FeZ5NN1r1tLZ8Fl1nNNghPQR44qhjvxj4PWA23avh8wbq/quVj+dIYB/gp+02xqH3Im5pi1RVPwa+DBwzqmpkNHvQFXQjShOxOwN9SFX9im7kaSwn0b1oOznJL5O8J8k2AEkOSXJOu7V3PV3StGvb7r3ACuCr7fbc6HMY9BC6F3qD/c5T6JK4ERvV57UXx7+ie+H5VLrr+Mskj+Cefd5rRx17T7prNJG+ePCYvwJeBLwCWJPkK+2FribIJGzLtAaY20Z6Rjx4vMZV9ZmqegrdE7SAd7eqD9O9Gt27qh5I92TNwHYfqKrH090K2Ad4/UjVqEOsBH5eVbMGHjtU1eCrwvFeUa6vfiQJe2pbPpt7JmErgbNHHfsBVfVKutuSv6YbXh+p27GqHjDeMavqkqo6AngQ3XU6NQPv+JK0QW8F/jd3T7B+Sdf/DHowsLotb6h/WEOXaACQ5P50o1330Eaxjquq+cAf0I2WvyTJdnSj5P8AzKmqWcBptD6vqm6qqtdW1cOA5wFHp82fHSO+lcBJo/qd7avq+MFQNnBO4/V5zwe2rarVbX0RsBNwwcCx3zHq2Pdvo4sb6ovH6vNOr6pn0SWQP6UbOdMEmYRtmb5DNx/ib5Jsk+RPgP3HapjkEUme0TqgW+mSkpFh/B3o5gPc3F79vHJguyckeWJ7Bfmrtu3Idldx94mm3wVuSvcGgPsl2SrdJP4nbMQ5jd4ndB3QH9HNIVsFfBM4mK7z/X5r82Vgn3STdrdpjyckeVS7XflR4IQkD2rnNTfJsweOuUuSHQfO+8+TzG7bXt+Kx7vtIWmUNvp+CvA3A8Wn0T1P/yzJ1kleRPfi7sutfqzn/6BTgUOTPCXdm4fexjj//5L8UZLfT7IVXf92G91zeFu6qRlrgduTHAIcNLDdoUn2ai9ub6CbyjFen/cp4LlJnt36u/ume0PUHhu4PIPG6/NeBXyjrZ/V1r9VVb9rZR8FXtH65yTZPskfJ9mBDffFVwHzctcE/zlJFrYXmr+hm3dnf7cRTMK2QFX1W+BP6N5leC3dcPLnx2m+HXA83ajQlXQjPG9qda+jmwB6E90T+5SB7R7Yyq6ju22wjm64HuBEujlm1yf5j9Y5HEo3jP7zdqyPATsycXfbZzvPn9F1Ct9s6zfSTTL99kiH1OZ5HUQ35+GX7Rzf3c4b4I10txjOabdc/5vuzQpU1U+BzwKXtePuTpfkXZjkZuD9wOETmcMm6W7eRjfpG4CqWkfXR7yWri95A3BoVV3TmrwfeH6S65J8YPTO2jylo4DP0I2KXQeM9xl/v0eXtN1I92ajs+lGrW6iSww/17b/M7o37YzYm65/uJnuhe4/V9XXW927gP/b+onXVdVKYCHd3YO1dCNQr2fj/iffbZ+t7Gy6F8cjSdi36KZUjKxTVcvpRho/2M5jBe0d5xPoi0feYb4uyfkt3qPp+s5r6e4y3PliXBs28s4NSZIkTSNHwiRJknpgEiZJktQDkzBJkqQemIRJkiT1wCRMkiSpB1v3HcC9seuuu9a8efP6DkPSNDnvvPOuqar1fVPB0LD/krY84/VhQ5mEzZs3j+XLl/cdhqRpkmTMr9UaRvZf0pZnvD7M25GSJEk9MAmTJEnqgUmYJElSD0zCJEmSemASJkmS1AOTMEmSpB6YhEmSJPXAJEySJKkHJmGSJEk9MAmTJEnqgUmYJElSD4byuyMlSdLwy3HZZPuut9Ym2/dUcSRMkiSpByZhkiRJPTAJkyRJ6oFJmCRJUg9MwiRJknpgEiZJktQDkzBJkqQemIRJkiT1wCRMkiSpByZhkiRJPTAJkyRJ6oFJmCRJUg9MwiRJknpgEiZJktQDkzBJapI8IskFA48bk7wmyc5JzkhySfu5U2ufJB9IsiLJD5Ps1/c5SBoeJmGS1FTVxVW1b1XtCzweuAX4AnAMcGZV7Q2c2dYBDgH2bo/FwIenP2pJw8okTJLGdiBwaVVdASwElrbypcBhbXkh8MnqnAPMSrLb9IcqaRiZhEnS2A4HPtuW51TVmrZ8JTCnLc8FVg5ss6qVSdIGmYRJ0ihJtgWeB/zb6LqqKqA2cn+LkyxPsnzt2rVTFKWkYTclSViSg5Nc3CanHjNG/XZJTmn15yaZN6r+wUluTvK6qYhHkibpEOD8qrqqrV81cpux/by6la8G9hzYbo9WdjdVtaSqFlTVgtmzZ2/CsCUNk0knYUm2Aj5E12nNB45IMn9UsyOB66pqL+AE4N2j6t8H/OdkY5GkKXIEd92KBFgGLGrLi4AvDpS/pL1L8gDghoHblpK0XlMxErY/sKKqLquq3wIn001WHTQ4qfVU4MAkAUhyGPBz4MIpiEWSJiXJ9sCzgM8PFB8PPCvJJcAz2zrAacBlwArgo8D/mcZQJQ25radgH2NNTH3ieG2q6vYkNwC7JLkVeCNdh+etSEm9q6pfAbuMKltH927J0W0LOGqaQpO0mel7Yv6xwAlVdfOGGjqxVZIkbU6mYiRsIhNTR9qsSrI1sCOwjm7E7PlJ3gPMAu5IcmtVfXD0QapqCbAEYMGCBRv1ziRJkqSZZiqSsO8Beyd5KF2ydTjwZ6PajExq/Q7wfOBrbRj/qSMNkhwL3DxWAiZJkrS5mXQS1uZ4vQo4HdgK+HhVXZjkbcDyqloGnAiclGQFcC1doiZJkrTFmoqRMKrqNLp3CQ2WvWVg+VbgBRvYx7FTEYskSdIw6HtiviRJ0hbJJEySJKkHJmGSJEk9MAmTJEnqgUmYJElSD0zCJEmSemASJkmS1AOTMEmSpB6YhEmSJPXAJEySJKkHJmGSJEk9MAmTJEnqgUmYJElSD0zCJEmSemASJkmS1AOTMEmSpB6YhEmSJPXAJEySJKkHJmGSJEk9MAmTJEnqgUmYJA1IMivJqUl+muSiJE9KsnOSM5Jc0n7u1NomyQeSrEjywyT79R2/pOFhEiZJd/d+4L+q6pHAY4GLgGOAM6tqb+DMtg5wCLB3eywGPjz94UoaViZhktQk2RF4GnAiQFX9tqquBxYCS1uzpcBhbXkh8MnqnAPMSrLbNIctaUiZhEnSXR4KrAX+Ncn3k3wsyfbAnKpa09pcCcxpy3OBlQPbr2plkrRBJmGSdJetgf2AD1fV44BfcdetRwCqqoDamJ0mWZxkeZLla9eunbJgJQ03kzBJussqYFVVndvWT6VLyq4auc3Yfl7d6lcDew5sv0cru5uqWlJVC6pqwezZszdZ8JKGi0mYJDVVdSWwMskjWtGBwE+AZcCiVrYI+GJbXga8pL1L8gDghoHblpK0Xlv3HYAkzTB/DXw6ybbAZcDL6F6wfi7JkcAVwAtb29OA5wArgFtaW0maEJMwSRpQVRcAC8aoOnCMtgUctcmDkrRZ8nakJElSD0zCJEmSemASJkmS1AOTMEmSpB6YhEmSJPXAJEySJKkHJmGSJEk9MAmTJEnqgUmYJElSD0zCJEmSemASJkmS1AOTMEmSpB5MSRKW5OAkFydZkeSYMeq3S3JKqz83ybxW/qwk5yX5Ufv5jKmIR5IkaaabdBKWZCvgQ8AhwHzgiCTzRzU7EriuqvYCTgDe3cqvAZ5bVb8PLAJOmmw8kiRJw2AqRsL2B1ZU1WVV9VvgZGDhqDYLgaVt+VTgwCSpqu9X1S9b+YXA/ZJsNwUxSZIkzWhTkYTNBVYOrK9qZWO2qarbgRuAXUa1+VPg/Kr6zVgHSbI4yfIky9euXTsFYUuSJPVnRkzMT/JouluUfzVem6paUlULqmrB7Nmzpy84SZKkTWDrKdjHamDPgfU9WtlYbVYl2RrYEVgHkGQP4AvAS6rq0imIR5IkbeFyXDbZvuutNSX7mYqRsO8Beyd5aJJtgcOBZaPaLKObeA/wfOBrVVVJZgFfAY6pqm9PQSySJElDYdJJWJvj9SrgdOAi4HNVdWGStyV5Xmt2IrBLkhXA0cDIx1i8CtgLeEuSC9rjQZONSZIkaaabituRVNVpwGmjyt4ysHwr8IIxtns78PapiEGSJGmYzIiJ+ZIkSVsakzBJkqQemIRJkiT1wCRMkiSpByZhkiRJPTAJk6QBSS5P8qP2kTnLW9nOSc5Ickn7uVMrT5IPJFmR5IdJ9us3eknDxCRMku7pj6pq36pa0NaPAc6sqr2BM7nrsw4PAfZuj8XAh6c9UklDyyRMkjZsIbC0LS8FDhso/2R1zgFmJdmtjwAlDR+TMEm6uwK+muS8JItb2ZyqWtOWrwTmtOW5wMqBbVe1srtJsjjJ8iTL165du6niljRkpuQT8yVpM/KUqlrdvkLtjCQ/Haxs33u7Ud/eW1VLgCUACxYsmJpv/pU09BwJk6QBVbW6/bwa+AKwP3DVyG3G9vPq1nw1sOfA5nu0MknaIJMwSWqSbJ9kh5Fl4CDgx8AyYFFrtgj4YlteBrykvUvyAOCGgduWkrRe3o6UpLvMAb6QBLr+8TNV9V9Jvgd8LsmRwBXAC1v704DnACuAW4CXTX/IkoaVSZgkNVV1GfDYMcrXAQeOUV7AUdMQmqTNkLcjJUmSemASJkmS1AOTMEmSpB6YhEmSJPXAJEySJKkHJmGSJEk9MAmTJEnqgUmYJElSD0zCJEmSemASJkmS1AOTMEmSpB6YhEmSJPXAJEySJKkHJmGSJEk9MAmTJEnqgUmYJElSD0zCJEmSemASJkmS1AOTMEmSpB6YhEmSJPXAJEySJKkHJmGSJEk9MAmTJEnqgUmYJElSD0zCJEmSemASJkmjJNkqyfeTfLmtPzTJuUlWJDklybatfLu2vqLVz+szbknDZeup2EmSg4H3A1sBH6uq40fVbwd8Eng8sA54UVVd3ureBBwJ/A74m6o6fSpikqRJeDVwEfDAtv5u4ISqOjnJR+j6rA+3n9dV1V5JDm/tXtRHwJr5clw22b7rrbXJ9q1NZ9IjYUm2Aj4EHALMB45IMn9Uszs7KuAEuo6K1u5w4NHAwcA/t/1JUi+S7AH8MfCxth7gGcCprclS4LC2vLCt0+oPbO0laYOm4nbk/sCKqrqsqn4LnEzXMQ0ar6NaCJxcVb+pqp8DK9r+JKkv/wS8Abijre8CXF9Vt7f1VcDctjwXWAnQ6m9o7SVpg6YiCbuzE2oGO6h7tBnVUU1kW0maFkkOBa6uqvOmeL+LkyxPsnzt2rVTuWtJQ2xoJubbiUmaBk8GnpfkcrpR/WfQzXedlWRkDu0ewOq2vBrYE6DV70g37/VuqmpJVS2oqgWzZ8/etGcgaWhMRRJ2ZyfUDHZQ92gzqqOayLaAnZikTa+q3lRVe1TVPLr5ql+rqhcDXwee35otAr7Ylpe1dVr916rKGdKSJmQqkrDvAXu3t3BvS9dxLRvVZryOahlweHub90OBvYHvTkFMkjSV3ggcnWQF3VSKE1v5icAurfxo4Jie4pM0hCb9ERVVdXuSVwGn031Excer6sIkbwOWV9Uyuo7qpNZRXUuXqNHafQ74CXA7cFRV/W6yMUnSZFXVWcBZbfkyxnjTUFXdCrxgWgOTxrApP/5Cm86UfE5YVZ0GnDaq7C0Dy+N2VFX1DuAdUxGHJEnSsBiaifmSJEmbE5MwSZKkHpiESZIk9cAkTJIkqQcmYZIkST0wCZMkSeqBSZgkSVIPTMIkSZJ6YBImSZLUA5MwSZKkHpiESZIk9cAkTJIkqQcmYZIkST0wCZMkSeqBSZgkSVIPTMIkSZJ6YBImSZLUA5MwSZKkHpiESZIk9cAkTJIkqQcmYZIkST0wCZMkSeqBSZgkSVIPTMIkSZJ6YBImSU2S+yb5bpIfJLkwyXGt/KFJzk2yIskpSbZt5du19RWtfl6f8UsaLiZhknSX3wDPqKrHAvsCByc5AHg3cEJV7QVcBxzZ2h8JXNfKT2jtJGlCTMIkqanOzW11m/Yo4BnAqa18KXBYW17Y1mn1BybJNIUraciZhEnSgCRbJbkAuBo4A7gUuL6qbm9NVgFz2/JcYCVAq78B2GV6I5Y0rEzCJGlAVf2uqvYF9gD2Bx452X0mWZxkeZLla9eunXSMkjYPJmGSNIaquh74OvAkYFaSrVvVHsDqtrwa2BOg1e8IrBtjX0uqakFVLZg9e/Ymj13ScDAJk6Qmyewks9ry/YBnARfRJWPPb80WAV9sy8vaOq3+a1VV0xexpGG29YabSNIWYzdgaZKt6F6kfq6qvpzkJ8DJSd4OfB84sbU/ETgpyQrgWuDwPoKWNJxMwiSpqaofAo8bo/wyuvlho8tvBV4wDaFJ2gx5O1KSJKkHJmGSJEk9MAmTJEnqgUmYJElSD0zCJEmSeuC7IyVJanKcX/2p6eNImCRJUg9MwiRJknowqSQsyc5JzkhySfu50zjtFrU2lyRZ1Mrun+QrSX6a5MIkx08mFkmSpGEy2ZGwY4Azq2pv4My2fjdJdgbeCjyR7hOn3zqQrP1DVT2S7hOqn5zkkEnGI0mSNBQmm4QtBJa25aXAYWO0eTZwRlVdW1XXAWcAB1fVLVX1dYCq+i1wPrDHJOORJEkaCpNNwuZU1Zq2fCUwZ4w2c4GVA+urWtmdkswCnks3mjamJIuTLE+yfO3atZOLWpIkqWcb/IiKJP8N/N4YVW8eXKmqSlIbG0CSrYHPAh9oX5I7pqpaAiwBWLBgwUYfR5IkaSbZYBJWVc8cry7JVUl2q6o1SXYDrh6j2Wrg6QPrewBnDawvAS6pqn+aUMSSJEmbgcnejlwGLGrLi4AvjtHmdOCgJDu1CfkHtTKSvB3YEXjNJOOQJEkaKpNNwo4HnpXkEuCZbZ0kC5J8DKCqrgX+Hvhee7ytqq5NsgfdLc35wPlJLkjy8knGI0mSNBQm9bVFVbUOOHCM8uXAywfWPw58fFSbVYDfDyFJkrZIfmK+JElSD0zCJEmSemASJkmS1OAnFugAAAyvSURBVAOTMEmSpB6YhEmSJPXAJEySJKkHk/qICkmSpluO89ONtHlwJEySJKkHJmGS1CTZM8nXk/wkyYVJXt3Kd05yRpJL2s+dWnmSfCDJiiQ/TLJfv2cgaZiYhEnSXW4HXltV84EDgKOSzAeOAc6sqr2BM9s6wCHA3u2xGPjw9IcsaViZhElSU1Vrqur8tnwTcBEwF1gILG3NlgKHteWFwCercw4wK8lu0xy2pCFlEiZJY0gyD3gccC4wp6rWtKorgTlteS6wcmCzVa1MkjbIJEySRknyAODfgddU1Y2DdVVVQG3k/hYnWZ5k+dq1a6cwUknDzCRMkgYk2YYuAft0VX2+FV81cpux/by6la8G9hzYfI9WdjdVtaSqFlTVgtmzZ2+64CUNFZMwSWqSBDgRuKiq3jdQtQxY1JYXAV8cKH9Je5fkAcANA7ctJWm9/LBWSbrLk4G/AH6U5IJW9nfA8cDnkhwJXAG8sNWdBjwHWAHcArxsesOVNMxMwiSpqapvAeN9HPuBY7Qv4KhNGpSkzZa3IyVJknpgEiZJktQDkzBJkqQemIRJkiT1wCRMkiSpByZhkiRJPTAJkyRJ6oFJmCRJUg9MwiRJknpgEiZJktQDkzBJkqQe+N2RkqQpl+PG+wpOSSMcCZMkSeqBSZgkSVIPTMIkSZJ6YBImSZLUA5MwSZKkHpiESZIk9cAkTJIkqQcmYZIkST0wCZMkSeqBSZgkSVIPTMIkSZJ6MKkkLMnOSc5Ickn7udM47Ra1NpckWTRG/bIkP55MLJIkScNksiNhxwBnVtXewJlt/W6S7Ay8FXgisD/w1sFkLcmfADdPMg5JkqShMtkkbCGwtC0vBQ4bo82zgTOq6tqqug44AzgYIMkDgKOBt08yDkmSpKGy9SS3n1NVa9rylcCcMdrMBVYOrK9qZQB/D/wjcMsk45CkSUvyceBQ4Oqqekwr2xk4BZgHXA68sKquSxLg/cBz6Pqwl1bV+X3EfW/luPQdgrRF2+BIWJL/TvLjMR4LB9tVVQE10QMn2Rd4eFV9YYLtFydZnmT52rVrJ3oYSdoYn6CN1A8Yb9rFIcDe7bEY+PA0xShpM7HBkbCqeuZ4dUmuSrJbVa1Jshtw9RjNVgNPH1jfAzgLeBKwIMnlLY4HJTmrqp7OGKpqCbAEYMGCBRNO9iRpoqrqG0nmjSpeyF192FK6/uuNrfyT7QXoOUlmjfSH0xOtpGE32Tlhy4CRdzsuAr44RpvTgYOS7NQm5B8EnF5VH66q3atqHvAU4GfjJWCS1KPxpl2sb6qFJG3QZJOw44FnJbkEeGZbJ8mCJB8DqKpr6eZ+fa893tbKJGmobOy0ixFOp5A0lklNzK+qdcCBY5QvB14+sP5x4OPr2c/lwGMmE4skbSLjTbtYDew50G6PVnYPTqeQNBY/MV+S1m+8aRfLgJekcwBwg/PBJG2MyX5EhSRtNpJ8lm4S/q5JVtF90PTxwOeSHAlcAbywNT+N7uMpVtB9RMXLpj1gSUPNJEySmqo6YpyqsaZdFHDUpo1I0ubM25GSJEk9MAmTJEnqgUmYJElSD0zCJEmSemASJkmS1AOTMEmSpB6YhEmSJPXAJEySJKkHJmGSJEk9MAmTJEnqgUmYJElSD0zCJEmSemASJkmS1AOTMEmSpB6YhEmSJPXAJEySJKkHJmGSJEk92LrvACRJ48tx6TsESZuII2GSJEk9MAmTJEnqgUmYJElSD0zCJEmSemASJkmS1AOTMEmSpB6YhEmSJPXAJEySJiHJwUkuTrIiyTF9xyNpeJiESdK9lGQr4EPAIcB84Igk8/uNStKwMAmTpHtvf2BFVV1WVb8FTgYW9hyTpCFhEiZJ995cYOXA+qpWJkkbNJTfHXneeeddk+SKCTbfFbhmU8aziRj39BvW2LeEuB+yKQPZ1JIsBha31ZuTXDxQPZN/fzM5NpjZ8c3k2GBmxzeTYwPYNcdmY+Mbsw8byiSsqmZPtG2S5VW1YFPGsykY9/Qb1tiNu1ergT0H1vdoZXdTVUuAJWPtYCZfh5kcG8zs+GZybDCz45vJscHUxuftSEm6974H7J3koUm2BQ4HlvUck6QhMZQjYZI0E1TV7UleBZwObAV8vKou7DksSUNiS0jCxrwFMASMe/oNa+zG3aOqOg04bRK7mMnXYSbHBjM7vpkcG8zs+GZybDCF8aWqpmpfkiRJmiDnhEmSJPVgs0nCNvTVIUm2S3JKqz83ybzpj/KeJhD30Ul+kuSHSc5MMiPeqj/Rr2pJ8qdJKsmMeKfLROJO8sJ2zS9M8pnpjnEsE/g7eXCSryf5fvtbeU4fcY6W5ONJrk7y43Hqk+QD7bx+mGS/6Y6xLzPt646S7Nn+hkb+9l/dyo9NsjrJBe3Ry99WksuT/KjFsLyV7ZzkjCSXtJ879RTbIwauzwVJbkzymr6u3VjPu/GuVR/PwXHie2+Sn7YYvpBkViufl+TXA9fwIz3ENu7vMcmb2rW7OMmzN/qAVTX0D7oJsZcCDwO2BX4AzB/V5v8AH2nLhwOnDEncfwTcvy2/cljibu12AL4BnAMsGIa4gb2B7wM7tfUHDUncS4BXtuX5wOV9x91ieRqwH/DjceqfA/wnEOAA4Ny+Y54pv9MeYtoN2K8t7wD8rP0tHQu8bgZcs8uBXUeVvQc4pi0fA7x7BsS5FXAl3edC9XLtxnrejXet+ngOjhPfQcDWbfndA/HNG6//mMbYxvw9tufHD4DtgIe25/RWG3O8zWUkbCJfHbIQWNqWTwUOTJJpjHEsG4y7qr5eVbe01XPoPoeobxP9qpa/p3sy3Tqdwa3HROL+38CHquo6gKq6eppjHMtE4i7ggW15R+CX0xjfuKrqG8C162myEPhkdc4BZiXZbXqi69WM+7qjqlpTVee35ZuAi5j5n/4/2K8vBQ7rMZYRBwKXVtVEP1B8yo3zvBvvWk37c3Cs+Krqq1V1e1vt7X/dBPqsQQuBk6vqN1X1c2AF3XN7wjaXJGwiXx1yZ5v2i74B2GVaohvfxn7lyZF0r1j6tsG425D2nlX1lekMbAMmcr33AfZJ8u0k5yQ5eNqiG99E4j4W+PMkq+jeqffX0xPapG2pX/szo8873XSNxwHntqJXtdtEH+/rlh/dC42vJjkv3TcQAMypqjVt+UpgTj+h3c3hwGcH1mfCtYPxr9VM/Fv8S+7+v+6hbarF2Ume2lNMY/0eJ33tNpckbLOX5M+BBcB7+45lQ5LcB3gf8Nq+Y7kXtqa7Jfl04AjgoyNzE2a4I4BPVNUedLcXTmq/B2mjJHkA8O/Aa6rqRuDDwMOBfYE1wD/2FNpTqmo/4BDgqCRPG6ys7v5Qr2/3T/eBvc8D/q0VzZRrdzcz4VqNJ8mbgduBT7eiNcCDq+pxwNHAZ5I8cLztN5FN9nvcXDrpiXx1yJ1tkmxNd8tm3bREN74JfeVJkmcCbwaeV1W/mabY1mdDce8APAY4K8nldPMMlqX/yfkTud6rgGVVdVsbXv4ZXVLWp4nEfSTwOYCq+g5wX7rvX5vpJvQc2AzNyPNOsg1dAvbpqvo8QFVdVVW/q6o7gI+ykbdbpkpVrW4/rwa+0OK4auTWWfvZ9/SBQ4Dzq+oqmDnXrhnvWs2Yv8UkLwUOBV7cEkXarb51bfk8unlX+0xnXOv5PU762m0uSdhEvjpkGbCoLT8f+NrIL7lHG4w7yeOAf6FLwPruYEasN+6quqGqdq2qeVU1j+7+/vOqank/4d5pIn8n/0E3CkaSXeme7JdNZ5BjmEjcv6Cbi0KSR9ElYWunNcp7ZxnwkvYOrQOAGwZumWzOZtzXHbU5sicCF1XV+wbKB+cH/S9gzHe6buLYtk+yw8gy3STuH3P3fn0R8MXpjm2UIxi4FTkTrt2A8a7VjHgOtqkfb6D7X3HLQPnsJFu15YfRvSie1j55Pb/HZcDh6T594aEttu9u1M6n6x0Hm/pBdwvmZ3RZ8ptb2dvofqHQ/VP6N7qJc98FHtZ3zBOM+7+Bq4AL2mNZ3zFPJO5Rbc9iBrw7coLXO3S3Un8C/Ag4vO+YJxj3fODbdO/UuQA4qO+YW1yfpRu+v41ulPFI4BXAKwau94faef1opvyd9PU77Tmep9DdovrhQH/zHOCk9rv5Id0/nd16iO1h7W/7B8CFA8+BXYAzgUtaX7lzj9dve7q7KzsOlPVy7cZ53o15rfp4Do4T3wq6+VUjf3sjn2bwp+13fgFwPvDcHmIb9/dId5fqUuBi4JCNPZ6fmC9JktSDzeV2pCRJ0lAxCZMkSeqBSZgkSVIPTMIkSZJ6YBImSZLUA5MwSZKkHpiESZIk9cAkTJIkqQf/P1ehkTsIP1BUAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 720x360 with 2 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 356
        },
        "id": "6EjKIhWGIzgx",
        "outputId": "9347c739-d7c0-4f28-b45c-67e9b30ac5b0"
      },
      "source": [
        "# Word count in test set\r\n",
        "fig,(ax1,ax2) = plt.subplots(1,2,figsize=(10,5))\r\n",
        "tweet_len = df_test[df_test['target']==1]['text'].str.split().map(lambda x: len(x))\r\n",
        "ax1.hist(tweet_len,color='red')\r\n",
        "ax1.set_title('disaster tweets')\r\n",
        "tweet_len=df_test[df_test['target']==0]['text'].str.split().map(lambda x: len(x))\r\n",
        "ax2.hist(tweet_len,color='green')\r\n",
        "ax2.set_title('Not disaster tweets')\r\n",
        "fig.suptitle('Words in test tweet')\r\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmEAAAFTCAYAAABxr9twAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3de5hddX3v8fdHAqiIhEtErgYFtdG2QiNSRYviBRCNpwcVbDV6sKk92NaqR1HPkYt320rrg0dF5RixChRrjYpSREC0ggRFFJESuTQJCJE7WgXke/5Yv4GdYSYzyVzWTPJ+Pc9+Zq3f+u21vntn5pfPXredqkKSJEnT6yF9FyBJkrQpMoRJkiT1wBAmSZLUA0OYJElSDwxhkiRJPTCESZIk9cAQJmnaJDk2yWc34HmXJzlgCkqSpN4YwqRNWJK3JfnasLarRmk7fHqre0BVPamqztuQ5yapJHtOtIbxBMgk1yZ57kS3tT42NNhK6p8hTNq0fQt4epLNAJLsBGwO7D2sbc/Wd9ySzJnkWiVpo2IIkzZtF9OFrqe0+WcC5wJXDmv7WVVdn2TnJMuS3JJkRZI/G1pR2yNzRpLPJrkDeHWSPZKcn+TOJGcDOwz0f2jre3OS25JcnGTHkYoc3MPUtnN6ks+09V6eZOEozxsKjj9McleSl7f2Q5Nc2rb770l+b+A5b02yuq37yiQHJjkIeDvw8raeH46wrVOA3YEvtz5vSbI0yZva8l3aXrmj2vzj2vv4kHHUtHOSLyRZk+SaJH/V2sesS9LMZQiTNmFVdTdwEfCs1vQs4ALg28PahsLMqcAqYGfgMOC9SZ4zsMpFwBnAXOCfgM8Bl9CFr3cBiwf6Lga2AXYDtgdeB/zXOEt/catlLrAMOHGU1zf0Gn6/qh5RVacl2Rs4Gfjztt2PA8uSbJnkCcDrgadW1dbAC4Brq+rrwHuB09p6fn+Ebb0S+E/gRa3PB4HzgQNalz8CruaB9/WPgAuq6r4xanoI8GXgh8AuwIHAG5K8YDx1SZq5DGGSzueBYPBMuhB2wbC285PsBjwDeGtV/bqqLgU+CbxqYF3frap/rar7gHnAU4H/U1W/qapv0YWJIffQBY49q+q3VXVJVd0xzpq/XVVnVtVvgVOA9QkfS4CPV9VFbbtLgd8A+wG/BbYEFiTZvKquraqfrce6hzsf2L8FqWcBH6R7D6ELYeePo6anAvOq6viquruqrgY+AfR2jp6kyWEIk/QtuqCwHd1/9lcB/053rth2wJNbn52BW6rqzoHnXke3d2bIyoHpnYFbq+qXw/oPOQU4Czg1yfVJPphk83HW/POB6V8BD12Pc9AeA7ypHfa7LcltdHvjdq6qFcAbgGOBm5KcmmTnca73QVqA+yXdod1nAl8Brm973AZD2Kg1tWU7D1v2dmDEQ7eSZg9DmKTv0h0W/DPgOwBtj9T1re36qrqmzW+XZOuB5+4OrB6Yr4HpG4Btk2w1rD9tG/dU1XFVtQB4OnAoa+9VmyorgfdU1dyBx8Or6vOtrs9V1f504aeADwyVPI51j9TnfLpDt1tU1eo2vxjYFrh0HDWtBK4ZtmzrqjpkPeqSNAMZwqRNXFX9F7AceCPdYcgh325t32r9VtLtIXtfO6n+94AjgRFvj1BV17X1HpdkiyT7Ay8aWp7k2Ul+t12FeQfd4cn7Jvv1ATcCjx2Y/wTwuiRPS2erJC9MsnWSJyR5TpItgV/TnaN238B65g+dSD/ObUEXul7PA+fVndfmv90Op66zJuB7wJ3tgoGHJdksyZOTPHU96pI0A/lHKwm6oPAouuA15ILWNnhriiOA+XR7xb4IHFNV31jHel8BPA24BTgG+MzAskfTncR/B3BFq+GUibyIURwLLG2H8l5WVcvp9vCdCNwKrABe3fpuCbwf+AXdIc9HAW9ry/65/bw5yfdH2db7gP/dtvXm1nY+sDUPvI/fBh4+MM+6ampB7VC6Q5rXtNo+Sbf3crx1SZqBUuWebEmSpOnmnjBJkqQeGMIkSZJ6YAiTJEnqgSFMkiSpB4YwSZKkHhjCJEmSemAIkyRJ6oEhTJIkqQeGMEmSpB4YwgRAkk8neXebfmaSK/uuSZKmSpJrkzy3Tb89ySf7rkmbHkOYHqSqLqiqJ0zV+gcD30xe5zi2eUCSVdO5TWlj1ALRTUm2Gmh7bZLzxvn885K8dkO3X1XvraoNfv5YBgPfTF7nOLZ5bJLPTuc2N3aGMM06Seb0XYOkSbcZ8Nd9FzHTpOP/1Rsp/2E3UUn2TvL9JHcmOQ146MCytfbwJHlrktWt75VJDmzt+yb5bpLbktyQ5MQkW7RlSXJC+3R7R5IfJXlykiXAnwBvSXJXki+3/jsn+UKSNUmuSfJXA9s/NskZST6b5A7g1cNey4PWmeQ1Q+tufa5K8s8D8yuTPKVNPzHJ2Uluaa/vZQP9tkzyd0n+M8mNST6W5GHtE/vXgJ3bNu9qr2HfJMvba74xyYcm4Z9L2hT8LfDmJHNHWpjk6UkuTnJ7+/n01v4e4JnAie3v8MRRnv/KJNcluTnJO4Ytu38PT5KHtrHm5ja2XZxkx7bsNUmuaGPh1Un+fGAdOyT5SnvOLUkuSPKQJKcAuwNfbvW9pfXfL8m/t/4/THLAwLrOS/KeJN8BfgU8dli9D1pnkqVJ3tSW75KkkhzV5h/XanpImz80yaVt2/+e5PcG1j3iWJzkIODtwMvbNn/Y2l/d3os7W/8/Gf2fWA9SVT42sQewBXAd8DfA5sBhwD3Au9vyA4BVbfoJwEpg5zY/H3hcm/4DYD9gTmu/AnhDW/YC4BJgLhDgd4Cd2rJPD22rzT+k9X1nq+2xwNXAC9ryY1t9L2l9HzbCaxq+zscCt7X+O7fXu2pg2a1t2Vbt9b2mvY69gV8AC1rfE4BlwHbA1sCXgfcNf58Gtvtd4JVt+hHAfn3/e/vwMdMfwLXAc4F/GRiHXguc16a3a3+zr2x/p0e0+e3b8vOA165j/QuAu4BnAVsCHwLuBZ7blh8LfLZN/3n7O3843d65PwAe2Za9EHhcG9P+iC4g7dOWvQ/4GN2YujldMMzg6xuoZxfgZuCQNg49r83PG3g9/wk8qb3ezUd7zwbm/wfw5Tb9CuBnwGkDy77UpvcGbgKe1l7f4rauLRnfWPzZgW1uBdwBPKHN7wQ8qe/fp9n0cE/Ypmk/ukHiH6rqnqo6A7h4lL6/pfvjXJBk86q6tqp+BlBVl1TVhVV1b1VdC3ycbmCCLjRtDTyRbiC6oqpuGGUbT6UbfI6vqrur6mrgE8DhA32+W1X/WlX3VdV/jfUC2zruBJ5CN/CeBVyf5Imtxguq6j7gUODaqvp/7XX8APgC8NIkAZYAf1NVt1TVncB7h9U13D3Ankl2qKq7qurCsWqVdL93An+ZZN6w9hcCV1XVKe3v9PPAT4EXjXO9hwFfqapvVdVvgP8D3DdK33uA7YE9q+q3bZy7A6CqvlpVP6vO+cC/0YWtoeftBDymjasXVEsmI/hT4MyqOrONaWcDy+lC2ZBPV9Xl7fXeM47XeD6wf9vb9Szgg8Az2rI/asuhG9M+XlUXtde3FPgN3f8L4xmLh7sPeHKSh1XVDVV1+ThqVWMI2zTtDKweNkBcN1LHqloBvIHuE9BNSU5NsjNAkse33e8/b4cJ3wvs0J73TeBE4CPteScleeQo9TyG7rDebUMPut3eOw70WbkBr/N8ur1Vz2rT59ENRoMD0mOApw3b9p8Ajwbm0X0avmRg2ddb+2iOBB4P/LQdxjh0A+qWNklV9WPgK8DRwxYN7c0edB3dHqXx2JmBMaSqfkm352kkp9B9aDs1yfVJPphkc4AkBye5sB3au40uNO3Qnve3wArg39rhueGvYdBj6D7oDY47+9OFuCHrNea1D8e/pPvg+Uy69/H6JE/gwWPem4Zteze692g8Y/HgNn8JvBx4HXBDkq+2D7oaJ0PYpukGYJe2p2fI7qN1rqrPVdX+dH+gBXygLfoo3afRvarqkXR/rBl43oer6g/oDgU8HvhfQ4uGbWIlcE1VzR14bF1Vg58KR/tEua7lQyHsmW36fB4cwlYC5w/b9iOq6i/oDkv+F93u9aFl21TVI0bbZlVdVVVHAI+ie5/OyMAVX5LGdAzwZ6wdsK6nG38G7Q6sbtNjjQ830AUNAJI8nG5v14O0vVjHVdUC4Ol0e8tflWRLur3kfwfsWFVzgTNpY15V3VlVb6qqxwIvBt6Ydv7sCPWtBE4ZNu5sVVXvHyxljNc02ph3GLBFVa1u84uBbYFLB7b9nmHbfnjbuzjWWDzSmHdWVT2PLkD+lG7PmcbJELZp+i7d+RB/lWTzJH8M7DtSxyRPSPKcNgD9mi6UDO3G35rufIC72qefvxh43lOTPK19gvxle+7Q825k7RNNvwfcme4CgIcl2SzdSfxPXY/XNHyd0A1Az6Y7h2wVcAFwEN3g+4PW5yvA49OdtLt5ezw1ye+0w5WfAE5I8qj2unZJ8oKBbW6fZJuB1/2nSea1597Wmkc77CFpmLb3/TTgrwaaz6T7O31FkjlJXk734e4rbflIf/+DzgAOTbJ/uouHjmeU//+SPDvJ7ybZjG58u4fub3gLulMz1gD3JjkYeP7A8w5Nsmf7cHs73akco415nwVelOQFbbx7aLoLonYd4+0ZNNqY93rgW23+vDb/7ar6bWv7BPC6Nj4nyVZJXphka8Yei28E5ueBE/x3TLKofdD8Dd15d45368EQtgmqqruBP6a7yvAWut3J/zJK9y2B99PtFfo53R6et7Vlb6Y7AfROuj/s0wae98jWdivdYYOb6XbXA3yK7hyz25L8axscDqXbjX5N29YngW0Yv7XW2V7nf9ANChe0+TvoTjL9ztCA1M7zej7dOQ/Xt9f4gfa6Ad5Kd4jhwnbI9Rt0FytQVT8FPg9c3ba7M13IuzzJXcA/AoeP5xw2SWs5nu6kbwCq6ma6MeJNdGPJW4BDq+oXrcs/AocluTXJh4evrJ2ndBTwObq9YrcCo93j79F0oe0OuouNzqfba3UnXTA8vT3/FXQX7QzZi258uIvug+7/rapz27L3Af+7jRNvrqqVwCK6owdr6PZA/S/W7//ktdbZ2s6n+3A8FMK+TXdKxdA8VbWcbk/jie11rKBdcT6OsXjoCvObk3y/1ftGurHzFrqjDPd/GNfYhq7ckCRJ0jRyT5gkSVIPDGGSJEk9MIRJkiT1wBAmSZLUA0OYJElSD+b0XcCG2GGHHWr+/Pl9lyFpmlxyySW/qKp1fVPBrOH4JW16RhvDZmUImz9/PsuXL++7DEnTJMmIX6s1Gzl+SZue0cYwD0dKkiT1wBAmSZLUA0OYJElSDwxhkiRJPTCESZIk9cAQJkmS1ANDmCRJUg8MYZIkST0whEmSJPXAECZJA5LMTXJGkp8muSLJHybZLsnZSa5qP7dtfZPkw0lWJLksyT591y9p9jCESdLa/hH4elU9Efh94ArgaOCcqtoLOKfNAxwM7NUeS4CPTn+5kmarWfndkZI0FZJsAzwLeDVAVd0N3J1kEXBA67YUOA94K7AI+ExVFXBh24u2U1XdMM2laxbIcZmyddcxNWXr1tRxT5gkPWAPYA3w/5L8IMknk2wF7DgQrH4O7NimdwFWDjx/VWuTpDEZwiTpAXOAfYCPVtXewC954NAjAG2v13rtdkiyJMnyJMvXrFkzacVKmt0MYZL0gFXAqqq6qM2fQRfKbkyyE0D7eVNbvhrYbeD5u7a2tVTVSVW1sKoWzps3b8qKlzS7GMIkqamqnwMrkzyhNR0I/ARYBixubYuBL7XpZcCr2lWS+wG3ez6YpPHyxHxJWttfAv+UZAvgauA1dB9YT09yJHAd8LLW90zgEGAF8KvWV5LGxRAmSQOq6lJg4QiLDhyhbwFHTXlRkjZKHo6UJEnqgSFMkiSpB4YwSZKkHhjCJEmSemAIkyRJ6oEhTJIkqQeGMEmSpB4YwiRJknpgCJMkSeqBIUySJKkHhjBJkqQeGMIkSZJ6YAiTJEnqwaSEsCQHJbkyyYokR4+wfMskp7XlFyWZP2z57knuSvLmyahHkiRppptwCEuyGfAR4GBgAXBEkgXDuh0J3FpVewInAB8YtvxDwNcmWoskSdJsMRl7wvYFVlTV1VV1N3AqsGhYn0XA0jZ9BnBgkgAkeQlwDXD5JNQiSZI0K0xGCNsFWDkwv6q1jdinqu4Fbge2T/II4K3AcZNQhyRJ0qzR94n5xwInVNVdY3VMsiTJ8iTL16xZM/WVSZIkTaE5k7CO1cBuA/O7traR+qxKMgfYBrgZeBpwWJIPAnOB+5L8uqpOHL6RqjoJOAlg4cKFNQl1S5Ik9WYyQtjFwF5J9qALW4cDrxjWZxmwGPgucBjwzaoq4JlDHZIcC9w1UgCTJEna2Ew4hFXVvUleD5wFbAacXFWXJzkeWF5Vy4BPAackWQHcQhfUJEmaUXJc+i5Bm5DJ2BNGVZ0JnDms7Z0D078GXjrGOo6djFokSRs3g5I2Fn2fmC9JkrRJMoRJkiT1wBAmSZLUA0OYJElSDwxhkiRJPTCESZIk9WBSblEhSZL6M5W37ahj/JKaqeKeMEmSpB4YwiRJknpgCJMkSeqBIUySJKkHhjBJkqQeGMIkaUCSa5P8KMmlSZa3tu2SnJ3kqvZz29aeJB9OsiLJZUn26bd6SbOJIUySHuzZVfWUqlrY5o8GzqmqvYBz2jzAwcBe7bEE+Oi0Vypp1jKESdLYFgFL2/RS4CUD7Z+pzoXA3CQ79VGgpNnHECZJayvg35JckmRJa9uxqm5o0z8HdmzTuwArB567qrVJ0pi8Y74krW3/qlqd5FHA2Ul+OriwqirJet1CvIW5JQC777775FUqaVZzT5gkDaiq1e3nTcAXgX2BG4cOM7afN7Xuq4HdBp6+a2sbvs6TqmphVS2cN2/eVJYvaRYxhElSk2SrJFsPTQPPB34MLAMWt26LgS+16WXAq9pVkvsBtw8ctpSkdfJwpCQ9YEfgi0mgGx8/V1VfT3IxcHqSI4HrgJe1/mcChwArgF8Br5n+kiXNVoYwSWqq6mrg90dovxk4cIT2Ao6ahtIkbYQ8HClJktQDQ5gkSVIPDGGSJEk9MIRJkiT1wBAmSZLUA0OYJElSDwxhkiRJPTCESZIk9cAQJkmS1ANDmCRJUg8MYZIkST0whEmSJPXAECZJktQDQ5gkSVIPDGGSJEk9MIRJkiT1wBAmSZLUA0OYJElSDwxhkiRJPTCESZIk9WBSQliSg5JcmWRFkqNHWL5lktPa8ouSzG/tz0tySZIftZ/PmYx6JEmSZroJh7AkmwEfAQ4GFgBHJFkwrNuRwK1VtSdwAvCB1v4L4EVV9bvAYuCUidYjSZI0G0zGnrB9gRVVdXVV3Q2cCiwa1mcRsLRNnwEcmCRV9YOqur61Xw48LMmWk1CTJEnSjDYZIWwXYOXA/KrWNmKfqroXuB3Yflif/w58v6p+M9JGkixJsjzJ8jVr1kxC2ZIkSf2ZESfmJ3kS3SHKPx+tT1WdVFULq2rhvHnzpq84SZKkKTAZIWw1sNvA/K6tbcQ+SeYA2wA3t/ldgS8Cr6qqn01CPZIkSTPeZISwi4G9kuyRZAvgcGDZsD7L6E68BzgM+GZVVZK5wFeBo6vqO5NQiyRJ0qww4RDWzvF6PXAWcAVwelVdnuT4JC9u3T4FbJ9kBfBGYOg2Fq8H9gTemeTS9njURGuSJEma6eZMxkqq6kzgzGFt7xyY/jXw0hGe927g3ZNRgyRJ0mwyI07MlyRJ2tQYwiRJknpgCJMkSeqBIUySJKkHhjBJkqQeGMIkaZgkmyX5QZKvtPk9klyUZEWS09o9EUmyZZtf0ZbP77NuSbOLIUySHuyv6e57OOQDwAlVtSdwK3Bkaz8SuLW1n9D6SdK4GMIkaUD7KrUXAp9s8wGeA5zRuiwFXtKmF7V52vIDW39JGpMhTJLW9g/AW4D72vz2wG3t20EAVgG7tOldgJVw/7eH3N76ryXJkiTLkyxfs2bNVNYuaRYxhElSk+RQ4KaqumQy11tVJ1XVwqpaOG/evMlctaRZbFK+tkiSNhLPAF6c5BDgocAjgX8E5iaZ0/Z27Qqsbv1XA7sBq5LMAbYBbp7+siXNRu4Jk6Smqt5WVbtW1XzgcOCbVfUnwLnAYa3bYuBLbXpZm6ct/2ZV1TSWLGkWM4RJ0tjeCrwxyQq6c74+1do/BWzf2t8IHN1TfZJmIQ9HStIIquo84Lw2fTWw7wh9fg28dFoLk7TRcE+YJElSDwxhkiRJPTCESZIk9cAQJkmS1ANDmCRJUg8MYZIkST0whEmSJPXAECZJktQDb9YqSZp0OS59lyDNeO4JkyRJ6oEhTJIkqQeGMEmSpB4YwiRJknpgCJMkSeqBIUySJKkHhjBJkqQeGMIkSZJ6YAiTJEnqgSFMkiSpB4YwSZKkHhjCJEmSemAIkyRJ6oEhTJIkqQeGMEmSpB4YwiRJknpgCJMkSerBpISwJAcluTLJiiRHj7B8yySnteUXJZk/sOxtrf3KJC+YjHokSZJmugmHsCSbAR8BDgYWAEckWTCs25HArVW1J3AC8IH23AXA4cCTgIOA/9vWJ0mStFGbjD1h+wIrqurqqrobOBVYNKzPImBpmz4DODBJWvupVfWbqroGWNHWJ0mStFGbjBC2C7ByYH5VaxuxT1XdC9wObD/O50qSJG10Zs2J+UmWJFmeZPmaNWv6LkeSJGlCJiOErQZ2G5jftbWN2CfJHGAb4OZxPheAqjqpqhZW1cJ58+ZNQtmSJEn9mYwQdjGwV5I9kmxBd6L9smF9lgGL2/RhwDerqlr74e3qyT2AvYDvTUJNkiRJM9qEQ1g7x+v1wFnAFcDpVXV5kuOTvLh1+xSwfZIVwBuBo9tzLwdOB34CfB04qqp+O9GaJGlDJHloku8l+WGSy5Mc19r3aLfXWdFut7NFax/19juSNJY5k7GSqjoTOHNY2zsHpn8NvHSU574HeM9k1CFJE/Qb4DlVdVeSzYFvJ/ka3YfHE6rq1CQfo7vtzkcZuP1OksPpbr/z8r6KlzS7zJoT8yVpqlXnrja7eXsU8By62+tAd7udl7Tp0W6/I0ljMoRJ0oAkmyW5FLgJOBv4GXBbO/UC1r6Vzmi335GkMRnCJGlAVf22qp5Cd7X2vsATJ7pOb7EjaSSGMEkaQVXdBpwL/CEwt91eB9a+lc5ot98Zvi5vsSPpQQxhktQkmZdkbpt+GPA8uqu+z6W7vQ50t9v5Upse7fY7kjSmSbk6UpI2EjsBS5NsRvch9fSq+kqSnwCnJnk38AO62+7Qfp7Sbr9zC919EiVpXAxhktRU1WXA3iO0X013ftjw9lFvvyNJY/FwpCRJUg8MYZIkST0whEmSJPXAECZJktQDQ5gkSVIPDGGSJEk9MIRJkiT1wBAmSZLUA0OYJElSDwxhkiRJPfBriyRJ0qhyXKZs3XXMpv199+4JkyRJ6oEhTJIkqQeGMEmSpB4YwiRJknpgCJMkSeqBIUySJKkHhjBJkqQeGMIkSZJ6YAiTJEnqgSFMkiSpB4YwSZKkHhjCJEmSemAIkyRJ6oEhTJIkqQeGMEmSpB4YwiRJknpgCJMkSeqBIUySJKkHhjBJkqQeGMIkSZJ6YAiTJEnqgSFMkiSpB4YwSZKkHkwohCXZLsnZSa5qP7cdpd/i1ueqJItb28OTfDXJT5NcnuT9E6lFkiYqyW5Jzk3ykzYu/XVrH3GsS+fDSVYkuSzJPv2+AkmzyUT3hB0NnFNVewHntPm1JNkOOAZ4GrAvcMxAWPu7qnoisDfwjCQHT7AeSZqIe4E3VdUCYD/gqCQLGH2sOxjYqz2WAB+d/pIlzVYTDWGLgKVteinwkhH6vAA4u6puqapbgbOBg6rqV1V1LkBV3Q18H9h1gvVI0garqhuq6vtt+k7gCmAXRh/rFgGfqc6FwNwkO01z2ZJmqYmGsB2r6oY2/XNgxxH67AKsHJhf1drul2Qu8CK6T5gjSrIkyfIky9esWTOxqiVpDEnm0+2lv4jRx7oxxzdJGs2csTok+Qbw6BEWvWNwpqoqSa1vAUnmAJ8HPlxVV4/Wr6pOAk4CWLhw4XpvR5LGK8kjgC8Ab6iqO5Lcv2xDxrokS+gOV7L77rtPZqmSZrExQ1hVPXe0ZUluTLJTVd3QdsHfNEK31cABA/O7AucNzJ8EXFVV/zCuiiVpCiXZnC6A/VNV/UtrHm2sWw3sNvD0XVvbWvwQKWkkEz0cuQxY3KYXA18aoc9ZwPOTbNtOyH9+ayPJu4FtgDdMsA5JmrB0u7w+BVxRVR8aWDTaWLcMeFW7SnI/4PaBw5aStE5j7gkbw/uB05McCVwHvAwgyULgdVX12qq6Jcm7gIvbc45vbbvSHdL8KfD9trv/xKr65ARrkqQN9QzglcCPklza2t7OKGMdcCZwCLAC+BXwmuktd2JyXMbuJGnKTCiEVdXNwIEjtC8HXjswfzJw8rA+qwBHAEkzRlV9m9HHpZHGugKOmtKiJG20vGO+JElSDwxhkiRJPTCESZIk9cAQJkmS1ANDmCRJUg8MYZIkST0whEmSJPXAECZJktQDQ5gkSVIPDGGSJEk9MIRJkiT1wBAmSZLUA0OYJElSDwxhkiRJPTCESZIk9cAQJkmS1ANDmCRJUg8MYZIkST0whEmSJPXAECZJktQDQ5gkSVIP5vRdgCRJ2jTluEzZuuuYmrJ1Txb3hEmSJPXAECZJktQDQ5gkSVIPDGGSJEk9MIRJkiT1wBAmSZLUA0OYJElSDwxhkiRJPTCESZIk9cAQJkmS1ANDmCRJUg8MYZIkST0whElSk+TkJDcl+fFA23ZJzk5yVfu5bWtPkg8nWZHksiT79Fe5pNnIECZJD/g0cNCwtqOBc6pqL+CcNg9wMLBXeywBPjpNNUraSBjCJKmpqm8BtwxrXgQsbdNLgZcMtH+mOhcCc5PsND2VStoYGMIkad12rKob2vTPgR3b9C7AyoF+q1qbJI2LIUySxqmqCqj1fV6SJUmWJ1m+Zs2aKahM0mxkCJOkdbtx6DBj+3lTa18N7DbQb9fW9iBVdVJVLayqhfPmzZvSYiXNHhMKYaNdNUKEoMQAAAlGSURBVDRCv8Wtz1VJFo+wfNng1UiSNIMsA4bGrcXAlwbaX9WuktwPuH3gsKUkjWmie8JGu2rofkm2A44BngbsCxwzGNaS/DFw1wTrkKQJS/J54LvAE5KsSnIk8H7geUmuAp7b5gHOBK4GVgCfAP5nDyVLmsXmTPD5i4AD2vRS4DzgrcP6vAA4u6puAUhyNt0l4J9P8gjgjXSXd58+wVokaUKq6ohRFh04Qt8CjpraiiRtzCa6J2y0q4YGresKoncBfw/8aoJ1SJIkzSpj7glL8g3g0SMsesfgTFVVknFfNZTkKcDjqupvkswfR/8ldHvM2H333ce7GUmSpBlpzBBWVc8dbVmSG5PsVFU3DLtqaNBqHjhkCd0VROcBfwgsTHJtq+NRSc6rqgMYQVWdBJwEsHDhwvW+RFySJGkmmejhyNGuGhp0FvD8JNu2E/KfD5xVVR+tqp2raj6wP/AfowUwSZKkjc1EQ9iIVw0lWZjkkwDthPx3ARe3x/FDJ+lLkiRtqiZ0dWRV3czIVw0tB147MH8ycPI61nMt8OSJ1CJJkjSbeMd8SZKkHhjCJEmSemAIkyRJ6oEhTJIkqQeGMEmSpB4YwiRJknow0S/wliRNoRyXvkuQNEXcEyZJktQDQ5gkSVIPDGGSJEk9MIRJkiT1wBAmSZLUA0OYJElSDwxhkiRJPTCESZIk9cAQJkmS1ANDmCRJUg8MYZIkST0whEmSJPXAL/CWJEkbnRyXKVt3HVOTsh73hEmSJPXAECZJktQDQ5gkSVIPDGGSJEk9MIRJkiT1wBAmSZLUA0OYJElSDwxhkiRJPTCESdIEJDkoyZVJViQ5uu96JM0ehjBJ2kBJNgM+AhwMLACOSLKg36okzRaGMEnacPsCK6rq6qq6GzgVWNRzTZJmCUOYJG24XYCVA/OrWpskjWlWfoH3JZdc8osk142z+w7AL6aynili3dNvtta+KdT9mKksZKolWQIsabN3JblyYPFM/febqXXBzK1tptYFM7e2WVlXjl3vLwcfcQyblSGsquaNt2+S5VW1cCrrmQrWPf1ma+3W3avVwG4D87u2trVU1UnASSOtYKa+DzO1Lpi5tc3UumDm1rap1+XhSEnacBcDeyXZI8kWwOHAsp5rkjRLzMo9YZI0E1TVvUleD5wFbAacXFWX91yWpFliUwhhIx4CmAWse/rN1tqtu0dVdSZw5gRWMVPfh5laF8zc2mZqXTBza9uk60pVTcd2JEmSNMBzwiRJknqw0YSwsb46JMmWSU5ryy9KMn/6q3ywcdT9xiQ/SXJZknOSzIhL9cf7VS1J/nuSSjIjrn4ZT91JXtbe88uTfG66axzJOH5Pdk9ybpIftN+VQ/qoc7gkJye5KcmPR1meJB9ur+uyJPtMd419mclfd5Tk2iQ/SnJpkuU91/Kg36Ek2yU5O8lV7ee2M6SuY5Osbu/bpX38HSbZrY0FQ2PYX7f2Xt+zddQ1E96zhyb5XpIfttqOa+17tLywouWHLSZ941U16x90J8T+DHgssAXwQ2DBsD7/E/hYmz4cOG2W1P1s4OFt+i9mS92t39bAt4ALgYWzoW5gL+AHwLZt/lGzpO6TgL9o0wuAa/uuu9XyLGAf4MejLD8E+BoQYD/gor5rnin/pj3Xdy2wQ991tFoe9DsEfBA4uk0fDXxghtR1LPDmnt+vnYB92vTWwH+0MaHX92wddc2E9yzAI9r05sBFbTw6HTi8tX9saIydzMfGsidsPF8dsghY2qbPAA5Mst53W5tkY9ZdVedW1a/a7IV09yHq23i/quVdwAeAX09nceswnrr/DPhIVd0KUFU3TXONIxlP3QU8sk1vA1w/jfWNqqq+Bdyyji6LgM9U50JgbpKdpqe6Xvl1R+M0yu/Q4Hi+FHjJtBbFuH63e1FVN1TV99v0ncAVdN/i0Ot7to66etfGn7va7ObtUcBz6PICTNF7trGEsPF8dcj9farqXuB2YPtpqW506/uVJ0fS7TXo25h1t8NKu1XVV6ezsDGM5/1+PPD4JN9JcmGSg6atutGNp+5jgT9NsoruSr2/nJ7SJmxT/dqfmf66C/i3JJeku9v/TLNjVd3Qpn8O7NhnMcO8vh1aP7mPw6SD2mk3e9Pt2Zkx79mwumAGvGdJNktyKXATcDbdnurbWl6AKfob3VhC2EYvyZ8CC4G/7buWsSR5CPAh4E1917IB5tAdkjwAOAL4RJK5vVY0PkcAn66qXekO8Z3S/h2kDbF/Ve0DHAwcleRZfRc0muqOFc2Uy/w/CjwOeApwA/D3fRWS5BHAF4A3VNUdg8v6fM9GqGtGvGdV9duqegrd0aZ9gSdOx3Y3lkF6PF8dcn+fJHPoDtncPC3VjW5cX3mS5LnAO4AXV9Vvpqm2dRmr7q2BJwPnJbmW7tj6shlwcv543u9VwLKquqeqrqE7b2GvaapvNOOp+0i68xeoqu8CD6X77rOZblx/AxuhGf26q2p1+3kT8EW6/5RmkhuHDlu3nzPhtAGq6sb2n/l9wCfo6X1Lsjld0PmnqvqX1tz7ezZSXTPlPRtSVbcB5wJ/SHd6xND9VKfkb3RjCWHj+eqQZcDiNn0Y8M32aaBPY9adZG/g43QBbEYMNIxRd1XdXlU7VNX8qppPdy7bi6uq16usGN/vyb/S7QUjyQ50hyevns4iRzCeuv8TOBAgye/QhbA101rlhlkGvKpdJbkfcPvAIZON2Yz9uqMkWyXZemgaeD4w4tWtPRoczxcDX+qxlvsNO5/xv9HD+9bOdf4UcEVVfWhgUa/v2Wh1zZD3bN7QEY8kDwOeR3fO2rl0eQGm6j3r40qEqXjQHYL5D7rjuO9obcfT/ecP3X9K/wysAL4HPLbvmsdZ9zeAG4FL22NZ3zWPp+5hfc9jBlwdOc73O3SHUn8C/Ih2ZUzfj3HUvQD4Dt1VdpcCz++75lbX5+kOMdxDt5fxSOB1wOsG3u+PtNf1o5nye9LXv+lMeNBdsfnD9ri879pG+R3aHjgHuKqNkdvNkLpOab/Hl9GFnp16qGt/ukONlw38v3FI3+/ZOuqaCe/Z79FdFX8ZXQh8Z2t/LF1eWEGXH7ac7G17x3xJkqQebCyHIyVJkmYVQ5gkSVIPDGGSJEk9MIRJkiT1wBAmSZLUA0OYJElSDwxhkiRJPTCESZIk9eD/A8E7ab7s2FpgAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 720x360 with 2 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "q39ccyTiI5jN"
      },
      "source": [
        "Observation: \r\n",
        "\r\n",
        "Tweets in both train and test set have about 120 to 140 characters and 20 to 30 words.\r\n",
        "\r\n",
        "For training set, the distributions of both characters count and word counts are similar for disaster and non-disaster tweets."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dm7_0wyhJ3NB"
      },
      "source": [
        "## 1.4 Stopwords in Training Set"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "C15sTzR7J6oN"
      },
      "source": [
        "# helper function to create training corpus from text\r\n",
        "def create_corpus(target):\r\n",
        "    corpus=[]\r\n",
        "    \r\n",
        "    for x in df_train[df_train['target']==target]['text'].str.split():\r\n",
        "        for i in x:\r\n",
        "            corpus.append(i)\r\n",
        "    return corpus"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 286
        },
        "id": "nNVFjcjmKGqH",
        "outputId": "d03cc89c-b9cb-42a1-e6da-5515f4c67184"
      },
      "source": [
        "# First we analyze non-disaster tweets (class=0)\r\n",
        "corpus=create_corpus(0)\r\n",
        "\r\n",
        "dic=defaultdict(int)\r\n",
        "for word in corpus:\r\n",
        "    if word in stop:\r\n",
        "        dic[word]+=1\r\n",
        "        \r\n",
        "top=sorted(dic.items(), key=lambda x:x[1],reverse=True)[:10] \r\n",
        "\r\n",
        "x,y=zip(*top)\r\n",
        "plt.bar(x,y)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<BarContainer object of 10 artists>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 35
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD8CAYAAACb4nSYAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAefklEQVR4nO3dbXBU5cHG8f8mS6J5z24MGF6KgSCCYMCAkY4Qws60A7Zl1NJphY4iZTAKDUFE0TK2CsTSmBgIOi1MdNCpIhVUqpXuhEBLZLoxCUJA3scBBUJ2Y8wLYJI9zweGfaAQk2x2E+y5fp/Ys2fPdd/ZcOXO2T0bi2EYBiIiYgohvT0AERHpOSp9ERETUemLiJiISl9ExERU+iIiJqLSFxExEWtHO6xdu5aKigpiY2PJy8vzbf/oo4/4+OOPCQkJYezYscycOROAzZs3U1JSQkhICA8//DCpqakAVFVVUVxcjNfrZcqUKUyfPj1IUxIRkfZ0WPoZGRn8+Mc/pqioyLdt3759lJeXs2rVKvr06UN9fT0AJ0+epKysjJdeeom6ujqef/55Xn75ZQDWr1/Ps88+i91u5+mnnyYtLY0BAwYEaVoiInItHZb+iBEjqKmpuWLbtm3b+NnPfkafPn0AiI2NBcDlcjFhwgT69OlDYmIi/fr148iRIwD069ePvn37AjBhwgRcLpdKX0Skh3VY+tdy6tQpPv/8c9566y369OnDrFmzGDp0KB6Ph5SUFN9+NpsNj8cDgN1u92232+0cPnz4msd2Op04nU4AcnNz/RmeiIi0w6/S93q9NDY2snz5co4ePUp+fj5r1qwJyIAcDgcOh8N3+6uvvgrIcTsjISGB2traHstTtrKVbZ78nsxOSkpq9z6/St9mszF+/HgsFgtDhw4lJCSEhoYGbDYbbrfbt5/H48FmswFcsd3tdvu2i4hIz/HrLZvjxo2juroauLgSb21tJTo6mrS0NMrKymhpaaGmpoZTp04xdOhQhgwZwqlTp6ipqaG1tZWysjLS0tICOhEREelYhyv9goIC9u/fT0NDA/PmzWPGjBlkZmaydu1aFi1ahNVq5bHHHsNisTBw4EDuvvtucnJyCAkJ4ZFHHiEk5OLPldmzZ7N8+XK8Xi+TJ09m4MCBQZ+ciIhcyXK9f7SyzukrW9nK/l/Iv17O6euKXBERE1Hpi4iYiEpfRMREVPoiIiai0hcRMRG/Ls76vmj7zU+7tP8ZPzJC//K+H48SEekdWumLiJiISl9ExERU+iIiJqLSFxExEZW+iIiJqPRFRExEpS8iYiIqfRERE1Hpi4iYiEpfRMREVPoiIiai0hcRMZEOP3Bt7dq1VFRUEBsbS15e3hX3ffDBB2zYsIF169YRExODYRgUFxdTWVlJeHg4WVlZJCcnA1BaWsq7774LwH333UdGRkbgZyMiIt+pw5V+RkYGS5cuvWp7bW0tn332GQkJCb5tlZWVnD59msLCQubOncu6desAaGxsZNOmTaxYsYIVK1awadMmGhsbAzgNERHpjA5Lf8SIEURFRV21/fXXX+fBBx/EYrH4tpWXlzNx4kQsFgvDhg2jqamJuro6qqqqGD16NFFRUURFRTF69GiqqqoCOxMREemQX+f0XS4XNpuNwYMHX7Hd4/FcsfK32+14PB48Hg92u9233Waz4fF4/BuxiIj4rct/ROXChQts3ryZZ599Nhjjwel04nQ6AcjNzb3ih0hX+fNHUbqqO+O7nNVqDdixlK1sZV9/+b09d984uvqAM2fOUFNTw+LFiwFwu90sWbKElStXYrPZqK2t9e3rdrux2WzYbDb279/v2+7xeBgxYsQ1j+9wOHA4HL7blx/vehSo8SUkJPTaXJWtbDNk93Z+T2YnJSW1e1+XT+8MGjSIdevWUVRURFFREXa7nRdffJG4uDjS0tLYuXMnhmFw6NAhIiIiiI+PJzU1lT179tDY2EhjYyN79uwhNTW1W5MSEZGu63ClX1BQwP79+2loaGDevHnMmDGDzMzMa+47ZswYKioqWLBgAWFhYWRlZQEQFRXF/fffz9NPPw3AAw88cM0Xh0VEJLg6LP3s7OzvvL+oqMj3b4vFwpw5c665X2ZmZrs/LEREpGfoilwRERNR6YuImIhKX0TERFT6IiImotIXETERlb6IiImo9EVETESlLyJiIip9ERETUemLiJiISl9ExERU+iIiJqLSFxExEZW+iIiJqPRFRExEpS8iYiIqfRERE1Hpi4iYiEpfRMREOvwbuWvXrqWiooLY2Fjy8vIA2LBhA59++ilWq5W+ffuSlZVFZGQkAJs3b6akpISQkBAefvhhUlNTAaiqqqK4uBiv18uUKVOYPn16EKclIiLX0uFKPyMjg6VLl16xbfTo0eTl5fGnP/2Jm2++mc2bNwNw8uRJysrKeOmll3jmmWdYv349Xq8Xr9fL+vXrWbp0Kfn5+ezatYuTJ08GZ0YiItKuDkt/xIgRREVFXbHtjjvuIDQ0FIBhw4bh8XgAcLlcTJgwgT59+pCYmEi/fv04cuQIR44coV+/fvTt2xer1cqECRNwuVxBmI6IiHyXDk/vdKSkpIQJEyYA4PF4SElJ8d1ns9l8PxDsdrtvu91u5/Dhw9c8ntPpxOl0ApCbm0tCQoLfYzvj9yM7rzvju5zVag3YsZStbGVff/m9PXffOLrz4HfffZfQ0FDuueeeQI0Hh8OBw+Hw3a6trQ3YsYMhUONLSEjotbkqW9lmyO7t/J7MTkpKavc+v0u/tLSUTz/9lGXLlmGxWICLK3u32+3bx+PxYLPZAK7Y7na7fdtFRKTn+PWWzaqqKt577z2WLFlCeHi4b3taWhplZWW0tLRQU1PDqVOnGDp0KEOGDOHUqVPU1NTQ2tpKWVkZaWlpAZuEiIh0Tocr/YKCAvbv309DQwPz5s1jxowZbN68mdbWVp5//nkAUlJSmDt3LgMHDuTuu+8mJyeHkJAQHnnkEUJCLv5cmT17NsuXL8fr9TJ58mQGDhwY3JmJiMhVOiz97Ozsq7ZlZma2u/99993Hfffdd9X2sWPHMnbs2C4OT0REAklX5IqImIhKX0TERFT6IiImotIXETERlb6IiIl0+2MY5NrafvPTLu3vz0dGhP7lfT8eJSJmppW+iIiJaKX/P0i/ZYhIe7TSFxExEZW+iIiJqPRFRExEpS8iYiIqfRERE1Hpi4iYiEpfRMREVPoiIiai0hcRMRGVvoiIiXT4MQxr166loqKC2NhY8vLyAGhsbCQ/P5+zZ89y0003sXDhQqKiojAMg+LiYiorKwkPDycrK4vk5GQASktLeffdd4GLf1IxIyMjeLMSEZFr6nCln5GRwdKlS6/YtmXLFkaNGkVhYSGjRo1iy5YtAFRWVnL69GkKCwuZO3cu69atAy7+kNi0aRMrVqxgxYoVbNq0icbGxiBMR0REvkuHpT9ixAiioqKu2OZyuZg0aRIAkyZNwuVyAVBeXs7EiROxWCwMGzaMpqYm6urqqKqqYvTo0URFRREVFcXo0aOpqqoKwnREROS7+PUpm/X19cTHxwMQFxdHfX09AB6Ph4SEBN9+drsdj8eDx+PBbrf7tttsNjwezzWP7XQ6cTqdAOTm5l5xvK7y59Mju6q98Zk1u6usVmvAjqVsZV/P+b09d984unsAi8WCxWIJxFgAcDgcOBwO3+3a2tqAHTsYenN8/wvZCQkJvTYPZZsru7fzezI7KSmp3fv8evdObGwsdXV1ANTV1RETEwNcXMFfPim3243NZsNms+F2u33bPR4PNpvNn2gREekGv0o/LS2NHTt2ALBjxw7GjRvn275z504Mw+DQoUNEREQQHx9Pamoqe/bsobGxkcbGRvbs2UNqamrgZiEiIp3S4emdgoIC9u/fT0NDA/PmzWPGjBlMnz6d/Px8SkpKfG/ZBBgzZgwVFRUsWLCAsLAwsrKyAIiKiuL+++/n6aefBuCBBx646sVhEREJvg5LPzs7+5rbly1bdtU2i8XCnDlzrrl/ZmYmmZmZXRyeiIgEkq7IFRExEZW+iIiJqPRFRExEpS8iYiIqfRERE1Hpi4iYiEpfRMREVPoiIiai0hcRMRGVvoiIiaj0RURMRKUvImIiKn0RERNR6YuImIhKX0TERFT6IiIm0u0/jC5yubbf/LRL+5/xIyP0L+9fd9ki3xda6YuImEi3Vvpbt26lpKQEi8XCwIEDycrK4uuvv6agoICGhgaSk5OZP38+VquVlpYW1qxZw7Fjx4iOjiY7O5vExMRAzUNERDrB75W+x+Pho48+Ijc3l7y8PLxeL2VlZbzxxhtMmzaN1atXExkZSUlJCQAlJSVERkayevVqpk2bxptvvhmwSYiISOd06/SO1+vl22+/pa2tjW+//Za4uDiqq6tJT08HICMjA5fLBUB5eTkZGRkApKens2/fPgzD6N7oRUSkS/w+vWOz2fjJT37Co48+SlhYGHfccQfJyclEREQQGhrq28fj8QAXfzOw2+0AhIaGEhERQUNDAzExMVcc1+l04nQ6AcjNzSUhIcHfIfr1Ql1XtTc+ZZsru6usVmvAjqXs70d+b8/dNw5/H9jY2IjL5aKoqIiIiAheeuklqqqquj0gh8OBw+Hw3a6tre32MYOpN8en7O9vdkJCQq/Nw6zZvZ3fk9lJSUnt3uf36Z29e/eSmJhITEwMVquVu+66i4MHD9Lc3ExbWxtwcXVvs9mAi6t+t9sNQFtbG83NzURHR/sbLyIifvC79BMSEjh8+DAXLlzAMAz27t3LgAEDGDlyJLt37wagtLSUtLQ0AO68805KS0sB2L17NyNHjsRisXR/BiIi0ml+n95JSUkhPT2dJUuWEBoayuDBg3E4HIwdO5aCggLeeustbrnlFjIzMwHIzMxkzZo1zJ8/n6ioKLKzswM2CRER6ZxuvU9/xowZzJgx44ptffv2ZeXKlVftGxYWRk5OTnfiRESkm3RFroiIiaj0RURMRKUvImIiKn0RERNR6YuImIg+T18kAPRZ/vJ9oZW+iIiJqPRFRExEpS8iYiIqfRERE1Hpi4iYiEpfRMREVPoiIiai0hcRMRGVvoiIiaj0RURMRKUvImIiKn0RERPp1geuNTU18eqrr3LixAksFguPPvooSUlJ5Ofnc/bsWW666SYWLlxIVFQUhmFQXFxMZWUl4eHhZGVlkZycHKh5iIhIJ3RrpV9cXExqaioFBQWsWrWK/v37s2XLFkaNGkVhYSGjRo1iy5YtAFRWVnL69GkKCwuZO3cu69atC8gERESk8/wu/ebmZg4cOEBmZiYAVquVyMhIXC4XkyZNAmDSpEm4XC4AysvLmThxIhaLhWHDhtHU1ERdXV0ApiAiIp3l9+mdmpoaYmJiWLt2LV988QXJyck89NBD1NfXEx8fD0BcXBz19fUAeDweEhISfI+32+14PB7fvpc4nU6cTicAubm5Vzymq/z5zPKuam98ylZ2T2V3ldVqDdixvk/ZvZ3f23P3jcPfB7a1tXH8+HFmz55NSkoKxcXFvlM5l1gsFiwWS5eO63A4cDgcvtu1tbX+DrFH9Ob4lK1sfyQkJPTaPHozu7fzezI7KSmp3fv8Pr1jt9ux2+2kpKQAkJ6ezvHjx4mNjfWdtqmrqyMmJgYAm812xYTdbjc2m83feBER8YPfpR8XF4fdbuerr74CYO/evQwYMIC0tDR27NgBwI4dOxg3bhwAaWlp7Ny5E8MwOHToEBEREVed2hERkeDq1ls2Z8+eTWFhIa2trSQmJpKVlYVhGOTn51NSUuJ7yybAmDFjqKioYMGCBYSFhZGVlRWQCYiISOd1q/QHDx5Mbm7uVduXLVt21TaLxcKcOXO6EyciIt2kK3JFRExEpS8iYiIqfRERE1Hpi4iYSLdeyBWR3tf2m592aX9/rh4O/cv7fjxKrkda6YuImIhW+iLiN/2W8f2jlb6IiImo9EVETESlLyJiIip9ERETUemLiJiISl9ExERU+iIiJqL36YvI91JXrxGArl8n8L94jYBW+iIiJqKVvohIF32ff8vQSl9ExES6vdL3er089dRT2Gw2nnrqKWpqaigoKKChoYHk5GTmz5+P1WqlpaWFNWvWcOzYMaKjo8nOziYxMTEQcxARkU7q9kr/ww8/pH///r7bb7zxBtOmTWP16tVERkZSUlICQElJCZGRkaxevZpp06bx5ptvdjdaRES6qFul73a7qaioYMqUKQAYhkF1dTXp6ekAZGRk4HK5ACgvLycjIwOA9PR09u3bh2EY3YkXEZEu6tbpnddee42ZM2dy7tw5ABoaGoiIiCA0NBQAm82Gx+MBwOPxYLfbAQgNDSUiIoKGhgZiYmKuOKbT6cTpdAKQm5tLQkKC3+Pz52Ncu6q98Slb2cpWdjCyu8vv0v/000+JjY0lOTmZ6urqgA3I4XDgcDh8t2trawN27GDozfEpW9nKVva1JCUltXuf36V/8OBBysvLqays5Ntvv+XcuXO89tprNDc309bWRmhoKB6PB5vNBlxc9bvdbux2O21tbTQ3NxMdHe1vvIiI+MHvc/q/+tWvePXVVykqKiI7O5vbb7+dBQsWMHLkSHbv3g1AaWkpaWlpANx5552UlpYCsHv3bkaOHInFYun+DEREpNMC/j79Bx98kK1btzJ//nwaGxvJzMwEIDMzk8bGRubPn8/WrVt58MEHAx0tIiIdCMgVuSNHjmTkyJEA9O3bl5UrV161T1hYGDk5OYGIExERP+mKXBERE1Hpi4iYiEpfRMREVPoiIiai0hcRMRGVvoiIiaj0RURMRKUvImIiKn0RERNR6YuImIhKX0TERFT6IiImotIXETERlb6IiImo9EVETESlLyJiIip9ERETUemLiJiI338usba2lqKiIr7++mssFgsOh4OpU6fS2NhIfn4+Z8+e5aabbmLhwoVERUVhGAbFxcVUVlYSHh5OVlYWycnJgZyLiIh0wO+VfmhoKLNmzSI/P5/ly5fz8ccfc/LkSbZs2cKoUaMoLCxk1KhRbNmyBYDKykpOnz5NYWEhc+fOZd26dQGbhIiIdI7fpR8fH+9bqd944430798fj8eDy+Vi0qRJAEyaNAmXywVAeXk5EydOxGKxMGzYMJqamqirqwvAFEREpLMCck6/pqaG48ePM3ToUOrr64mPjwcgLi6O+vp6ADweDwkJCb7H2O12PB5PIOJFRKST/D6nf8n58+fJy8vjoYceIiIi4or7LBYLFoulS8dzOp04nU4AcnNzr/hB0VVn/H5k57U3PmUrW9nKDkZ2d3Wr9FtbW8nLy+Oee+7hrrvuAiA2Npa6ujri4+Opq6sjJiYGAJvNRm1tre+xbrcbm8121TEdDgcOh8N3+/LHXI96c3zKVraylX0tSUlJ7d7n9+kdwzB49dVX6d+/P/fee69ve1paGjt27ABgx44djBs3zrd9586dGIbBoUOHiIiI8J0GEhGRnuH3Sv/gwYPs3LmTQYMGsXjxYgB++ctfMn36dPLz8ykpKfG9ZRNgzJgxVFRUsGDBAsLCwsjKygrMDEREpNP8Lv3hw4ezcePGa963bNmyq7ZZLBbmzJnjb5yIiASArsgVETERlb6IiImo9EVETESlLyJiIip9ERETUemLiJiISl9ExERU+iIiJqLSFxExEZW+iIiJqPRFRExEpS8iYiIqfRERE1Hpi4iYiEpfRMREVPoiIiai0hcRMRGVvoiIiaj0RURMxO+/keuvqqoqiouL8Xq9TJkyhenTp/f0EERETKtHV/per5f169ezdOlS8vPz2bVrFydPnuzJIYiImFqPlv6RI0fo168fffv2xWq1MmHCBFwuV08OQUTE1CyGYRg9FbZ7926qqqqYN28eADt37uTw4cM88sgjvn2cTidOpxOA3NzcnhqaiIgpXHcv5DocDnJzc3ul8J966qkez1S2spVtjvzenvslPVr6NpsNt9vtu+12u7HZbD05BBERU+vR0h8yZAinTp2ipqaG1tZWysrKSEtL68khiIiYWuhzzz33XE+FhYSE0K9fP1avXs0//vEP7rnnHtLT03sqvlOSk5OVrWxl/49m93Z+b88deviFXBER6V3X3Qu5IiISPCp9ERETMVXpNzU18fHHHwNQXV1tyusALv8aXO9mzZrVIzkffvghCxcupLCwMKg5zz77bFCPf73qjXn31HP6fdTjn73Tm5qamti2bRs/+tGPensovUZfg6tt27aN3/3ud9jt9qDmvPDCC0E9/vWqN+bdlee0ra2N0NDQHhjV9cFUL+QWFBTgcrlISkrCarUSHh5OdHQ0J06cIDk5mfnz52OxWDh27Bivv/4658+fJyYmhqysLOLj44M2rj/+8Y+43W5aWlqYOnUqDocjaFmXfw1Gjx4NXPwQPID777+fCRMmBDTvWnObNWsWU6dOpaKigrCwMBYvXkxcXBw1NTW8/PLLnD9/nnHjxvH3v/+dDRs2BHQ8W7duZfv27QBkZmby5Zdfsn37dpKSkpg8eTL33ntvQPMuN2vWLDZs2EB1dTXvvPPONb/3AuHtt98mKiqKadOmAfDXv/6V2NhY3G73Vc91dXU1H3zwge/CofXr1zNkyBAyMjICMhb4/3nX1dVRUFBAc3MzXq+XOXPmcNtttwUs55I///nPvuc0IyODAwcOUFNTQ3h4OHPnzuUHP/gBGzdu5MyZM9TU1GC328nOzg5Idk1NDStWrCAlJYVDhw75vpbvvPMO9fX1LFiwgMLCQl544QViYmLwer389re/Zfny5cTExARkDB0yTOTMmTNGTk6OYRiGsW/fPuPXv/61UVtba7S1tRlLly41Dhw4YLS0tBjPPPOMUV9fbxiGYezatcsoKioK6rgaGhoMwzCMCxcuGDk5OcY333wTtKzLvwaffPKJ8Yc//MFoa2sz6urqjHnz5hkejyegedea289//nPD5XIZhmEYGzZsMDZt2mQYhmHk5uYapaWlhmEYxkcffWTMnDkzoGM5evSokZOTY5w7d844d+6csXDhQuPYsWNGVlaW7/kOpkvzae97L1DOnDljPPnkk4ZhGEZbW5vx+OOPt/tc79u3z1i5cqXvsevWrTO2b98esLEYxv/P+/333zf+9re/+cbV3Nwc0JzLXXpO169fb2zcuNEwDMPYu3ev8cQTTxiGYRhvv/228eSTTxoXLlwIaO6ZM2eMX/ziF8YXX3xhtLW1GU8++aRRVFRkeL1e4z//+Y/x4osvGhs3bjS2bt1qGIZhVFVVGatWrQroGDpiqtM7/23o0KG+X/8GDx5MTU0NERERnDhxgueffx64+MmgwVzlw8Xzj5c+eK62tpZTp04RHR0d1EyAzz//nB/+8IeEhIQQFxfHiBEjOHr0aEAvmLvW3KxWK3feeSdw8X3Ln332GQAHDx5k0aJFAEycOJE333wzYOOAi/MdP348N9xwAwDjx4/nwIEDAc3orGt97w0fPjwgx05MTCQqKorjx49TX1/P4MGD232ub7zxxoBkdsaQIUN45ZVXaG1tZfz48QwePDjomZ9//rnve+r222+nsbGR5uZmANLS0ggLCwt4ZmJiIoMGDQJg4MCBjBo1CovFwqBBgzh79iyzZ89m1apVTJs2je3btzN58uSAj+G7mLr0+/Tp4/t3SEgIXq8XgAEDBrB8+fIeGUN1dTV79+7lhRdeIDw8nOeee46WlpYeyQ629uYWGhrqO5UREhJCW1ub7zGBOsVxvWvvey9QpkyZQmlpKV9//TWTJ0/2/WD9b6GhoRiXneEN5vfeiBEj+P3vf09FRQVFRUXce++9TJo0KWh5HQkPDw/KcS9/bi0Wi++2xWLB6/WSkJBAbGws+/bt48iRIyxYsCAo42iPqd69c+ONN3Lu3Lnv3CcpKYlvvvmGQ4cOAdDa2sqJEyeCNqbm5mYiIyMJDw/nyy+/5PDhw0HLgiu/BrfddhuffPIJXq+Xb775hgMHDjB06NCAZXV1brfeeiu7du0C4N///nfAxnHJ8OHDcblcXLhwgfPnz+NyuYJyTvl6MH78eKqqqjh69CipqantPtcJCQmcPHmSlpYWmpqa2Lt3b9DGdPbsWeLi4nA4HEyZMoXjx48HLeuS4cOH869//Qu4uAiJjo4mIiIi6LkdyczMZPXq1aSnpxMS0rM1bKqVfnR0NLfeeiuLFi0iLCyM2NjYq/axWq0sWrSI4uJimpubaWtrY+rUqQwcODAoY0pNTeWf//wnCxcu5OabbyYlJSUoOZdc/jVITU1l0KBBLF68GICZM2cSFxcXsKyuzu3hhx/m5Zdf5r333mPcuHEBG8clycnJZGRksHTpUuDif7xbbrkl4DnXA6vVysiRI4mMjCQkJITx48dz6NChaz7Xd999N4sWLSIxMTGoX49LLxqHhoZyww038Pjjjwct65IZM2awdu1annjiCcLDw3nssceCntkZaWlpvPLKKz1+agdM9u4dEbPwer0sWbKEnJwcbr755t4ejvyXo0eP8vrrr/OHP/yhx7NNtdIXMYOTJ0+Sm5vL+PHjVfjXoS1btrBt27YeP5d/iVb6IiImYqoXckVEzE6lLyJiIip9ERETUemLiJiISl9ExET+DwJoZhfL277NAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 282
        },
        "id": "5HgBeNYlKGzZ",
        "outputId": "5bae723c-6a19-4417-ddb6-9473908c969e"
      },
      "source": [
        "# Then we analyze disaster tweets (class=1)\r\n",
        "corpus=create_corpus(1)\r\n",
        "\r\n",
        "dic=defaultdict(int)\r\n",
        "for word in corpus:\r\n",
        "    if word in stop:\r\n",
        "        dic[word]+=1\r\n",
        "\r\n",
        "top=sorted(dic.items(), key=lambda x:x[1],reverse=True)[:10] \r\n",
        "    \r\n",
        "\r\n",
        "\r\n",
        "x,y=zip(*top)\r\n",
        "plt.bar(x,y)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<BarContainer object of 10 artists>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 36
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD4CAYAAAAAczaOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAWlklEQVR4nO3da2xT5wHG8cfES1rnbgfowmXgcissDEZCo1YlWfAnYBVaN6a1MHUdRTRbEIHSMVahbmtLtIkmDZdVK1U6UaSNsS6jjK6dlSZstEjOEjrIKLCCJnqBkDgNCeEW590HhAXlGsexU97/7xM+Pj7P+8bmycnx8bHDGGMEALDCoHgPAAAQO5Q+AFiE0gcAi1D6AGARSh8ALELpA4BFnPEewM188sknMcvKyspSS0tLzPLIJptssvtDdnb2de9jTx8ALELpA4BFKH0AsAilDwAWofQBwCKUPgBYhNIHAItQ+gBgEUofACwy4D+R2xehxx/s1fonern9hJe39/IRABBf7OkDgEVu6z39eOKvDAADEXv6AGARSh8ALELpA4BFKH0AsMhN38jduHGjGhoalJ6errVr10qSOjs7VV5erpMnT2rw4MEqLS1VSkqKjDGqqqpSY2OjkpKSVFxcLK/XK0mqra3V66+/Lkn61re+pcLCwv6bFQDgmm66p19YWKhVq1Zdsay6ulo5OTmqrKxUTk6OqqurJUmNjY06fvy4KisrtWjRIm3atEnSxV8S27Zt0/PPP6/nn39e27ZtU2dnZz9MBwBwIzct/YkTJyolJeWKZYFAQAUFBZKkgoICBQIBSVJ9fb1mzJghh8OhcePG6fTp02pra9PevXs1efJkpaSkKCUlRZMnT9bevXv7YToAgBuJ6Dz99vZ2ZWZmSpIyMjLU3t4uSQoGg8rKygqv5/F4FAwGFQwG5fF4wsvdbreCweA1t+33++X3+yVJZWVlV2yvt3p77ntv3Whs8czuLafTGdXtkU022QMn+/P6/OEsh8Mhh8MRjbFIknw+n3w+X/h2vL7I+FbEc2zRzLbpC6PJJtuG7Kh/MXp6erra2tokSW1tbUpLS5N0cQ/+8om1trbK7XbL7XartbU1vDwYDMrtdkcSDQDog4hKPzc3V3V1dZKkuro65eXlhZfv2rVLxhgdOnRILpdLmZmZmjJlit5//311dnaqs7NT77//vqZMmRK9WQAAbslND+9UVFToP//5jzo6OrR48WLNmzdPc+fOVXl5uWpqasKnbErS1KlT1dDQoCVLligxMVHFxcWSpJSUFD300EP66U9/Kkn69re/fdWbwwCA/nfT0l+6dOk1l69evfqqZQ6HQwsXLrzm+kVFRSoqKurl8AAA0cQncgHAIpQ+AFiE0gcAi1D6AGARSh8ALELpA4BFKH0AsAilDwAWofQBwCKUPgBYhNIHAItQ+gBgEUofACxC6QOARSh9ALAIpQ8AFunzF6Nj4Ak9/mCv1j8RQUbCy9sjeBSAeGNPHwAsQukDgEUofQCwCKUPABah9AHAIpQ+AFiE0gcAi1D6AGARSh8ALELpA4BFKH0AsAilDwAWofQBwCJ9usrmjh07VFNTI4fDoREjRqi4uFifffaZKioq1NHRIa/Xq5KSEjmdTl24cEHr16/XkSNHlJqaqqVLl2rIkCHRmgcA4BZEvKcfDAb15ptvqqysTGvXrlVPT4/effddvfbaa5o9e7bWrVun5ORk1dTUSJJqamqUnJysdevWafbs2dqyZUvUJgEAuDV9OrzT09Oj8+fPKxQK6fz588rIyFBTU5Py8/MlSYWFhQoEApKk+vp6FRYWSpLy8/O1f/9+GWP6NnoAQK9EfHjH7Xbrm9/8pp544gklJibqa1/7mrxer1wulxISEsLrBINBSRf/MvB4PJKkhIQEuVwudXR0KC0t7Yrt+v1++f1+SVJZWZmysrIiHWJEXw7SGzca2+2cfbP83nA6nVHbFtlkk31zEZd+Z2enAoGANmzYIJfLpRdeeEF79+7t84B8Pp98Pl/4dktLS5+32V/iObZ4/1yilZ+VlRW3uZBN9u2anZ2dfd37Ij68s2/fPg0ZMkRpaWlyOp269957dfDgQXV1dSkUCkm6uHfvdrslXdzrb21tlSSFQiF1dXUpNTU10ngAQAQiLv2srCwdPnxY586dkzFG+/bt0/DhwzVp0iTt2bNHklRbW6vc3FxJ0rRp01RbWytJ2rNnjyZNmiSHw9H3GQAAblnEh3fGjh2r/Px8/eQnP1FCQoJGjRoln8+nr3/966qoqNDvf/97jR49WkVFRZKkoqIirV+/XiUlJUpJSdHSpUujNgkAwK3p03n68+bN07x5865YNnToUK1Zs+aqdRMTE7Vs2bK+xAEA+ohP5AKARSh9ALAIpQ8AFqH0AcAilD4AWITSBwCLUPoAYBFKHwAs0qcPZwGfF3r8wV6tH8kVQRNe3h7BowBI7OkDgFUofQCwCKUPABah9AHAIryRi9sGbyIDN8eePgBYhNIHAItQ+gBgEUofACxC6QOARSh9ALAIpQ8AFqH0AcAilD4AWITSBwCLUPoAYBFKHwAsQukDgEUofQCwCKUPABah9AHAIn36EpXTp0/rpZde0rFjx+RwOPTEE08oOztb5eXlOnnypAYPHqzS0lKlpKTIGKOqqio1NjYqKSlJxcXF8nq90ZoHAOAW9GlPv6qqSlOmTFFFRYV+/etfa9iwYaqurlZOTo4qKyuVk5Oj6upqSVJjY6OOHz+uyspKLVq0SJs2bYrKBAAAty7i0u/q6tKBAwdUVFQkSXI6nUpOTlYgEFBBQYEkqaCgQIFAQJJUX1+vGTNmyOFwaNy4cTp9+rTa2tqiMAUAwK2K+PBOc3Oz0tLStHHjRv3vf/+T1+vVo48+qvb2dmVmZkqSMjIy1N7eLkkKBoPKysoKP97j8SgYDIbXBQD0v4hLPxQK6ejRo3rsscc0duxYVVVVhQ/lXOJwOORwOHq1Xb/fL7/fL0kqKyu74hdFb0Xyxde9caOx3c7ZN8q3Nbu3nE5n1LZFNtm9EXHpezweeTwejR07VpKUn5+v6upqpaenq62tTZmZmWpra1NaWpokye12q6WlJfz41tZWud3uq7br8/nk8/nCty9/zEATz7HF++di69yjlZ2VlRW3eZB9+2dnZ2df976Ij+lnZGTI4/Hok08+kSTt27dPw4cPV25ururq6iRJdXV1ysvLkyTl5uZq165dMsbo0KFDcrlcHNoBgBjr0ymbjz32mCorK9Xd3a0hQ4aouLhYxhiVl5erpqYmfMqmJE2dOlUNDQ1asmSJEhMTVVxcHJUJAABuXZ9Kf9SoUSorK7tq+erVq69a5nA4tHDhwr7EAQD6iE/kAoBFKH0AsAilDwAWofQBwCKUPgBYhNIHAItQ+gBgEUofACxC6QOARSh9ALAIpQ8AFqH0AcAilD4AWITSBwCLUPoAYBFKHwAsQukDgEUofQCwCKUPABah9AHAIpQ+AFiE0gcAizjjPQDgdhB6/MFerX8igoyEl7dH8CjgSuzpA4BFKH0AsAilDwAWofQBwCKUPgBYhNIHAItQ+gBgEUofACzS5w9n9fT0aOXKlXK73Vq5cqWam5tVUVGhjo4Oeb1elZSUyOl06sKFC1q/fr2OHDmi1NRULV26VEOGDInGHAAAt6jPe/o7d+7UsGHDwrdfe+01zZ49W+vWrVNycrJqamokSTU1NUpOTta6des0e/Zsbdmypa/RAIBe6lPpt7a2qqGhQTNnzpQkGWPU1NSk/Px8SVJhYaECgYAkqb6+XoWFhZKk/Px87d+/X8aYvsQDAHqpT4d3Xn31Vc2fP19nzpyRJHV0dMjlcikhIUGS5Ha7FQwGJUnBYFAej0eSlJCQIJfLpY6ODqWlpV2xTb/fL7/fL0kqKytTVlZWxOOL5PomvXGjsd3O2TfKJzv22b3ldDqjti2yB37250Vc+v/617+Unp4ur9erpqamqA3I5/PJ5/OFb7e0tERt29EWz7HF++di69xvh+ysrKy4zYPs2MjOzr7ufRGX/sGDB1VfX6/GxkadP39eZ86c0auvvqquri6FQiElJCQoGAzK7XZLurjX39raKo/Ho1AopK6uLqWmpkYaDwCIQMSl//DDD+vhhx+WJDU1NemNN97QkiVL9MILL2jPnj26//77VVtbq9zcXEnStGnTVFtbq3HjxmnPnj2aNGmSHA5HdGYBWIzLOqM3on6e/iOPPKIdO3aopKREnZ2dKioqkiQVFRWps7NTJSUl2rFjhx555JFoRwMAbiIqX6IyadIkTZo0SZI0dOhQrVmz5qp1EhMTtWzZsmjEAQAixDdnAYgYh5a+eLgMAwBYhNIHAItQ+gBgEUofACxC6QOARSh9ALAIpQ8AFqH0AcAilD4AWITSBwCLUPoAYBFKHwAswgXXAHwhcbG3yLCnDwAWofQBwCKUPgBYhNIHAItQ+gBgEc7eAYBe+iKfOcSePgBYhNIHAItQ+gBgEUofACxC6QOARSh9ALAIpQ8AFqH0AcAilD4AWITSBwCLRHwZhpaWFm3YsEGfffaZHA6HfD6fZs2apc7OTpWXl+vkyZMaPHiwSktLlZKSImOMqqqq1NjYqKSkJBUXF8vr9UZzLgCAm4h4Tz8hIUELFixQeXm5nnvuOb311lv66KOPVF1drZycHFVWVionJ0fV1dWSpMbGRh0/flyVlZVatGiRNm3aFLVJAABuTcSln5mZGd5Tv/POOzVs2DAFg0EFAgEVFBRIkgoKChQIBCRJ9fX1mjFjhhwOh8aNG6fTp0+rra0tClMAANyqqFxls7m5WUePHtWYMWPU3t6uzMxMSVJGRoba29slScFgUFlZWeHHeDweBYPB8LqX+P1++f1+SVJZWdkVj+mtSK5s1xs3GtvtnH2jfLLJJrt/s/uqz6V/9uxZrV27Vo8++qhcLtcV9zkcDjkcjl5tz+fzyefzhW+3tLT0dYj9Jp5ji/fPxda5k032FyE7Ozv7uvf16eyd7u5urV27Vg888IDuvfdeSVJ6enr4sE1bW5vS0tIkSW63+4pJtLa2yu129yUeANBLEZe+MUYvvfSShg0bpjlz5oSX5+bmqq6uTpJUV1envLy88PJdu3bJGKNDhw7J5XJddWgHANC/Ij68c/DgQe3atUsjR47UihUrJEnf+973NHfuXJWXl6umpiZ8yqYkTZ06VQ0NDVqyZIkSExNVXFwcnRkAAG5ZxKU/YcIEbd269Zr3rV69+qplDodDCxcujDQOABAFfCIXACxC6QOARSh9ALAIpQ8AFqH0AcAilD4AWITSBwCLUPoAYBFKHwAsQukDgEUofQCwCKUPABah9AHAIpQ+AFiE0gcAi1D6AGARSh8ALELpA4BFKH0AsAilDwAWofQBwCKUPgBYhNIHAItQ+gBgEUofACxC6QOARSh9ALAIpQ8AFqH0AcAilD4AWITSBwCLOGMduHfvXlVVVamnp0czZ87U3LlzYz0EALBWTPf0e3p69Morr2jVqlUqLy/X7t279dFHH8VyCABgtZiW/n//+1/dddddGjp0qJxOp+677z4FAoFYDgEArOYwxphYhe3Zs0d79+7V4sWLJUm7du3S4cOH9cMf/jC8jt/vl9/vlySVlZXFamgAYIUB90auz+dTWVlZXAp/5cqVMc8km2yyyY6lmJa+2+1Wa2tr+HZra6vcbncshwAAVotp6d9999369NNP1dzcrO7ubr377rvKzc2N5RAAwGoJzzzzzDOxChs0aJDuuusurVu3Tn/729/0wAMPKD8/P1bxt8Tr9ZJNNtlk31bZl4vpG7kAgPgacG/kAgD6D6UPABaxqvRPnz6tt956S5LU1NQU188BPP3003HLvp6dO3eqtLRUlZWV8R5K1F3+3A90CxYsiPcQomIgvJ4G2v+z119/Pd5DsK/033777XgPQ5L07LPPxnsIV3n77bf19NNPa8mSJfEeStQNpOfeFr15PYVCoX4Zw0D7f/bnP/853kOw643ciooKBQIBZWdny+l0KikpSampqTp27Ji8Xq9KSkrkcDh05MgR/e53v9PZs2eVlpam4uJiZWZmRnUsCxYs0ObNm9XU1KQ//vGP1xxHf9qxY4feeecdSVJRUZE+/vhjvfPOO8rOztY3vvENzZkzp1/zJelXv/qVWltbdeHCBc2aNUs+n6/fsi5/7idPnizp4sX/JOmhhx7SfffdF9W8a81twYIFmjVrlhoaGpSYmKgVK1YoIyNDzc3NevHFF3X27Fnl5eXpr3/9qzZv3hy1sXz+uc7Ly9OaNWs0fvx4HTp0SG63W0899ZQSExOjlvnb3/42/HoqLCzUgQMH1NzcrKSkJC1atEhf+cpXtHXrVp04cULNzc3yeDxaunRp1PIvufT/rK2tTRUVFerq6lJPT48WLlyoe+65J+p5l/v8a+DEiRPavn27Ro4cqREjRsRv58pY5MSJE2bZsmXGGGP2799vvv/975uWlhYTCoXMqlWrzIEDB8yFCxfMz372M9Pe3m6MMWb37t1mw4YNUR/L/PnzbziO/vThhx+aZcuWmTNnzpgzZ86Y0tJSc+TIEVNcXByedyx0dHQYY4w5d+6cWbZsmTl16lS/ZV3+3L/33nvmF7/4hQmFQqatrc0sXrzYBIPBqOZda27f+c53TCAQMMYYs3nzZrNt2zZjjDFlZWWmtrbWGGPMm2++GX5tRMP1nuvvfve75ujRo8YYY9auXWvq6uqilnnJpdfTK6+8YrZu3WqMMWbfvn3mySefNMYY84c//ME89dRT5ty5c1HPvuTSz3L79u3mT3/6kzHGmFAoZLq6uvot85JrvQai+dxGKuaXVh5IxowZI4/HI0kaNWqUmpub5XK5dOzYMf3yl7+UdPHKoNHey7+VcUyYMKHf8j744ANNnz5dd9xxhyRp+vTpOnDgQL/lXc/OnTvDF9xraWnRp59+qtTU1H7P/eCDD3T//fdr0KBBysjI0MSJE/Xhhx9G9YOC15qb0+nUtGnTJF08Z/vf//63JOngwYNavny5JGnGjBnasmVL1MZxved6yJAhGjVqVHgsJ0+ejFrmtcZwaX5f/epX1dnZqa6uLklSbm5uVP/CuJ67775bv/nNb9Td3a3p06eH596frvUaGAisLv0vfelL4X8PGjRIPT09kqThw4frueeei/s4bmdNTU3at2+fnn32WSUlJemZZ57RhQsX4j2sqLje3BISEsKH7QYNGnTFcez+Ppz3eZ9/zZ0/fz6m+ZckJSXFJGfixIn6+c9/roaGBm3YsEFz5sxRQUFBv+UN5Ne3VW/k3nnnnTpz5swN18nOztapU6d06NAhSVJ3d7eOHTsWi+HFzIQJExQIBHTu3DmdPXtWgUCg349vfl5XV5eSk5OVlJSkjz/+WIcPH+7XvMuf+3vuuUfvvfeeenp6dOrUKR04cEBjxoyJWlZv5zZ+/Hjt3r1bkvTPf/4zauOQBsZzPWHCBP3jH/+QdLEMU1NT5XK5YjqGkydPKiMjQz6fTzNnztTRo0f7Ne96rwGn06nu7u5+zb4Zq/b0U1NTNX78eC1fvlyJiYlKT0+/ah2n06nly5erqqpKXV1dCoVCmjVrlkaMGBGHEfcPr9erwsJCrVq1StLFN/dGjx4d0zFMmTJFf//731VaWqovf/nLGjt2bL/mXf7cT5kyRSNHjtSKFSskSfPnz1dGRkbUsno7tx/84Ad68cUX9Ze//EV5eXlRG4d07ec6OTk5qhk3M2/ePG3cuFFPPvmkkpKS9KMf/Sim+dLFXzZvvPGGEhISdMcdd+jHP/5xv+Zd7zUwc+ZMrVixQqNHj47bG7lWnb0DALaz6vAOANiO0gcAi1D6AGARSh8ALELpA4BFKH0AsAilDwAW+T886Jb7nqN3SwAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c5i_k_9mLUc5"
      },
      "source": [
        "Observation: \r\n",
        "\r\n",
        "1. Both disaster tweets and non-disaster tweets have the common stopwords 'the', 'a', 'in'.\r\n",
        "\r\n",
        "2. Non-disaster tweets have more personal words such as 'you' and 'my' which show up less in disaster tweets."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RwMuYBfBKG8M"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u-N9UtaoMKQ3"
      },
      "source": [
        "## 1.5 Puctuations"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 337
        },
        "id": "9dRkwyAIMNXs",
        "outputId": "0854f7db-3e81-490d-ff9d-7bfb113003c0"
      },
      "source": [
        "# Common puctuations in disaster tweets\r\n",
        "plt.figure(figsize=(10,5))\r\n",
        "corpus=create_corpus(1)\r\n",
        "\r\n",
        "dic=defaultdict(int)\r\n",
        "import string\r\n",
        "special = string.punctuation\r\n",
        "for i in (corpus):\r\n",
        "    if i in special:\r\n",
        "        dic[i]+=1\r\n",
        "        \r\n",
        "x,y=zip(*dic.items())\r\n",
        "plt.bar(x,y)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<BarContainer object of 18 artists>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 37
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlYAAAEvCAYAAACHYI+LAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAfQUlEQVR4nO3de3BU9f3/8dduNgFzJZvlYkJAIyAa46CEMaBAGtZLxUFKGWYAoVottcEwiFLwMgRFJAoRBKO0pYNjtdYMjpFxOmqXFCJlGIMktYZRBLUOxhCTXZOQBCHJ/v5g2B98gS6wn71keT7+Ys+es+/3+wSyL845e9bi9Xq9AgAAQMCs4W4AAAAgWhCsAAAADCFYAQAAGEKwAgAAMIRgBQAAYAjBCgAAwBCCFQAAgCG2cDdwUn19fbhbOI3D4VBTU1PU1gtHTWaMjprMGB01mTE6ajJjeKSnp5/zufMOVj09PVq6dKnsdruWLl2qxsZGrVu3Tm1tbcrKylJRUZFsNpuOHz+ul156SV999ZWSkpK0cOFCDRgwwMggAAAAkey8TwX+/e9/V0ZGhu/x66+/rsmTJ2vDhg1KSEhQZWWlJKmyslIJCQnasGGDJk+erDfeeMN81wAAABHovIJVc3Oz9u7dq0mTJkmSvF6v6urqlJeXJ0nKz89XdXW1JGnPnj3Kz8+XJOXl5emzzz4T35oDAAAuBed1KvDVV1/VPffco87OTklSW1ub4uPjFRMTI0my2+1yu92SJLfbrbS0NElSTEyM4uPj1dbWpuTk5NNe0+VyyeVySZJKSkrkcDjMTGSIzWYLaU+hrheOmswYHTWZMTpqMmN01GTGyOM3WH3yySdKSUlRVlaW6urqjBV2Op1yOp2+x5F2YdqlcHEeM/b+euGoyYzRUZMZo6MmM4ZHQBevf/HFF9qzZ49qamp07NgxdXZ26tVXX1VHR4e6u7sVExMjt9stu90u6cTRq+bmZqWlpam7u1sdHR1KSkoyNw0AAECE8nuN1axZs7Rx40aVlZVp4cKFuu6667RgwQJlZ2dr9+7dkqTt27crNzdXkjR69Ght375dkrR7925lZ2fLYrEEbwIAAIAIcdE3CJ09e7bee+89FRUV6ciRIyooKJAkFRQU6MiRIyoqKtJ7772n2bNnG2sWAAAgkl3QDUKzs7OVnZ0tSRo4cKBWrVp1xjpxcXFatGiRme4AAAB6Eb7SBgAAwJCI+UqbaNT9mynnve7hC3ztmD9tvcAtAABAsHHECgAAwBCCFQAAgCEEKwAAAEMIVgAAAIYQrAAAAAwhWAEAABhCsAIAADCEYAUAAGAIwQoAAMAQghUAAIAhBCsAAABDCFYAAACGEKwAAAAMIVgBAAAYQrACAAAwhGAFAABgCMEKAADAEIIVAACAIQQrAAAAQwhWAAAAhhCsAAAADCFYAQAAGGLzt8KxY8dUXFysrq4udXd3Ky8vTzNmzFBZWZn27dun+Ph4SdL8+fN1xRVXyOv1avPmzaqpqVGfPn1UWFiorKysoA8CAAAQbn6DVWxsrIqLi9W3b191dXVp2bJlGjVqlCRpzpw5ysvLO239mpoaNTQ0aP369fryyy+1adMmPfvss8HpHgAAIIL4PRVosVjUt29fSVJ3d7e6u7tlsVjOuf6ePXs0YcIEWSwWjRgxQu3t7fJ4POY6BgAAiFB+j1hJUk9Pj5YsWaKGhgbdfvvtGj58uD788EO9+eab2rJli6677jrNnj1bsbGxcrvdcjgcvm3T0tLkdruVmpp62mu6XC65XC5JUklJyWnbRAKbzRZwT4cN9XI2JvaXiRkjuV44ajJjdNRkxuioyYzRUTMcMwbivIKV1WrV6tWr1d7erjVr1ujbb7/VrFmz1K9fP3V1dekPf/iD3n33XU2fPv28CzudTjmdTt/jpqamC+8+iBwOR8T1dCoTvYV6xnDsU2bs/fXCUZMZo6MmM0ZHzUh8P05PTz/ncxf0qcCEhARlZ2ertrZWqampslgsio2N1c9+9jMdOHBAkmS320/bAc3NzbLb7RfZOgAAQO/hN1i1traqvb1d0olPCH766afKyMjwXTfl9XpVXV2tzMxMSVJubq6qqqrk9Xq1f/9+xcfHn3EaEAAAIBr5PRXo8XhUVlamnp4eeb1ejR07VqNHj9ZTTz2l1tZWSdLQoUM1b948SdINN9ygvXv3asGCBYqLi1NhYWFwJwAAAIgQfoPV0KFD9fzzz5+xvLi4+KzrWywWPfDAA4F3BgAA0Mtw53UAAABDCFYAAACGEKwAAAAMIVgBAAAYQrACAAAwhGAFAABgCMEKAADAEIIVAACAIQQrAAAAQwhWAAAAhhCsAAAADCFYAQAAGEKwAgAAMIRgBQAAYAjBCgAAwBCCFQAAgCEEKwAAAEMIVgAAAIYQrAAAAAwhWAEAABhCsAIAADCEYAUAAGAIwQoAAMAQghUAAIAhNn8rHDt2TMXFxerq6lJ3d7fy8vI0Y8YMNTY2at26dWpra1NWVpaKiopks9l0/PhxvfTSS/rqq6+UlJSkhQsXasCAAaGYBQAAIKz8HrGKjY1VcXGxVq9ereeff161tbXav3+/Xn/9dU2ePFkbNmxQQkKCKisrJUmVlZVKSEjQhg0bNHnyZL3xxhtBHwIAACAS+A1WFotFffv2lSR1d3eru7tbFotFdXV1ysvLkyTl5+erurpakrRnzx7l5+dLkvLy8vTZZ5/J6/UGqX0AAIDI4fdUoCT19PRoyZIlamho0O23366BAwcqPj5eMTExkiS73S632y1JcrvdSktLkyTFxMQoPj5ebW1tSk5ODtIIAAAAkeG8gpXVatXq1avV3t6uNWvWqL6+PuDCLpdLLpdLklRSUiKHwxHwa5pks9kC7umwoV7OxsT+MjFjJNcLR01mjI6azBgdNZkxOmqGY8ZAnFewOikhIUHZ2dnav3+/Ojo61N3drZiYGLndbtntdkknjl41NzcrLS1N3d3d6ujoUFJS0hmv5XQ65XQ6fY+bmpoCHMUsh8MRcT2dykRvoZ4xHPuUGXt/vXDUZMboqMmM0VEzEt+P09PTz/mc32usWltb1d7eLunEJwQ//fRTZWRkKDs7W7t375Ykbd++Xbm5uZKk0aNHa/v27ZKk3bt3Kzs7WxaLJdAZAAAAIp7fI1Yej0dlZWXq6emR1+vV2LFjNXr0aA0ePFjr1q3T3/72N1155ZUqKCiQJBUUFOill15SUVGREhMTtXDhwqAPAQAAEAn8BquhQ4fq+eefP2P5wIEDtWrVqjOWx8XFadGiRWa6AwAA6EW48zoAAIAhBCsAAABDCFYAAACGEKwAAAAMIVgBAAAYQrACAAAwhGAFAABgCMEKAADAEIIVAACAIQQrAAAAQwhWAAAAhhCsAAAADCFYAQAAGEKwAgAAMIRgBQAAYAjBCgAAwBCCFQAAgCEEKwAAAEMIVgAAAIYQrAAAAAwhWAEAABhCsAIAADCEYAUAAGAIwQoAAMAQghUAAIAhNn8rNDU1qaysTD/++KMsFoucTqfuvPNOlZeXa9u2bUpOTpYkzZw5UzfeeKMk6Z133lFlZaWsVqvuu+8+jRo1KrhTAAAARAC/wSomJkZz5sxRVlaWOjs7tXTpUl1//fWSpMmTJ2vKlCmnrX/o0CHt2rVLL7zwgjwej1asWKEXX3xRVisHxwAAQHTzm3ZSU1OVlZUlSbrsssuUkZEht9t9zvWrq6s1btw4xcbGasCAARo0aJAOHDhgrmMAAIAI5feI1akaGxv19ddfa9iwYfr888/1wQcfqKqqSllZWZo7d64SExPldrs1fPhw3zZ2u/2sQczlcsnlckmSSkpK5HA4AhzFLJvNFnBPhw31cjYm9peJGSO5XjhqMmN01GTG6KjJjNFRMxwzBuK8g9XRo0dVWlqqe++9V/Hx8brttts0ffp0SdJbb72l1157TYWFhedd2Ol0yul0+h43NTVdQNvB53A4Iq6nU5noLdQzhmOfMmPvrxeOmswYHTWZMTpqRuL7cXp6+jmfO68Ln7q6ulRaWqrx48frpptukiT169dPVqtVVqtVkyZN0sGDByWdOELV3Nzs29btdstutwfSPwAAQK/gN1h5vV5t3LhRGRkZuuuuu3zLPR6P788ff/yxMjMzJUm5ubnatWuXjh8/rsbGRn3//fcaNmxYEFoHAACILH5PBX7xxReqqqrSkCFDtHjxYkknbq3wr3/9S998840sFov69++vefPmSZIyMzM1duxYLVq0SFarVffffz+fCAQAAJcEv8Fq5MiRKi8vP2P5yXtWnc20adM0bdq0wDoDAADoZTiUBAAAYAjBCgAAwBCCFQAAgCEEKwAAAEMIVgAAAIYQrAAAAAwhWAEAABhCsAIAADCEYAUAAGAIwQoAAMAQghUAAIAhBCsAAABDCFYAAACGEKwAAAAMIVgBAAAYQrACAAAwhGAFAABgCMEKAADAEIIVAACAIQQrAAAAQwhWAAAAhhCsAAAADCFYAQAAGEKwAgAAMMTmb4WmpiaVlZXpxx9/lMVikdPp1J133qkjR45o7dq1+uGHH9S/f389/PDDSkxMlNfr1ebNm1VTU6M+ffqosLBQWVlZoZgFAAAgrPwesYqJidGcOXO0du1arVy5Uh988IEOHTqkiooK5eTkaP369crJyVFFRYUkqaamRg0NDVq/fr3mzZunTZs2BX0IAACASOA3WKWmpvqOOF122WXKyMiQ2+1WdXW1Jk6cKEmaOHGiqqurJUl79uzRhAkTZLFYNGLECLW3t8vj8QRxBAAAgMjg91TgqRobG/X1119r2LBhamlpUWpqqiSpX79+amlpkSS53W45HA7fNmlpaXK73b51T3K5XHK5XJKkkpKS07aJBDabLeCeDhvq5WxM7C8TM0ZyvXDUZMboqMmM0VGTGaOjZjhmDMR5B6ujR4+qtLRU9957r+Lj4097zmKxyGKxXFBhp9Mpp9Ppe9zU1HRB2webw+GIuJ5OZaK3UM8Yjn3KjL2/XjhqMmN01GTG6KgZie/H6enp53zuvD4V2NXVpdLSUo0fP1433XSTJCklJcV3is/j8Sg5OVmSZLfbT9sBzc3NstvtF908AABAb+E3WHm9Xm3cuFEZGRm66667fMtzc3O1Y8cOSdKOHTs0ZswY3/Kqqip5vV7t379f8fHxZ5wGBAAAiEZ+TwV+8cUXqqqq0pAhQ7R48WJJ0syZMzV16lStXbtWlZWVvtstSNINN9ygvXv3asGCBYqLi1NhYWFwJwAAAIgQfoPVyJEjVV5eftbnli1bdsYyi8WiBx54IPDOAAAAehnuvA4AAGAIwQoAAMAQghUAAIAhBCsAAABDCFYAAACGEKwAAAAMIVgBAAAYQrACAAAwhGAFAABgCMEKAADAEIIVAACAIQQrAAAAQwhWAAAAhhCsAAAADCFYAQAAGEKwAgAAMIRgBQAAYAjBCgAAwBCCFQAAgCEEKwAAAEMIVgAAAIYQrAAAAAwhWAEAABhCsAIAADDE5m+Fl19+WXv37lVKSopKS0slSeXl5dq2bZuSk5MlSTNnztSNN94oSXrnnXdUWVkpq9Wq++67T6NGjQpi+wAAAJHDb7DKz8/XHXfcobKystOWT548WVOmTDlt2aFDh7Rr1y698MIL8ng8WrFihV588UVZrRwYAwAA0c9v4rn22muVmJh4Xi9WXV2tcePGKTY2VgMGDNCgQYN04MCBgJsEAADoDfwesTqXDz74QFVVVcrKytLcuXOVmJgot9ut4cOH+9ax2+1yu91n3d7lcsnlckmSSkpK5HA4LraVoLDZbAH3dNhQL2djYn+ZmDGS64WjJjNGR01mjI6azBgdNcMxYyAuKljddtttmj59uiTprbfe0muvvabCwsILeg2n0ymn0+l73NTUdDGtBI3D4Yi4nk5lordQzxiOfcqMvb9eOGoyY3TUZMboqBmJ78fp6ennfO6iLn7q16+frFarrFarJk2apIMHD0o6cYSqubnZt57b7Zbdbr+YEgAAAL3ORQUrj8fj+/PHH3+szMxMSVJubq527dql48ePq7GxUd9//72GDRtmplMAAIAI5/dU4Lp167Rv3z61tbXpwQcf1IwZM1RXV6dvvvlGFotF/fv317x58yRJmZmZGjt2rBYtWiSr1ar777+fTwQCAIBLht9gtXDhwjOWFRQUnHP9adOmadq0aYF1BQAA0AtxOAkAAMAQghUAAIAhBCsAAABDCFYAAACGEKwAAAAMIVgBAAAYQrACAAAwhGAFAABgCMEKAADAEIIVAACAIQQrAAAAQwhWAAAAhhCsAAAADCFYAQAAGEKwAgAAMIRgBQAAYAjBCgAAwBCCFQAAgCEEKwAAAEMIVgAAAIYQrAAAAAwhWAEAABhCsAIAADCEYAUAAGCIzd8KL7/8svbu3auUlBSVlpZKko4cOaK1a9fqhx9+UP/+/fXwww8rMTFRXq9XmzdvVk1Njfr06aPCwkJlZWUFfQgAAIBI4PeIVX5+vh5//PHTllVUVCgnJ0fr169XTk6OKioqJEk1NTVqaGjQ+vXrNW/ePG3atCk4XQMAAEQgv8Hq2muvVWJi4mnLqqurNXHiREnSxIkTVV1dLUnas2ePJkyYIIvFohEjRqi9vV0ejycIbQMAAESei7rGqqWlRampqZKkfv36qaWlRZLkdrvlcDh866WlpcntdhtoEwAAIPL5vcbKH4vFIovFcsHbuVwuuVwuSVJJSclpgSwS2Gy2gHs6bKiXszGxv0zMGMn1wlGTGaOjJjNGR01mjI6a4ZgxEBcVrFJSUuTxeJSamiqPx6Pk5GRJkt1uV1NTk2+95uZm2e32s76G0+mU0+n0PT51u0jgcDgirqdTmegt1DOGY58yY++vF46azBgdNZkxOmpG4vtxenr6OZ+7qFOBubm52rFjhyRpx44dGjNmjG95VVWVvF6v9u/fr/j4eN8pQwAAgGjn94jVunXrtG/fPrW1tenBBx/UjBkzNHXqVK1du1aVlZW+2y1I0g033KC9e/dqwYIFiouLU2FhYdAHAAAAiBR+g9XChQvPunzZsmVnLLNYLHrggQcC7woAAKAX4s7rAAAAhhCsAAAADCFYAQAAGEKwAgAAMIRgBQAAYAjBCgAAwBCCFQAAgCEEKwAAAEMIVgAAAIYQrAAAAAwhWAEAABhCsAIAADCEYAUAAGAIwQoAAMAQghUAAIAhBCsAAABDCFYAAACGEKwAAAAMIVgBAAAYQrACAAAwhGAFAABgCMEKAADAEIIVAACAIQQrAAAAQwhWAAAAhtgC2Xj+/Pnq27evrFarYmJiVFJSoiNHjmjt2rX64Ycf1L9/fz388MNKTEw01S8AAEDECihYSVJxcbGSk5N9jysqKpSTk6OpU6eqoqJCFRUVuueeewItAwAAEPECDlb/V3V1tZYvXy5JmjhxopYvX06wQq/W/Zsp573u4Qt87Zg/bb3ALQAAkSzgYLVy5UpJ0q233iqn06mWlhalpqZKkvr166eWlpazbudyueRyuSRJJSUlcjgcgbZilM1mC7inC32TvRAm9peJGSO5nqma/BzDWy8cNZkxOmoyY3TUDMeMgQgoWK1YsUJ2u10tLS165plnlJ6eftrzFotFFovlrNs6nU45nU7f46ampkBaMc7hcERcT6cy0VuoZwzHPuXnaN6l8HNkxuioyYzRUTMSf4//37xzqoA+FWi32yVJKSkpGjNmjA4cOKCUlBR5PB5JksfjOe36KwAAgGh20cHq6NGj6uzs9P35008/1ZAhQ5Sbm6sdO3ZIknbs2KExY8aY6RQAACDCXfSpwJaWFq1Zs0aS1N3drVtuuUWjRo3SVVddpbVr16qystJ3uwUAAIBLwUUHq4EDB2r16tVnLE9KStKyZcsCagoAAKA34s7rAAAAhhCsAAAADCFYAQAAGEKwAgAAMIRgBQAAYAjBCgAAwBCCFQAAgCEBfwkzIkv3b6ac97oX8uXCMX/aeuHNAABwieGIFQAAgCEEKwAAAEMIVgAAAIYQrAAAAAwhWAEAABhCsAIAADCEYAUAAGAI97ECAPQawbpXn8T9+mAGR6wAAAAMIVgBAAAYwqlAAAAucXwdmjkcsQIAADCEI1YICBeSAgDw/xGsAACIIPyHtXfjVCAAAIAhHLFCr3MpXGTJjKfrrTMCuPQELVjV1tZq8+bN6unp0aRJkzR16tRglQIAiFNIQCQISrDq6enRn//8Zz355JNKS0vTY489ptzcXA0ePDgY5QAAYXIpHHm8FGYMtWj+T0BQgtWBAwc0aNAgDRw4UJI0btw4VVdXE6yACBXNv+ROuhRmBBB+Fq/X6zX9ort371Ztba0efPBBSVJVVZW+/PJL3X///b51XC6XXC6XJKmkpMR0CwAAACEXtk8FOp1OlZSURGyoWrp0aVTXC0dNZoyOmswYHTWZMTpqMmPkCUqwstvtam5u9j1ubm6W3W4PRikAAICIEZRgddVVV+n7779XY2Ojurq6tGvXLuXm5gajFAAAQMSIWb58+XLTL2q1WjVo0CBt2LBB77//vsaPH6+8vDzTZYIuKysrquuFoyYzRkdNZoyOmswYHTWZMbIE5eJ1AACASxFfaQMAAGAIwQoAAMAQvisQIfHdd9/plVdeUWdnpxITE/XII48oOTk53G3hIvz1r3/V9ddfr46ODn333Xf6xS9+EdR65eXl6tu3r6ZMOf8bfAIn/fGPf9SECRM0cuTIcLdi1I8//qh33nlHdXV1iomJ0ZVXXqnp06fL4XCEuzVjvF6vLBaLysvLNWPGDN/jSMcRK4RMUVGRSktLdfXVV+sf//hHuNsJiuXLl6uxsTFk9erq6lRWVhayepL05ZdfasSIEdq3b5+uueaakNYOhWPHjmnx4sWaOXOmWltbQ1Kzq6tLxcXF6u7uDkk9SZo/f37IaoXTyb+v0aShoUHPPvusrr76apWUlOi5557TzTffrDVr1qihoSHc7Rnz0UcfaevWrTp+/LjeffddffTRR+Fu6bwQrM7hyJEj4W4hqmRkZPi+4uj48eOKjY0Nc0e4UH/5y1/06KOP6uDBg3riiSdUWVmpTZs2acuWLeFuzai4uDitXr06pPfes9lsuu6667Rr166Q1QyHVatWye12h6zeoUOHdPnll8tqja63uk2bNmn+/PkaN26cbLYTJ55ycnJUVFSk1157LczdmTNhwgTZ7XZt3bpVDodDEyZMCHdL54VTgeewdetW/ec//1FBQYFuvvlmxcfHh7slo1atWqXf/va3Ib9xa21trWpra/XMM8+EtC4CN2fOHI0dO1ZVVVWaO3eunnrqKa1YsSLcbUWNMWPG6M0339T48eNDUi8cp+Ife+yxkNarra3VqFGjgl5n2bJl6uzsPGP5nDlzdP311xutVV9fr+TkZA0dOlSffPKJysvLNWDAAHm9Xj366KOyWq1qbW01/vN97733tGvXLtlsNuXn5+uaa65RdXW1Ro4cGbQjgjt37pTb7daUKVPU1NSknTt36pZbbglKLZMu6WD1v/4xzJo1S/X19aqsrNSSJUt0zTXXqKCgIGrO04f6F5wk9fT0aOPGjSouLlZCQkLI60eTxx9/XMePH9fRo0d15MgRLV68WJI0e/bsoL6RfP311xo6dKjq6+uVkZERtDqXoiFDhujAgQMhq7dq1aqQ1QqXf//73/rd734X9DpPP/100Guc9N///lfDhw9XT0+PtmzZomXLlqmjo0OPPPKIJGnQoEFqbGw0HqxaWlq0YsUKHT58WG+//ba2bt2qsWPHatiwYUbrnOrmm2/2XWN19913q7fcHeqSDlb+/jGkp6frnnvu0axZs7Rz506tWrVKEydO1K9//Wujfbz//vvatm2bpBOBJ1q//sfj8Sg+Pl6XX355uFvp9Z599llJJ66x2r59e9Cvl/nmm29UVlYmt9utpKQk/fTTT5KkxYsXa+XKlYqLiwta7RkzZgTttSOJ1WqVzWZTZ2enLrvssnC30+v99NNPam9vD8nv01AesZLkOyo1cOBAJSQkKCEhQYMHD5Yktba2KiUlxXjN2bNnSzrxvlhUVGT89c/m5IXqJ38H9IYL16VLPFj5+8fg9XpVV1enyspKHTx4UD//+c81adIk433ccccduuOOO4y/bqRJSEjQ3Llzw90GLsIVV1yh1atX68knn9TTTz+tV155RXfffbfvlznM6OrqiurrD59++mk99NBDIQk7dXV1ys7ODnodKbRHrDIzM1VRUaFbb71Vhw8fVkdHh+8Tut9++61aWlrUv3//kPWDM13Swep//WP46KOPtGXLFmVmZqqgoEAPPfRQVF0AGY5rrDo6OrRt27aQXPMghe86smjV2tqqhIQEWa1W1dfXhyxUffjhh+rTp48mTpwYknrh0tbWpqSkJN/FyNGmp6dHDQ0NSkxMDEm9mpqaXvlVav4MHjxYzc3Nqq+v17Rp0/TUU09pwIABGj16tLZu3RqSU5/43/hKm3P4/PPPlZ6ezr2WcEGWL1+uwsJCDRgwINytIEDz58/XqlWrQvY7YPfu3dq/f3/UHtX99ttv9c9//lO/+tWvQlJvyZIlWrlyZVQG1UOHDmnDhg2aPXu2cnJyJJ24/tHtdis3NzfM3SH6/sYZEi0XqQPoHXbu3KlZs2aFu42gGTJkSMhClSQ999xzIasVaoMHD9bvf/97vf3223r99dfV09OjYcOG6Ze//GW4W4MIVgBwmmPHjumJJ55QV1dXyE7/d3V1acyYMUpPTw9JPfR+aWlpmjdvXrjbwFlwKhAwaPv27RozZgy3kwCASxTBCgAAwJDo+ZgbAABAmBGsAAAADCFYAQAAGEKwAgAAMOT/AVqpz/uPaCK2AAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 720x360 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 337
        },
        "id": "Um_j-Y3kMa-f",
        "outputId": "3cb2342a-76b7-4667-f17e-1dcc8859099a"
      },
      "source": [
        "# Common puctuations in non-disaster tweets\r\n",
        "plt.figure(figsize=(10,5))\r\n",
        "corpus=create_corpus(0)\r\n",
        "\r\n",
        "dic=defaultdict(int)\r\n",
        "import string\r\n",
        "special = string.punctuation\r\n",
        "for i in (corpus):\r\n",
        "    if i in special:\r\n",
        "        dic[i]+=1\r\n",
        "        \r\n",
        "x,y=zip(*dic.items())\r\n",
        "plt.bar(x,y,color='green')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<BarContainer object of 20 artists>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 38
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlYAAAEvCAYAAACHYI+LAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAeMUlEQVR4nO3de3CU5d3G8WuXTYAcybIcTDg1AqIxDkoyBpRDdT1UrDKUoYMURkeLNgwUrYyKnQRFIAViOBilLR2mtpaa0THS6ghdMiRihjFIqFOYikGR4RBDsktCAkgO+/7BsAMF3gB7P5vN7vfzF3t6rvuHQi7u59ldm9/v9wsAAABBs3f1AgAAACIFxQoAAMAQihUAAIAhFCsAAABDKFYAAACGUKwAAAAMoVgBAAAY4ujqBZx39OjRrl5CgMvlUn19fbfPiLScSJolVDnMEt05zBLdOZE0SyhzrkZqauoVH2PHCgAAwBCKFQAAgCEUKwAAAEMoVgAAAIZQrAAAAAyhWAEAABhCsQIAADCEYgUAAGAIxQoAAMAQihUAAIAhYfOVNqGQ9sc0o8c78ssjRo8HAAC6N3asAAAADKFYAQAAGEKxAgAAMIRiBQAAYAjFCgAAwBCKFQAAgCEUKwAAAEMoVgAAAIZQrAAAAAyhWAEAABhCsQIAADCEYgUAAGAIxQoAAMAQR2dPOHv2rPLz89XW1qb29nbl5ORo+vTpKi4u1r59+xQXFydJmjt3roYNGya/36+NGzequrpaPXv2VG5urtLT0y0fBAAAoKt1WqxiYmKUn5+vXr16qa2tTXl5eRo9erQkadasWcrJybno+dXV1aqtrdXatWv19ddfa8OGDVq2bJk1qwcAAAgjnZ4KtNls6tWrlySpvb1d7e3tstlsV3z+rl27NGHCBNlsNo0cOVItLS3y+XzmVgwAABCmOt2xkqSOjg698MILqq2t1QMPPKARI0Zo69at2rRpk9577z3deuutmjlzpmJiYuT1euVyuQKv7du3r7xer1JSUi46psfjkcfjkSQVFBRc9JruIpg1OxyOkMwcSTmRNEuocpglunOYJbpzImmWUOYE66qKld1u18qVK9XS0qJVq1bp0KFDeuyxx9SnTx+1tbXp97//vT788ENNmzbtqoPdbrfcbnfgdn19/bWvvosFs2aXyxWSmSMpJ5JmCVUOs0R3DrNEd04kzRLKnKuRmpp6xceu6V2B8fHxysjI0J49e5SSkiKbzaaYmBj9+Mc/Vk1NjSTJ6XReNHhDQ4OcTud1Lh0AAKD76LRYNTU1qaWlRdK5dwh++eWXSktLC1w35ff7VVVVpcGDB0uSsrKyVFFRIb/fr/379ysuLu6S04AAAACRqNNTgT6fT8XFxero6JDf79fYsWM1ZswYvfLKK2pqapIkDR06VHPmzJEk3X777dq9e7fmz5+v2NhY5ebmWjsBAABAmOi0WA0dOlQrVqy45P78/PzLPt9ms+mpp54KfmUAAADdDJ+8DgAAYAjFCgAAwBCKFQAAgCEUKwAAAEMoVgAAAIZQrAAAAAyhWAEAABhCsQIAADCEYgUAAGAIxQoAAMAQihUAAIAhFCsAAABDKFYAAACGUKwAAAAMoVgBAAAYQrECAAAwhGIFAABgCMUKAADAEIoVAACAIRQrAAAAQyhWAAAAhlCsAAAADKFYAQAAGEKxAgAAMIRiBQAAYIijsyecPXtW+fn5amtrU3t7u3JycjR9+nTV1dVp9erVOnnypNLT0zVv3jw5HA61trbqjTfe0DfffKPExEQtWLBA/fv3D8UsAAAAXarTHauYmBjl5+dr5cqVWrFihfbs2aP9+/frr3/9qyZPnqx169YpPj5eZWVlkqSysjLFx8dr3bp1mjx5st555x3LhwAAAAgHnRYrm82mXr16SZLa29vV3t4um82mvXv3KicnR5I0adIkVVVVSZJ27dqlSZMmSZJycnL0n//8R36/36LlAwAAhI9OTwVKUkdHh1544QXV1tbqgQce0IABAxQXF6cePXpIkpxOp7xeryTJ6/Wqb9++kqQePXooLi5OJ0+eVFJSkkUjAAAAhIerKlZ2u10rV65US0uLVq1apaNHjwYd7PF45PF4JEkFBQVyuVxBHzPUglmzw+EIycyRlBNJs4Qqh1miO4dZojsnkmYJZU6wrqpYnRcfH6+MjAzt379fp06dUnt7u3r06CGv1yun0ynp3O5VQ0OD+vbtq/b2dp06dUqJiYmXHMvtdsvtdgdu19fXBzlK6AWzZpfLFZKZIyknkmYJVQ6zRHcOs0R3TiTNEsqcq5GamnrFxzq9xqqpqUktLS2Szr1D8Msvv1RaWpoyMjK0c+dOSdL27duVlZUlSRozZoy2b98uSdq5c6cyMjJks9mCnQEAACDsdbpj5fP5VFxcrI6ODvn9fo0dO1ZjxozRoEGDtHr1av3973/Xj370I91zzz2SpHvuuUdvvPGG5s2bp4SEBC1YsMDyIQAAAMJBp8Vq6NChWrFixSX3DxgwQMuXL7/k/tjYWD333HNmVgcAANCN8MnrAAAAhlCsAAAADKFYAQAAGEKxAgAAMIRiBQAAYAjFCgAAwBCKFQAAgCEUKwAAAEMoVgAAAIZQrAAAAAyhWAEAABhCsQIAADCEYgUAAGAIxQoAAMAQihUAAIAhFCsAAABDKFYAAACGUKwAAAAMoVgBAAAYQrECAAAwhGIFAABgCMUKAADAEIoVAACAIRQrAAAAQyhWAAAAhlCsAAAADHF09oT6+noVFxfrxIkTstlscrvdeuihh1RSUqJt27YpKSlJkjRjxgzdcccdkqQPPvhAZWVlstvteuKJJzR69GhrpwAAAAgDnRarHj16aNasWUpPT9fp06f14osv6rbbbpMkTZ48WY888shFzz98+LAqKyv1+uuvy+fzacmSJVqzZo3sdjbHAABAZOu07aSkpCg9PV2S1Lt3b6Wlpcnr9V7x+VVVVRo3bpxiYmLUv39/DRw4UDU1NeZWDAAAEKY63bG6UF1dnb799lsNHz5c//3vf7VlyxZVVFQoPT1ds2fPVkJCgrxer0aMGBF4jdPpvGwR83g88ng8kqSCggK5XK4gRwm9YNbscDhCMnMk5UTSLKHKYZbozmGW6M6JpFlCmROsqy5WZ86cUWFhoR5//HHFxcXp/vvv17Rp0yRJ7777rt5++23l5uZedbDb7Zbb7Q7crq+vv4Zlh4dg1uxyuUIycyTlRNIsocphlujOYZbozomkWUKZczVSU1Ov+NhVXfjU1tamwsJCjR8/XnfeeackqU+fPrLb7bLb7br33nt14MABSed2qBoaGgKv9Xq9cjqdwawfAACgW+i0WPn9fq1fv15paWl6+OGHA/f7fL7Arz///HMNHjxYkpSVlaXKykq1traqrq5Ox44d0/Dhwy1YOgAAQHjp9FTgV199pYqKCg0ZMkQLFy6UdO6jFT777DMdPHhQNptN/fr105w5cyRJgwcP1tixY/Xcc8/JbrfrySef5B2BAAAgKnRarEaNGqWSkpJL7j//mVWXM3XqVE2dOjW4lQEAAHQzbCUBAAAYQrECAAAwhGIFAABgCMUKAADAEIoVAACAIRQrAAAAQyhWAAAAhlCsAAAADKFYAQAAGEKxAgAAMIRiBQAAYAjFCgAAwBCKFQAAgCEUKwAAAEMoVgAAAIZQrAAAAAyhWAEAABhCsQIAADCEYgUAAGAIxQoAAMAQihUAAIAhFCsAAABDKFYAAACGUKwAAAAMoVgBAAAYQrECAAAwxNHZE+rr61VcXKwTJ07IZrPJ7XbroYceUnNzs4qKinT8+HH169dPzz77rBISEuT3+7Vx40ZVV1erZ8+eys3NVXp6eihmAQAA6FKd7lj16NFDs2bNUlFRkZYuXaotW7bo8OHDKi0tVWZmptauXavMzEyVlpZKkqqrq1VbW6u1a9dqzpw52rBhg+VDAAAAhINOi1VKSkpgx6l3795KS0uT1+tVVVWVJk6cKEmaOHGiqqqqJEm7du3ShAkTZLPZNHLkSLW0tMjn81k4AgAAQHjo9FTgherq6vTtt99q+PDhamxsVEpKiiSpT58+amxslCR5vV65XK7Aa/r27Suv1xt47nkej0cej0eSVFBQcNFruotg1uxwOEIycyTlRNIsocphlujOYZbozomkWUKZE6yrLlZnzpxRYWGhHn/8ccXFxV30mM1mk81mu6Zgt9stt9sduF1fX39Nrw8HwazZ5XKFZOZIyomkWUKVwyzRncMs0Z0TSbOEMudqpKamXvGxq3pXYFtbmwoLCzV+/HjdeeedkqTk5OTAKT6fz6ekpCRJktPpvGjwhoYGOZ3O6148AABAd9FpsfL7/Vq/fr3S0tL08MMPB+7PyspSeXm5JKm8vFzZ2dmB+ysqKuT3+7V//37FxcVdchoQAAAgEnV6KvCrr75SRUWFhgwZooULF0qSZsyYoSlTpqioqEhlZWWBj1uQpNtvv127d+/W/PnzFRsbq9zcXGsnAAAACBOdFqtRo0appKTkso/l5eVdcp/NZtNTTz0V/MoAAAC6GT55HQAAwBCKFQAAgCEUKwAAAEMoVgAAAIZQrAAAAAyhWAEAABhCsQIAADCEYgUAAGAIxQoAAMAQihUAAIAhFCsAAABDKFYAAACGUKwAAAAMoVgBAAAYQrECAAAwhGIFAABgCMUKAADAEIoVAACAIRQrAAAAQyhWAAAAhlCsAAAADKFYAQAAGEKxAgAAMIRiBQAAYAjFCgAAwBBHZ0948803tXv3biUnJ6uwsFCSVFJSom3btikpKUmSNGPGDN1xxx2SpA8++EBlZWWy2+164oknNHr0aAuXDwAAED46LVaTJk3Sgw8+qOLi4ovunzx5sh555JGL7jt8+LAqKyv1+uuvy+fzacmSJVqzZo3sdjbGAABA5Ou08dxyyy1KSEi4qoNVVVVp3LhxiomJUf/+/TVw4EDV1NQEvUgAAIDuoNMdqyvZsmWLKioqlJ6ertmzZyshIUFer1cjRowIPMfpdMrr9V729R6PRx6PR5JUUFAgl8t1vUvpMsGs2eFwhGTmSMqJpFlClcMs0Z3DLNGdE0mzhDInWNdVrO6//35NmzZNkvTuu+/q7bffVm5u7jUdw+12y+12B27X19dfz1K6VDBrdrlcIZk5knIiaZZQ5TBLdOcwS3TnRNIsocy5GqmpqVd87LoufurTp4/sdrvsdrvuvfdeHThwQNK5HaqGhobA87xer5xO5/VEAAAAdDvXVax8Pl/g159//rkGDx4sScrKylJlZaVaW1tVV1enY8eOafjw4WZWCgAAEOY6PRW4evVq7du3TydPntQzzzyj6dOna+/evTp48KBsNpv69eunOXPmSJIGDx6ssWPH6rnnnpPdbteTTz7JOwIBAEDU6LRYLViw4JL77rnnnis+f+rUqZo6dWpwqwIAAOiG2E4CAAAwhGIFAABgCMUKAADAEIoVAACAIRQrAAAAQyhWAAAAhlCsAAAADKFYAQAAGEKxAgAAMIRiBQAAYAjFCgAAwBCKFQAAgCEUKwAAAEMoVgAAAIZQrAAAAAyhWAEAABhCsQIAADCEYgUAAGAIxQoAAMAQihUAAIAhFCsAAABDKFYAAACGUKwAAAAMoVgBAAAYQrECAAAwhGIFAABgiKOzJ7z55pvavXu3kpOTVVhYKElqbm5WUVGRjh8/rn79+unZZ59VQkKC/H6/Nm7cqOrqavXs2VO5ublKT0+3fAgAAIBw0OmO1aRJk7Ro0aKL7istLVVmZqbWrl2rzMxMlZaWSpKqq6tVW1urtWvXas6cOdqwYYM1qwYAAAhDnRarW265RQkJCRfdV1VVpYkTJ0qSJk6cqKqqKknSrl27NGHCBNlsNo0cOVItLS3y+XwWLBsAACD8dHoq8HIaGxuVkpIiSerTp48aGxslSV6vVy6XK/C8vn37yuv1Bp57IY/HI4/HI0kqKCi46HXdRTBrdjgcIZk5knIiaZZQ5TBLdOcwS3TnRNIsocwJ1nUVqwvZbDbZbLZrfp3b7Zbb7Q7crq+vD3YpIRfMml0uV0hmjqScSJolVDnMEt05zBLdOZE0SyhzrkZqauoVH7uudwUmJycHTvH5fD4lJSVJkpxO50VDNzQ0yOl0Xk8EAABAt3NdxSorK0vl5eWSpPLycmVnZwfur6iokN/v1/79+xUXF3fZ04AAAACRqNNTgatXr9a+fft08uRJPfPMM5o+fbqmTJmioqIilZWVBT5uQZJuv/127d69W/Pnz1dsbKxyc3MtHwAAACBcdFqsFixYcNn78/LyLrnPZrPpqaeeCn5VAAAA3VDQF6/jYml/TDN+zCO/PGL8mAAAwDy+0gYAAMAQihUAAIAhFCsAAABDKFYAAACGUKwAAAAMoVgBAAAYQrECAAAwhGIFAABgCMUKAADAEIoVAACAIRQrAAAAQyhWAAAAhlCsAAAADKFYAQAAGEKxAgAAMIRiBQAAYAjFCgAAwBCKFQAAgCEUKwAAAEMoVgAAAIZQrAAAAAyhWAEAABhCsQIAADCEYgUAAGCII5gXz507V7169ZLdblePHj1UUFCg5uZmFRUV6fjx4+rXr5+effZZJSQkmFovAABA2AqqWElSfn6+kpKSArdLS0uVmZmpKVOmqLS0VKWlpfrFL34RbAwAAEDYM34qsKqqShMnTpQkTZw4UVVVVaYjAAAAwlLQO1ZLly6VJN13331yu91qbGxUSkqKJKlPnz5qbGwMNgIAAKBbCKpYLVmyRE6nU42NjXrttdeUmpp60eM2m002m+2yr/V4PPJ4PJKkgoICuVyuYJbSJUK15mByHA5HSNYZipxImiVUOcwS3TnMEt05kTRLKHOCFVSxcjqdkqTk5GRlZ2erpqZGycnJ8vl8SklJkc/nu+j6qwu53W653e7A7fr6+mCW0iVCteZgclwuV0jWGYqcSJolVDnMEt05zBLdOZE0Syhzrsb/biRd6LqvsTpz5oxOnz4d+PWXX36pIUOGKCsrS+Xl5ZKk8vJyZWdnX28EAABAt3LdO1aNjY1atWqVJKm9vV133323Ro8erRtvvFFFRUUqKysLfNwCAABANLjuYjVgwACtXLnykvsTExOVl5cX1KIAAAC6Iz55HQAAwBCKFQAAgCEUKwAAAEMoVgAAAIZQrAAAAAyhWAEAABhCsQIAADCEYgUAAGAIxQoAAMCQoL6EGV0n7Y9pRo935JdHjB4PAIBoxI4VAACAIexYAd0QO5YAEJ7YsQIAADCEHSt0OdO7LxI7MACArsGOFQAAgCEUKwAAAEMoVgAAAIZwjRWuiGufAAC4NhQrAFGBj6gAEAqcCgQAADCEHSsA6GbYfQPCF8UKUYMfRuGJa/kARBKKFQCgy1CsEWm4xgoAAMAQdqwAg/jXNwBEN4oVAOCyuC4RuHaWFas9e/Zo48aN6ujo0L333qspU6ZYFQUAYYEdy/BFSUSoWHKNVUdHh/70pz9p0aJFKioq0meffabDhw9bEQUAABA2LNmxqqmp0cCBAzVgwABJ0rhx41RVVaVBgwZZEQcAQJdjxxKSZPP7/X7TB925c6f27NmjZ555RpJUUVGhr7/+Wk8++WTgOR6PRx6PR5JUUFBgegkAAAAh12Uft+B2u1VQUBCWperFF1+MiIxIy4mkWUKVwyzRncMs0Z0TSbOEMidYlhQrp9OphoaGwO2GhgY5nU4rogAAAMKGJcXqxhtv1LFjx1RXV6e2tjZVVlYqKyvLiigAAICw0WPx4sWLTR/Ubrdr4MCBWrdunT755BONHz9eOTk5pmMslZ6eHhEZkZYTSbOEKodZojuHWaI7J5JmCWVOMCy5eB0AACAa8V2BAAAAhlCsAAAADLHkGitEhxMnTmjTpk165513tG3bNh04cEDDhg1TXFxcVy8t6h05ckQrVqzQRx99pMrKSo0ZM0Y9e/Y0nvOHP/xB8fHxcrlcxo/9v0pKSnTgwAHddNNNluaEYqa//e1vstlsOnjwoD7//HPdfPPNlmV1d36/XzabTSUlJcrIyAjcRmRramrSa6+9po8++kgpKSlKSzv34asrVqzQzTffrN69e3fxCq+MHasusnjxYtXV1XX1Mq5bbW2tli1bpptuukkFBQX63e9+p7vuukurVq1SbW2t8by9e/equLjY+HEj2bx581RYWKibbrpJ//rXvyzJ+PrrrzVy5EhLjt1VQjHT+Yx9+/Z1+1I1d+5cS4//6aefavPmzWptbdWHH36oTz/91NK8s2fPKj8/Xx0dHZZltLW1KT8/X+3t7ZZlhFJdXZ22b99u9Jg7duzQfffdp2XLlunjjz+WJO3atUvDhg0L+49voljhumzYsEFz587VuHHj5HCc+2akzMxMzZs3T2+//XYXrw5paWmBr5RqbW1VTEyM8YzDhw/rhhtukN0eOX+NWD3TX/7yFz3//PM6cOCAXn75ZZWVlWnDhg167733LMmLBBMmTJDT6dTmzZvlcrk0YcIES/PKysp05513Wvr/tcPh0K233qrKykrLMkJl69atWr58ud59910tXrxYJ06cMHJch8Ohs2fPqrW1VXa7Xe3t7fr444/16KOPGjm+lSz5rkCEj+XLl+vpp5822vCPHj2qpKQkDR06VF988YVKSkrUv39/+f1+Pf/887Lb7WpqalJSUpKxzEjwz3/+U5WVlXI4HJo0aZJuvvlmVVVVadSoUZbtkOzZs0d79uzRa6+9ZsmxR48eLencX66SdP/99xvPCaULZ7LCrFmzNHbsWFVUVGj27Nl65ZVXtGTJEqMZeXl5On369GWzb7vtNqNZkiz/c75jxw55vV498sgjqq+v144dO3T33Xdbmjd//nzLjn9edna2Nm3apPHjx1ueZZXTp0+rpKREixYt0qFDh3TLLbcYu+Tg7rvv1po1a+TxeDRz5kxt2bJFEyZMsOSSBtMoVhHupZdeMn7M7777TiNGjFBHR4fee+895eXl6dSpU/rNb34jSRo4cKDq6uqM/IW7aNEitba26syZM2pubtbChQslSTNnzrT0B6AVGhsbtWTJEn3//fd6//33tXnzZo0dO1bDhw+3JK+jo0Pr169Xfn6+4uPjjR//3//+t371q19J6v6F6rwLZ7LKt99+q6FDh+ro0aOB60ZMevXVV40f8/+zfPlyS49/1113Ba6xevTRR2XlJwS1tbXp+++/V//+/S3LOG/IkCGqqamxPMdKNptNNptNzc3NkmT09y0uLi7w86u5uVmlpaVauHCh1q9fr5aWFv30pz8N28sQKFYX+OSTT7Rt2zZJ5wpJuJ/H7Urnd6UGDBig+Ph4xcfHa9CgQZLOXXSYnJxsJGfZsmWSzl1jtX37dsuv57DSzJkzJUmpqamaN2+e5Xk+n09xcXG64YYbjB/7hx9+UEtLS0j/jEyfPt3S41s908GDB1VcXCyv16vExET98MMPkqSFCxdq6dKlio2NNZIT6h0rq52/UP38f38rL1xvamqy5B8hl2O32+VwOHT69OmwvhD7/9OrVy89/fTT2rRpk06cOKFDhw7p5z//ufFdpffff19Tp07Vjh07NGrUKOXk5KiwsFAvv/yy0RxTKFYXePDBB/Xggw929TLC3uDBg1VaWqr77rtP33//vU6dOqVTp07pyJEjOnTokBobG9WvX7+uXmbUi4+P1+zZsy059t69e5WRkWHJsbuK1TMNGzZMK1eu1G9/+1u9+uqreuutt/Too48G/kFiSqh3rCJJbGysWltbQ5bX1tZmyfWPoZSVlaUhQ4boiy++0IEDB/SPf/xD06ZNM3b8Y8eOqaGhQRkZGfruu+8UGxsrm82ms2fPGsswLXKuOsVlLV++XF6v1+gxBw0apIaGBh09elRTp07VK6+8oj//+c8aM2aMNm/ebPmpFCtZ8fvVVU6dOhXYgTWturr6olOxW7duDVxnZZWtW7eqvLzcsuP/70xWOL8jYrfbdfToUeOlCsFJSEhQR0dHSH5onzx5UomJiYE3/3RHZ86c0fHjxyVJvXv31qBBg3TmzBmjGZs2bdKMGTMknTstvHXrVr300kv6yU9+YjTHJL7SpossXrxYubm5ITmXb4XDhw9r3bp1mjlzpjIzMyWdu3bE6/XyhdtR4IUXXtDSpUu79Q+F/xWJM+HavfXWW7rrrrssP226c+dO7d+/37Jd5VBobm7WmjVr1NzcrKamJrlcLv3617+O+stoKFZdpLsXK0lqaGjQ+++/r5qaGnV0dGj48OH62c9+xmlAAN3WN998o48++sjy6yBXrVqlxx57TKmpqcaPffLkycueEs7Ly1NiYqLxvLq6Ou3bt0+TJk0yfuzuiH+a4br17dtXc+bM6eplAIAx6enpysjIUEdHh2WfZdXW1qbs7GxLSpUkJSYmauXKlZYc+3Li4+M1bNiwkOWFO3asusj27duVnZ0dsnegAAAA61GsAAAADOFdgQAAAIZQrAAAAAyhWAEAABhCsQIAADDk/wDfN6yshUUWDwAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 720x360 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MCreUhphMh7A"
      },
      "source": [
        "Observation:\r\n",
        "\r\n",
        "Both disaster and non-disaster tweets have similar distribution of puctuations."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SmSc9qGDM15C"
      },
      "source": [
        "## 1.6 Common Words"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 282
        },
        "id": "3kpjFK5TM6A0",
        "outputId": "64f277c0-65b1-42f0-dec1-ec18853a79f6"
      },
      "source": [
        "# Plot the most common words in training set\r\n",
        "counter=Counter(corpus)\r\n",
        "most=counter.most_common()\r\n",
        "x=[]\r\n",
        "y=[]\r\n",
        "for word,count in most[:40]:\r\n",
        "    if (word not in stop) :\r\n",
        "        x.append(word)\r\n",
        "        y.append(count)\r\n",
        "\r\n",
        "sns.barplot(x=y,y=x)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<matplotlib.axes._subplots.AxesSubplot at 0x7f9e7c3d7358>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 41
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAD4CAYAAAAD6PrjAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAUUUlEQVR4nO3df2zU9eHH8ddBW8q1tHA9CrZSWUtZFjBma1G+OgeTi19ChC2bM7iM6Xf5zh91dEWqXEMYGwuzQijMwdaijRjiXJgCBpK57VRQpoxSrBlMC0zQIpRyHLSF0h/Xvr9/kL3jDfiC0t7nrvd8/NXez9e9805ffd/n/blzGWOMAACQNMTpAACA2EEpAAAsSgEAYFEKAACLUgAAWJQCAMBKcjrA9Tp+/LjTEWKG1+tVMBh0OkbMYDwiMR6REnk8cnJyrngdKwUAgBX3K4XkV99wOkLMaJWU7HSIGMJ4RGI8IsXzePR8664Be2xWCgAAi1IAAFiUAgDAohQAABalAACwKAUAgEUpAACsmC2FefPmOR0BABJOzJYCACD6KAUAgBV3H3MRCAQUCAQkSZWVlQ6nAYDBJe5KwefzyefzOR0DAAYl3j4CAFiUAgDAohQAAFbMlsLGjRudjgAACSdmSwEAEH2UAgDAohQAABalAACwKAUAgBV3ZzT/p55v3eV0hJjh9XoVDAadjhEzGI9IjEckxuPyWCkAACxKAQBgUQoAAItSAABYcX+g+dNtDzsdIWZ86nSAKMqdXeN0BGBQYqUAALAoBQCARSkAACxKAQBgUQoAAItSAABYlAIAwKIUAAAWpQAAsCgFAIAVdx9zEQgEFAgEJEmVlZUOpwGAwSXuSsHn88nn8zkdAwAGpZgshddee02vv/66JKmiokIej8fhRACQGGKyFGbOnKmZM2c6HQMAEg4HmgEAFqUAALAoBQCARSkAACxKAQBgUQoAAItSAABYMXmewueRO7vG6Qgxw+v1KhgMOh0DQBxjpQAAsCgFAIBFKQAALEoBAGDF/YHmba//j9MR8BmzZzzvdAQA14GVAgDAohQAABalAACwKAUAgEUpAAAsSgEAYFEKAABrwM5TaG9v17JlyyRJZ8+e1ZAhQ5SRkaFTp05p1KhRWr169UA9NQDgCxqwUhgxYoRWrlwpSdq0aZNSU1M1Z84ctbS06Omnnx6opwUAXAdHzmju6+tTdXW1Dh48KI/HoyeffFIpKSlqbm5WbW2t2traNGzYMD388MPKzc11IiIAJCRHjimcOHFCM2fOVFVVldxut3bv3i1JWr9+vX70ox/p6aef1rx58/Tcc89dct9AICC/3y+/3x/t2AAw6DmyUsjOztb48eMlSfn5+Tp16pQ6OzvV2Nioqqoqe7twOHzJfX0+n3w+X7SiAkBCcaQUkpOT7c9DhgxRd3e3+vr6lJaWZo9DAACiL2a2pLrdbmVnZ+vdd9+VJBljdPToUWdDAUCCiamPzi4tLdWzzz6rzZs3KxwO64477rBvMwEABp7LGGOcDnE9ajb+t9MR8Bmx9H0KXq9XwWDQ6Rgxg/GIlMjjkZOTc8XrYubtIwCA8ygFAIBFKQAALEoBAGBRCgAAK6a2pH4RsbTbxWmJvJsCQP9gpQAAsCgFAIBFKQAALEoBAGBRCgAAK+53H/3vnlVOR4hbz9260OkIAGIMKwUAgEUpAAAsSgEAYFEKAACLUgAAWJQCAMAa8FKYN2+eJCkUCmnVqovbR3fs2KHa2tqBfmoAwOcUtZWCx+PRwoXsiweAWBa1UmhpablsKezbt0+LFy9WW1ub3n//fS1evFiLFi1SVVWVOjs7oxUPACCHz2jes2ePtm/froqKCvX19Wnz5s1asmSJUlNTtXXrVm3fvl333ntvxH0CgYACgYAkqbKy0onYADBoOVYK+/fv10cffaTFixfL7Xarvr5ex44d05IlSyRJ4XBYEydOvOR+Pp9PPp8v2nEBICE4VgpjxoxRS0uLTpw4oYKCAhljdPPNN6usrMypSACQ8Bzbkjp69GgtXLhQa9euVVNTkyZOnKjGxkY1NzdLkjo7O3X8+HGn4gFAQnL0mEJubq5KS0tVVVWlRYsW6bHHHtOvf/1r9fT0SJLmzp2rnJwcJyMCQEJxGWOM0yGux6ytbHP9ogb7R2d7vV4Fg0GnY8QMxiNSIo/H//fPNmc0AwAsSgEAYFEKAACLUgAAWJQCAMBydEtqfxjsO2g+j0TeTQGgf7BSAABYlAIAwKIUAAAWpQAAsCgFAIAV97uPHnrnTacjOGr97d90OgKAQYSVAgDAohQAABalAACwKAUAgEUpAAAsSgEAYFEKAADrms5TaGtr0+rVq9Xe3q7k5GQtXbpUqampA50NABBl11QKf/nLX/SVr3xF9913n0KhkJKS4v6cNwDAZVzTX/ekpCSdOHFCkuTxeOzlnZ2dWrFihc6fP69wOKy5c+dqypQpamlp0a9+9SsVFhbq4MGDKigo0PTp0/XHP/5Rra2tKi0t1YQJE7Rp0yadPHlSzc3Nam9v15w5c+Tz+SRJTzzxhFauXDkALxkAcCXXVApjx47Vq6++qoKCAt1999328uTkZJWXl8vtdqutrU2LFy9WcXGxJKm5uVmPP/64brzxRlVUVGjXrl1atmyZ9u7dq82bN+vJJ5+UJH3yySdavny5Ojs7tWjRIn3ta1+Tx+O5YiEEAgEFAgFJUmVl5XW9eABApKuWQigU0pYtW/TMM89o+fLlysjI0NSpU1VeXq5ly5bppZde0gcffCCXy6VQKKTW1lZJUnZ2tvLy8iRJ48aN08033yyXy6W8vDydOnXKPn5xcbFSUlKUkpKiSZMm6fDhw7r11luvmMfn89nVBACgf121FD788EPl5eVpxIgR8vv9+uUvf6nW1laNHj1ae/bsUVtbmyorK5WUlKTHHntM3d3dki6uIv7N5XLZ310ul/r6+iKu+6z//B0AED1X3ZJ600036cCBAwqFQho5cqQeeOAB1dbW6utf/7o6OjqUmZmppKQk7d+/P2IFcK3q6urU3d2t9vZ2HThwQAUFBZKksrKyz/9qAADX5aorhdzcXM2dO1fLly9XUlKSMjMzVVZWphdffFELFizQ3/72Ny1cuFAFBQXKzc393AFuuukm/eIXv1B7e7u++93vyuPxqK2tTcaYL/SCAABfnMs4+Nd306ZNSk1N1Zw5cyIur6+v18mTJzVr1qyrPsY9L784UPHiwme/T8Hr9SoYDDqYJrYwHpEYj0iJPB45OTlXvC4mTzgoKipyOgIAJCRHS+G+++5z8ukBAP+Bzz4CAFiUAgDAohQAAFZMHmj+PD67+wYAcH1YKQAALEoBAGBRCgAAi1IAAFhxf6C59N1mpyM44pn/Gut0BACDECsFAIBFKQAALEoBAGBRCgAAi1IAAFiUAgDAohQAAFbUSmHevHmSpJaWFv385z+P1tMCAD4HVgoAACvqZzQPGTJE6enpkqQdO3Zoz5496urqUnNzs2bPnq1wOKy33npLycnJqqiosLcFAAy8qK8UvF6vysvL7e9NTU0qLy/XU089pZdeekkpKSlasWKFCgsLtXPnzkvuHwgE5Pf75ff7oxkbABKC4599NGnSJA0fPlzDhw+X2+1WcXGxJCkvL0+ffPLJJbf3+Xzy+XzRjgkACcHxYwrJycn25yFDhigpKcn+3Nvb61QsAEhIjpcCACB2UAoAAMtljDFOh7ge976yz+kIjrjc9yl4vV4Fg0EH0sQmxiMS4xEpkccjJyfnitexUgAAWJQCAMCiFAAAFqUAALAoBQCA5fgZzdfrcrtwAABfDCsFAIBFKQAALEoBAGBRCgAAK+4PNL//dorTERxxy53dTkcAMAixUgAAWJQCAMCiFAAAFqUAALAoBQCARSkAACxKAQBgOXqewqeffqrf/e53unDhgtLT07VgwQKtX79eJ0+elCQ98sgjKiwsdDIiACQUx09emz9/vsaMGaPf//73CgQCmjVrliZPnqyGhgb94Q9/0JIlS5yOCAAJw9FSyM3NtT/39PQoPT1dkydPliSFw2ElJyc7FQ0AElJMHFNoaGhQQ0ODZsyYIUkKBoN64YUX9L3vfe+S2wYCAfn9fvn9/mjHBIBBz/G3j/r6+lRdXa2lS5cqLS1NkrRhwwbde++9KigouOT2Pp9PPp8v2jEBICE4vlI4c+aM3G63brjhBnvZxx9/rK9+9asOpgKAxOR4KaSlpemHP/xhxGUPPPCA3G63Q4kAIHE5XgodHR16/fXXIy7761//qq6uLocSAUDicvyYgsfj0cKFCyMuq6iocCgNACQ2x1cKAIDYQSkAACxKAQBgUQoAAItSAABYju8+ul633NntdAQAGDRYKQAALEoBAGBRCgAAi1IAAFhxf6B5yEttTke4Ln33ZzgdAQAsVgoAAItSAABYlAIAwKIUAAAWpQAAsCgFAIBFKQAArJgrhT179ujYsWNOxwCAhBRzpVBXV0cpAIBDonJG88svv6y3335bGRkZysrKUn5+vm699VbV1taqra1Nw4YN08MPP6xz585p7969+uc//6lXXnlFCxcu1NixY6MREQCgKJTC4cOH9fe//10rV65Ub2+vFi1apPz8fK1fv14//vGPdcMNN+jQoUN67rnntHTpUhUXF6uoqEhTp0697OMFAgEFAgFJUmVl5UDHB4CEMuCl0NjYqClTpiglJUWSVFRUpJ6eHjU2NqqqqsreLhwOX9Pj+Xw++Xy+AckKAInOkQ/EM8YoLS1NK1eudOLpAQBXMOAHmr/85S+rvr5e3d3d6uzs1L59+5SSkqLs7Gy9++67ki6WxNGjRyVJw4cP14ULFwY6FgDgMgZ8pTBhwgQVFRXpiSeeUGZmpsaNGye3263S0lI9++yz2rx5s8LhsO644w6NHz9et99+u2pqavSnP/1Jjz/+OAeaASCKXMYYM9BP0tnZqdTUVHV1dWnp0qV66KGHlJ+f3y+P3bzqw355HKf05/cpeL1eBYPBfnu8eMd4RGI8IiXyeOTk5FzxuqgcU6ipqdGxY8fU09OjadOm9VshAAD6V1RK4ac//Wk0ngYAcJ1i7oxmAIBzKAUAgEUpAAAsR05e60/9uXsHABIdKwUAgEUpAACsqJy8BgCID3G9UvD7/U5HiCmMRyTGIxLjEYnxuLy4LgUAQP+iFAAAVlyXAl+2E4nxiMR4RGI8IjEel8eBZgCAFdcrBQBA/6IUAABW3H7MRUNDg55//nn19fVpxowZ+va3v+10pAEXDAa1bt06nT17Vi6XSz6fT7NmzdK5c+e0evVqnTp1SqNHj9aCBQuUnp4uY4yef/55vffeexo2bJhKSkoG3XdZ9PX1ye/3y+PxyO/3q6WlRWvWrFF7e7vy8/M1f/58JSUlqaenR2vXrtVHH32kESNGqKysTNnZ2U7H73fnz59XdXW1mpqa5HK59OijjyonJydh58f27dv1xhtvyOVyady4cSopKdHZs2cTeo5clYlDvb295ic/+Ylpbm42PT09pry83DQ1NTkda8CFQiHzr3/9yxhjTEdHhyktLTVNTU1m48aNZsuWLcYYY7Zs2WI2btxojDGmvr7eLF++3PT19ZnGxkZTUVHhWPaBsm3bNrNmzRrz1FNPGWOMWbVqldm1a5cxxpiamhrz5z//2RhjzGuvvWZqamqMMcbs2rXLVFVVORN4gP3mN78xgUDAGGNMT0+POXfuXMLOj9OnT5uSkhLT1dVljLk4N958882EnyNXE5dvHx0+fFhjx47VmDFjlJSUpNtvv111dXVOxxpwo0aNsv/JDR8+XLm5uQqFQqqrq9O0adMkSdOmTbNjsXfvXn3jG9+Qy+XSxIkTdf78eZ05c8ax/P3t9OnT2rdvn2bMmCFJMsbowIEDmjp1qiRp+vTpEWMxffp0SdLUqVO1f/9+mUG2x6Kjo0MffPCB7rrrLklSUlKS0tLSEnZ+SBdXkt3d3ert7VV3d7dGjhyZ0HPkWsTl20ehUEhZWVn296ysLB06dMjBRNHX0tKiI0eOaMKECWptbdWoUaMkSSNHjlRra6uki+Pk9XrtfbKyshQKhext492GDRv0gx/8QBcuXJAktbe3y+12a+jQoZIkj8ejUCgkKXLODB06VG63W+3t7crIGDyfstvS0qKMjAz99re/1ccff6z8/Hw9+OCDCTs/PB6PZs+erUcffVQpKSm65ZZblJ+fn9Bz5FrE5Uoh0XV2dmrVqlV68MEH5Xa7I65zuVxyuVwOJYue+vp6ZWZmDrr3wK9Hb2+vjhw5orvvvlsrVqzQsGHDtHXr1ojbJMr8kKRz586prq5O69atU01NjTo7O9XQ0OB0rJgXlysFj8ej06dP299Pnz4tj8fjYKLoCYfDWrVqle68807ddtttkqTMzEydOXNGo0aN0pkzZ+x/Nh6PR8Fg0N53MI1TY2Oj9u7dq/fee0/d3d26cOGCNmzYoI6ODvX29mro0KEKhUL29f57zmRlZam3t1cdHR0aMWKEw6+if2VlZSkrK0uFhYWSLr4FsnXr1oScH5L0j3/8Q9nZ2fb13nbbbWpsbEzoOXIt4nKlUFBQoBMnTqilpUXhcFjvvPOOiouLnY414Iwxqq6uVm5uru655x57eXFxsXbu3ClJ2rlzp6ZMmWIvf+utt2SM0cGDB+V2uwfNWwPf//73VV1drXXr1qmsrEyTJ09WaWmpJk2apN27d0uSduzYYedFUVGRduzYIUnavXu3Jk2aNOj+Yx45cqSysrJ0/PhxSRf/KN54440JOT8kyev16tChQ+rq6pIxxo5HIs+RaxG3ZzTv27dPL7zwgvr6+vTNb35T3/nOd5yONOA+/PBD/exnP1NeXp6drPfff78KCwu1evVqBYPBS7Yc1tbW6v3331dKSopKSkpUUFDg8KvofwcOHNC2bdvk9/t18uRJrVmzRufOndOXvvQlzZ8/X8nJyeru7tbatWt15MgRpaenq6ysTGPGjHE6er87evSoqqurFQ6HlZ2drZKSEhljEnZ+bNq0Se+8846GDh2q8ePH65FHHlEoFEroOXI1cVsKAID+F5dvHwEABgalAACwKAUAgEUpAAAsSgEAYFEKAACLUgAAWP8H7z3sL1LcC3oAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YdPJwUiwNMsv"
      },
      "source": [
        "From the graphs above, puctuations and stopwords are most common in our training text, which means data cleaning is needed."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ygCjFIg9Npgj"
      },
      "source": [
        "## 1.7 Ngram Analysis"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AjjeM5gLNwSI"
      },
      "source": [
        "We perform a bigram analysis over the tweets.\r\n",
        "Let's check the most common bigrams in tweets."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EoLUSqFMNtBv"
      },
      "source": [
        "# Helper function\r\n",
        "def get_top_tweet_bigrams(corpus, n=None):\r\n",
        "    vec = CountVectorizer(ngram_range=(2, 2)).fit(corpus)\r\n",
        "    bag_of_words = vec.transform(corpus)\r\n",
        "    sum_words = bag_of_words.sum(axis=0) \r\n",
        "    words_freq = [(word, sum_words[0, idx]) for word, idx in vec.vocabulary_.items()]\r\n",
        "    words_freq =sorted(words_freq, key = lambda x: x[1], reverse=True)\r\n",
        "    return words_freq[:n]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 337
        },
        "id": "d_WdDBBdN8y-",
        "outputId": "61268b60-9b30-4d19-c63a-dd0869a2ed22"
      },
      "source": [
        "plt.figure(figsize=(10,5))\r\n",
        "top_tweet_bigrams=get_top_tweet_bigrams(df_train['text'])[:10]\r\n",
        "x,y=map(list,zip(*top_tweet_bigrams))\r\n",
        "sns.barplot(x=y,y=x)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<matplotlib.axes._subplots.AxesSubplot at 0x7f9e859d9898>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 44
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmsAAAEvCAYAAAAabYYDAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3df3RU9Z3/8ddMfhIhQJIGDWghQQmILMvGSLFCgNHa6HGX6rIuDWdpUZdvVAyG0MBZNlpNQbOkLgsUAaEe1J6ihXWhCDiV1HIwbQSBGkhSAqxhCZmMQRJIQpKZ+/2D47QphB/JJPNh8nz8NXfu/dz7vrw153U+d+69NsuyLAEAAMBI9kAXAAAAgI4R1gAAAAxGWAMAADAYYQ0AAMBghDUAAACDEdYAAAAMRlgDAAAwWGigC+hOp06dCnQJ6CFxcXFyu92BLgM9gF73LvS79+jtvU5ISOhwHTNrAAAABgvqmbWw9z8KdAnoIWclhQW6CPQIet270O/ew9Ret/79lECXwMwaAACAyQhrAAAABiOsAQAAGIywBgAAYDDCGgAAgMEIawAAAAa77rDmcrmUnZ192XVFRUWqq6vzLf/617/WhQsXOl8dAABAL+fXmbWioiKdOXPGt7x9+3bCGgAAQBd06qG4Xq9Xq1evVkVFhWJiYrRgwQLt379flZWVWr58ucLDwzV58mTV1dXpxRdfVHR0tPLy8jRz5kxNnTpVhw4d0oABA5SVlaXo6Oh2+/7qq6+0du1auVwuSdITTzyhESNGaNu2bdq9e7ckacqUKXrooYe6eOoAAADm61RYq66u1nPPPac5c+aosLBQxcXFmjhxonbs2KGZM2cqKSlJ0sXLoHl5eb5AduHCBSUlJWnWrFl677339O6772r27Nnt9r1hwwaNGjVKOTk58nq9am5u1rFjx7R7927l5+dLkhYtWqRRo0Zp2LBhXTl3AAAA43XqMmh8fLyGDh0qSUpMTFRtbe01jbPZbJowYYIk6b777lNZWdkl23z++ed64IEHLhZntysqKkplZWVKTU1VZGSkIiMjlZqaqiNHjlwy1ul0Kjc3V7m5uZ05LQAAAON0amYtLOzPb++y2+1qaWnp1MFtNlunxnXE4XDI4XD4dZ8AAACB5NcbDCIjI9XU1NRuubm52bdsWZaKi4slSXv27FFycvIl+7jrrru0a9cuSRd/G9fY2Kjk5GSVlJTowoULam5uVklJiUaOHOnP0gEAAIzUqZm1jqSlpWnt2rUKDw9Xfn6+HA6H8vPzFRMTo7y8PEVEROjo0aPavHmzoqOjNW/evEv2MWvWLK1Zs0YfffSR7Ha7nnzySd1xxx1KS0vTokWLJF28wYDfqwEAgN7AZlmW1VMHmzlzpjZu3NhTh1Ptz97qsWMBAIDg0/r3U3rkOAkJCR2u4w0GAAAABuvRsNaTs2oAAADBgJk1AAAAgxHWAAAADEZYAwAAMJhfH91hmp66gwOBFxcXJ7fbHegy0APode9Cv3sPet0xZtYAAAAMRlgDAAAwGGENAADAYIQ1AAAAgxHWAAAADBbUd4PWb/l/gS7BL6Kn/SzQJQAAgABhZg0AAMBghDUAAACDEdYAAAAMRlgDAAAwGGENAADAYIQ1AAAAg10xrLlcLmVnZ192XVFRkerq6nzLv/71r3XhwgX/VgcAANDLdXpmraioSGfOnPEtb9++nbAGAADgZ1d9KK7X69Xq1atVUVGhmJgYLViwQPv371dlZaWWL1+u8PBwTZ48WXV1dXrxxRcVHR2tvLw8zZw5U1OnTtWhQ4c0YMAAZWVlKTo6Wtu3b9eHH36okJAQDRkyRFlZWZcc76233tLBgwdls9k0depUffe739Uf//hHbdy4UR6PR0lJSXryyScVFhbWbf8wAAAAJrhqWKuurtZzzz2nOXPmqLCwUMXFxZo4caJ27NihmTNnKikpSdLFy6B5eXmKjo6WJF24cEFJSUmaNWuW3nvvPb377ruaPXu23n//fa1YsUJhYWE6f/78JcdzOp2qra3Vq6++qpCQEJ07d04tLS1atWqVFi9erISEBK1YsUK7du3SQw895Od/DgAAALNc9TJofHy8hg4dKklKTExUbW3tNe3YZrNpwoQJkqT77rtPZWVlkqTbbrtNy5cv18cff6yQkJBLxh06dEj333+/b13fvn116tQpxcfHKyEhQZI0adIkHTly5JKxTqdTubm5ys3NvaYaAQAATHfVmbW/vNRot9vV0tLSqQPZbDZJ0sKFC3X48GHt27dPW7Zs0X/8x39cNrR1hsPhkMPh8Mu+AAAATNDpGwwiIyPV1NTUbrm5udm3bFmWiouLJUl79uxRcnKyvF6v3G63Ro8ere9///tqbGxsN0aSxowZow8//FAej0eSdO7cOSUkJMjlcun06dOSpI8//lijRo3qbOkAAAA3jKvOrHUkLS1Na9euVXh4uPLz8+VwOJSfn6+YmBjl5eUpIiJCR48e1ebNmxUdHa158+bJ6/Xqv/7rv9TY2ChJ+u53v6ubbrqp3X6nTp2q6upqzZ8/X6GhoZo6daoefPBBZWZmqrCw0HeDwf3339+1MwcAALgB2CzLsrpjxzNnztTGjRu7Y9fXrGzl3wf0+P4SPe1ngS7BeHFxcXK73YEuAz2AXvcu9Lv36O29/vp3+ZfDGwwAAAAM1m1hLdCzagAAAMGAmTUAAACDEdYAAAAMRlgDAAAwGGENAADAYJ1+ztqNgEdeAACAGx0zawAAAAYjrAEAABiMsAYAAGAwwhoAAIDBgvoGg707fxjoEi4x4TvrA10CAAC4gTCzBgAAYDDCGgAAgMEIawAAAAYjrAEAABiMsAYAAGAwwhoAAIDB/B7W/u3f/u26ti8qKlJdXZ1v+emnn1Z9fb2/ywIAALgh+T2svfzyy9e1fVFRkc6cOePvMgAAAIKC3x+KO3PmTG3cuFGlpaV699131a9fP1VVVSkxMVHPPvusbDabb9vi4mJVVlZq+fLlCg8PV35+viRpx44d2rdvn9ra2vT8889r8ODBam5u1vr161VVVSWPx6N//Md/1N133+3v8gEAAIzSrb9ZO378uGbNmqXCwkLV1NSovLy83frx48crKSlJc+fOVUFBgcLDwyVJ/fr10yuvvKIHHnhAW7dulSRt3rxZo0eP1pIlS5SXl6e33npLzc3N3Vk+AABAwHXr66aGDx+u2NhYSdLQoUPlcrmUnJx81XH33HOPJCkxMVF/+MMfJEmHDh3Svn37fOGtpaVFbrdbQ4YM8Y1zOp1yOp2SpKVLl/r1XAAAAAKhW8NaWFiY77PdbpfX672mcaGhob4xHo9HkmRZlrKzs5WQkNDhOIfDIYfD0YWKAQAAzBLwR3dERkaqqanpqtv9zd/8jT744ANZliXp4iVWAACAYBfwsJaWlqa1a9cqJydHLS0tHW732GOPyePxaP78+Xr++ef1y1/+sgerBAAACAyb9fVUVRB6b8ODgS7hEhO+sz7QJQSluLg4ud3uQJeBHkCvexf63Xv09l5f6WdeAZ9ZAwAAQMcIawAAAAYjrAEAABiMsAYAAGAwwhoAAIDBCGsAAAAG69Y3GAQaj8kAAAA3OmbWAAAADEZYAwAAMBhhDQAAwGCENQAAAIMF9Q0Ga/b8IGDHfurbGwJ2bAAAEDyYWQMAADAYYQ0AAMBghDUAAACDEdYAAAAMRlgDAAAwGGENAADAYD0S1rZv36558+Zp+fLl7b4/ceKE9u/f71vetGmT/ud//qcnSgIAALgh9Mhz1nbt2qXFixcrNja23fcnTpxQZWWlxo0b1xNlAAAA3HD8Gta2bdum3bt3S5KmTJmihx56SGvWrFFNTY1+8pOfaPLkyXr44YclSW1tbfrlL3+plpYWlZWVadq0aZKkkydP6oUXXpDb7VZ6errS09MlSR9//LE++OADtbW16fbbb9cTTzwhu52ruAAAILj5LawdO3ZMu3fvVn5+viRp0aJFGjVqlJ566ikdPHhQeXl5io6O/vOBQ0P1T//0T6qsrNTs2bMlXbwMeurUKeXl5ampqUlZWVl64IEHdPr0ae3du1cvvfSSQkNDtW7dOv3ud7/TpEmT/FU+AACAkfwW1srKypSamqrIyEhJUmpqqo4cOaJhw4Zd137GjRunsLAwhYWFqX///jp79qw+//xzHT9+XAsXLpQktbS0tAt+X3M6nXI6nZKkpUuXdvGMAAAAAs+4d4OGhv65JLvdLo/HI8uyNGnSJM2YMeOKYx0OhxwOR3eXCAAA0GP89qOv5ORklZSU6MKFC2publZJSYlGjhx5xTGRkZFqamq66r7vuusuFRcX6+zZs5Kkc+fOqba21i91AwAAmMxvM2uJiYlKS0vTokWLJF28weBql0BHjx6t999/Xzk5Ob4bDC5nyJAhevzxx/Xyyy/LsiyFhIRo9uzZ+sY3vuGv8gEAAIxksyzLCnQR3eWFTd8J2LGf+vaGgB27N4qLi5Pb7Q50GegB9Lp3od+9R2/vdUJCQofrePYFAACAwQhrAAAABiOsAQAAGIywBgAAYDDCGgAAgMEIawAAAAYz7g0G/sTjMwAAwI2OmTUAAACDEdYAAAAMRlgDAAAwGGENAADAYEF9g8ETv1/fI8dZd88Pe+Q4AACg92FmDQAAwGCENQAAAIMR1gAAAAxGWAMAADAYYQ0AAMBghDUAAACDdSmsffHFFyouLvZXLQAAAPgrnQ5rlmVp06ZN+v3vfy+Xy3XN41auXHnZgOdyubRnzx7f8okTJ7R///7OlgcAABAUOh3W3G63HnnkEf3whz/U6dOnu1xIbW3tJWHts88+6/J+AQAAbmSdeoOBy+XSK6+8omXLlkm6GKzKyso0ffr0axp/+PBhbdu2TV999ZUyMjI0fvx4vfPOOzp58qRycnJ07733aufOnWppaVFZWZmmTZumkydPqqamRqdPn1ZDQ4MeeeQRORyOzpQPAABwwwjI66a++uor/fjHP9apU6f0yiuvaPz48ZoxY4a2bt2q3NxcSdKAAQNUWVmp2bNnS5I2bdqkL774Qvn5+WpubtaPfvQjjRs3TjExMb79Op1OOZ1OSdLSpUt7/sQAAAD8LCBh7e6775bdbteQIUN09uzZax6XkpKi8PBwhYeH684779TRo0eVmprqW+9wOJhtAwAAQaVTv1kLCQmR1+v1Lbe2tl7X+LCwMN9ny7KueZzNZrviMgAAQLDpVFjr37+/6uvr1dDQoNbWVr/ctdmnTx81NTX5liMjI9stS1JJSYlaWlrU0NCg0tJSJSUldfm4AAAAJuvUZdDQ0FA9+uijWrRokWJiYpSQkNDlQm677TbZ7Xbl5ORo0qRJSktL0/vvv6+cnBxNmzZNkvTNb35TL774ohoaGvToo4+2+70aAABAMLJZ13MdMoA2bdqkyMhIPfLII9c8Jn3Ly91Y0Z+tu+eHPXIcdCwuLk5utzvQZaAH0OvehX73Hr2911ea+OJ1UwAAAAbz692g69atU3l5ebvv0tPTNXny5C7v+1qf4QYAABBM/BrWnnjiCX/uDgAAoNfjMigAAIDBCGsAAAAGC8gbDHoKd2kCAIAbHTNrAAAABiOsAQAAGIywBgAAYDDCGgAAgMEIawAAAAYL6rtBn9q7u9Nj10zo+lsXAAAAuoqZNQAAAIMR1gAAAAxGWAMAADAYYQ0AAMBghDUAAACDEdYAAAAMFvCw9oc//EEnT570Lb/wwguqrKwMYEUAAADmCHhYKykpaRfWAAAA8Gd+eSjutm3btHv3xQfQTpkyRQ899JBcLpeWLFmiESNGqKKiQjExMVqwYIHCw8N948rLy/Xpp5/q8OHD+tWvfqXs7GxJ0ieffKJ169apsbFRc+bM0ciRI+X1evX222/r8OHDam1t1Xe+8x3df//9/igfAADAWF2eWTt27Jh2796t/Px85efn6ze/+Y2OHz8uSaqurtaDDz6owsJCRUVFqbi4uN3YESNGKCUlRTNnzlRBQYFuvvlmSZLX69WSJUv0L//yL3rvvfckSR999JGioqK0ZMkSLVmyRL/5zW/kcrna7c/pdCo3N1e5ubldPS0AAAAjdHlmraysTKmpqYqMjJQkpaam6siRI0pJSVF8fLyGDh0qSUpMTFRtbe017TM1NdU35utAdvDgQX3xxRe+wNfY2Kjq6mrFx8f7xjkcDjkcjq6eEgAAgDG69d2gYWFhvs92u10tLS3XNc5ut8vr9UqSLMvSD37wA40dO9b/hQIAABiqy5dBk5OTVVJSogsXLqi5uVklJSUaOXLkNY/v06ePmpqarrrd2LFjtWvXLrW1tUmSTp06pebm5k7XDQAAcCPo8sxaYmKi0tLStGjRIkkXbzAYNmzYJb8n68iECRP0+uuv64MPPtDzzz/f4XZTpkyRy+XSj370I0lSdHS0cnJyulo+AACA0WyWZVmBLqK7PPze250eu2bCZD9Wgu4WFxcnt9sd6DLQA+h170K/e4/e3uuEhIQO1wX8OWsAAADoGGENAADAYIQ1AAAAgxHWAAAADEZYAwAAMFi3PhQ30LijEwAA3OiYWQMAADAYYQ0AAMBghDUAAACDEdYAAAAMRlgDAAAwWFDfDfr0J8c6PXbltxL9WAkAAEDnMLMGAABgMMIaAACAwQhrAAAABiOsAQAAGIywBgAAYDDCGgAAgMH8FtbOnz+vnTt3dmlMaWmpli5d6q+SAAAAbnh+DWu7du3q9jEAAAC9id8eivvOO+/o9OnTysnJ0ZgxY5SRkaG33npLBw4ckCQ9+uijmjBhwhXHjBs3Ts3NzVq2bJmqqqqUmJioZ599VjabTceOHdObb76p5uZmRUdHKzMzUwMHDvRX+QAAAEbyW1ibMWOGqqqqVFBQIEkqLi7WiRMnVFBQoPr6ei1cuFAjR45sF7D+ekxpaamOHz+uwsJCDRw4UIsXL1Z5ebmGDx+u9evXa8GCBYqOjtbevXv1i1/8QpmZmf4qHwAAwEjd9rqpsrIy3XvvvbLb7RowYIBGjRqlyspKpaSkXHHc8OHDFRsbK0kaOnSoXC6XoqKiVFVVpZdeekmS5PV6Lzur5nQ65XQ6JYnfvgEAgKBg3LtBw8LCfJ/tdru8Xq8kaciQIcrPz7/iWIfDIYfD0a31AQAA9CS/3WDQp08fNTU1+ZZHjhypTz75RF6vV/X19Tpy5IiGDx9+xTEdSUhIUH19vSoqKiRJbW1tqqqq8lfpAAAAxvLbzFq/fv00YsQIZWdna+zYscrIyFBFRYVycnIkSRkZGRowYMAVx4wbN+7yRYaGKjs7Wxs2bFBjY6M8Ho/S09N16623+qt8AAAAI9ksy7ICXUR3mfarPZ0eu/JbiX6sBN0tLi5Obrc70GWgB9Dr3oV+9x69vdcJCQkdruMNBgAAAAYjrAEAABiMsAYAAGAwwhoAAIDBCGsAAAAGI6wBAAAYzLg3GPgTj98AAAA3OmbWAAAADEZYAwAAMBhhDQAAwGCENQAAAIMF9Q0GW34fct1jpt3j6YZKAAAAOoeZNQAAAIMR1gAAAAxGWAMAADAYYQ0AAMBghDUAAACDEdYAAAAM1m1h7fz589q5c+d1jXG5XMrOzu6migAAAG483RrWdu3a1V27BwAA6BW67aG477zzjk6fPq2cnByNGTNGGRkZeuutt3TgwAFJ0qOPPqoJEyZcMs7j8Wj58uU6fvy4hgwZomeeeUYRERE6duyY3nzzTTU3Nys6OlqZmZkaOHBgd5UPAABghG4LazNmzFBVVZUKCgokScXFxTpx4oQKCgpUX1+vhQsXauTIkZcErlOnTmnOnDlKTk7WqlWrtHPnTqWnp2v9+vVasGCBoqOjtXfvXv3iF79QZmZmd5UPAABghB573VRZWZnuvfde2e12DRgwQKNGjVJlZaVSUlLabRcbG6vk5GRJ0sSJE7V9+3aNHTtWVVVVeumllyRJXq/3srNqTqdTTqdTkrR06dJuPiMAAIDuZ9y7QW0222WXhwwZovz8/CuOdTgccjgc3VYbAABAT+u2Gwz69OmjpqYm3/LIkSP1ySefyOv1qr6+XkeOHNHw4cMvGed2u1VRUSFJ2rNnj5KTk5WQkKD6+nrf921tbaqqququ0gEAAIzRbTNr/fr104gRI5Sdna2xY8cqIyNDFRUVysnJkSRlZGRowIABl4xLSEjQjh079LOf/UyDBw/WAw88oNDQUGVnZ2vDhg1qbGyUx+NRenq6br311u4qHwAAwAg2y7KsQBfRXVZuqbnuMdPu8XRDJehucXFxcrvdgS4DPYBe9y70u/fo7b1OSEjocB1vMAAAADAYYQ0AAMBghDUAAACDEdYAAAAMRlgDAAAwGGENAADAYMa9wcCfeAwHAAC40TGzBgAAYDDCGgAAgMEIawAAAAYjrAEAABgsqG8wqNl+7ac3KL2tGysBAADoHGbWAAAADEZYAwAAMBhhDQAAwGCENQAAAIMR1gAAAAxGWAMAADBYl8La9u3bNW/ePC1fvrxT410ul/bs2eNbLioq0htvvNGVkgAAAIJKl56ztmvXLi1evFixsbHXtL3H41FISIhvuba2Vnv27NG3v/3trpQBAAAQtDod1tasWaOamhr95Cc/0eTJk5WWlqZVq1bJ5XIpIiJCTz31lL75zW9q06ZNqqmpkcvlUmxsrLKysnz7eOedd3Ty5Enl5ORo0qRJ6tu3r86cOaP8/HzV1NQoNTVVGRkZkqSDBw9q06ZNamtr06BBg5SZmanIyMiu/wsAAAAYrNNh7amnntLBgweVl5en6OhorV+/XsOGDdOCBQv0+eefa8WKFSooKJAknTx5Ui+99JLCw8Pb7WPGjBnaunWrcnNzJV28DHrixAm9+uqrCg0NVVZWlh588EGFh4dr8+bNWrx4sSIjI/Xf//3f2rZtmx577LEunDoAAID5/Pa6qbKyMmVnZ0uSRo8erXPnzqmxsVGSlJKScklQ68jo0aMVFRUlSRoyZIjcbrfOnz+vkydPavHixZKktrY23XHHHZeMdTqdcjqdkqSlS5d2+ZwAAAACrUfeDRoREXHN24aFhfk+2+12eTweWZalu+66q90l1MtxOBxyOBydrhMAAMA0fnt0R3Jysn73u99JkkpLS9WvXz/fDFlH+vTpo6ampqvu+4477lB5eblOnz4tSWpubtapU6e6XjQAAIDh/DazNn36dK1atUrz589XRESEnn766auOue2222S329vdYHA50dHRevrpp/Wf//mfam1tlSQ9/vjjSkhI8Ff5AAAARrJZlmUFuoju8tk61zVvOyi9rRsrQXeLi4uT2+0OdBnoAfS6d6HfvUdv7/WVJqB4gwEAAIDBCGsAAAAGI6wBAAAYjLAGAABgMMIaAACAwQhrAAAABuuRNxgECo/jAAAANzpm1gAAAAxGWAMAADAYYQ0AAMBghDUAAACDBfUNBiGbqq9pO8/0W7q5EgAAgM5hZg0AAMBghDUAAACDEdYAAAAMRlgDAAAwGGENAADAYIQ1AAAAg/VIWNu8efM1rXO5XMrOzu6JkgAAAG4IPRLWtmzZ0ql1AAAAvZ1fH4r76quv6ssvv1Rra6vS09PlcDj09ttvq6WlRTk5Obr11ls1d+5c3/Z/ve7xxx+X1+vV6tWrVVFRoZiYGC1YsEDh4eE6ffq03njjDdXX1ysiIkL/+q//qsGDB/uzfAAAAOP4NaxlZmaqb9++amlp0cKFC3XPPffo+9//vnbs2KGCgoJLtv/rdS6XS9XV1Xruuec0Z84cFRYWqri4WBMnTtSaNWv05JNP6pZbbtGf/vQnrVu3Tnl5ef4sHwAAwDh+DWvbt29XSUmJJMntdqu6ulr9+vW7rn3Ex8dr6NChkqTExETV1taqublZ5eXlKiws9G3X1tZ2yVin0ymn0ylJWrp0aSfPAgAAwBx+C2ulpaX64x//qJdfflkRERF64YUX1Nraet37CQsL83222+1qaWmR1+vVTTfddNnZub/kcDjkcDiu+5gAAACm8tsNBo2NjbrpppsUERGh//u//9Of/vQn37rQ0NDLzoRdbd3XoqKiFB8fr08++USSZFmWTpw44a/SAQAAjOW3mbWxY8fqww8/1Lx583TLLbfo9ttv962bOnWqcnJyNGzYsHY3GPz1uscff7zD/c+dO1dr167V5s2b1dbWpnvvvdd3uRQAACBY2SzLsgJdRHepeW3fNW3nmX5LN1eC7hYXFye32x3oMtAD6HXvQr97j97e64SEhA7X8QYDAAAAgxHWAAAADEZYAwAAMBhhDQAAwGCENQAAAIMR1gAAAAzm19dNmYZHcgAAgBsdM2sAAAAGI6wBAAAYLKjfYAAAAHCjC9qZtdzc3ECXgB5Ev3sPet270O/eg153LGjDGgAAQDAgrAEAABgsaMOaw+EIdAnoQfS796DXvQv97j3odce4wQAAAMBgQTuzBgAAEAyC8g0GBw4c0IYNG+T1ejV16lT9wz/8Q6BLwnVatWqV9u/fr/79+2vZsmWSpHPnzumnP/2pamtr9Y1vfEPz5s1T3759ZVmWNmzYoM8++0wRERHKzMxUYmKiJKmoqEibN2+WJH3ve99TWlpaoE4JHXC73Vq5cqW++uor2Ww2ORwOpaen0+8g1dLSory8PLW1tcnj8Wj8+PGaPn26XC6XXnvtNTU0NCgxMVHPPvusQkND1draqhUrVujYsWPq16+fsrKyFB8fL0nasmWLPvroI9ntdv3gBz/Q2LFjA3x2uByv16vc3FzFxMQoNzeXXneGFWQ8Ho/1zDPPWKdPn7ZaW1ut+fPnW1VVVYEuC9eptLTUqqystJ5//nnfdxs3brS2bNliWZZlbdmyxdq4caNlWZa1b98+Kz8/3/J6vVZ5ebm1cOFCy7Isq6GhwXr66aethoaGdp9hlrq6OquystKyLMtqbGy05s6da1VVVdHvIOX1eq2mpibLsiyrtbXVWrhwoVVeXm4tW7bM2rNnj2VZlvX6669bO3futCzLsnbs2GG9/vrrlmVZ1p49e6zCwkLLsiyrqqrKmj9/vtXS0mLV1NRYzzzzjOXxeAJwRriarVu3Wq+99pq1ZMkSy7Iset0JQXcZ9OjRo7r55ps1aNAghYaGasKECSopKQl0WY0fxhMAAAQsSURBVLhOo0aNUt++fdt9V1JSokmTJkmSJk2a5Ovrp59+qokTJ8pms+mOO+7Q+fPndebMGR04cEBjxoxR37591bdvX40ZM0YHDhzo8XPBlQ0cONA3M9anTx8NHjxYdXV19DtI2Ww2RUZGSpI8Ho88Ho9sNptKS0s1fvx4SVJaWlq7fn89Qzp+/Hh9/vnnsixLJSUlmjBhgsLCwhQfH6+bb75ZR48eDcg5oWNffvml9u/fr6lTp0qSLMui150QdGGtrq5OsbGxvuXY2FjV1dUFsCL4y9mzZzVw4EBJ0oABA3T27FlJF3seFxfn2+7rnv/1fwsxMTH8t2A4l8ul48ePa/jw4fQ7iHm9XuXk5OiJJ57QXXfdpUGDBikqKkohISGS2vfuL/saEhKiqKgoNTQ00O8bxM9//nNlZGTIZrNJkhoaGuh1JwRdWEPvYLPZfP/zIzg0Nzdr2bJlmjVrlqKiotqto9/BxW63q6CgQKtXr1ZlZaVOnToV6JLQDfbt26f+/fv7Zs7ReUF3g0FMTIy+/PJL3/KXX36pmJiYAFYEf+nfv7/OnDmjgQMH6syZM4qOjpZ0sedut9u33dc9j4mJ0eHDh33f19XVadSoUT1eN66ura1Ny5Yt03333ad77rlHEv3uDW666SbdeeedqqioUGNjozwej0JCQlRXV+f7u/313/TY2Fh5PB41NjaqX79+l/yt/8sxMEN5ebk+/fRTffbZZ2ppaVFTU5N+/vOf0+tOCLqZtaSkJFVXV8vlcqmtrU179+5VSkpKoMuCH6SkpOi3v/2tJOm3v/2t7r77bt/3H3/8sSzLUkVFhaKiojRw4ECNHTtWBw8e1Llz53Tu3DkdPHiw991BdAOwLEurV6/W4MGD9fDDD/u+p9/Bqb6+XufPn5d08c7QQ4cOafDgwbrzzjtVXFws6eJdvV//3f67v/s7FRUVSZKKi4t15513ymazKSUlRXv37lVra6tcLpeqq6s1fPjwgJwTLm/GjBlavXq1Vq5cqaysLI0ePVpz586l150QlA/F3b9/v9588015vV5NnjxZ3/ve9wJdEq7Ta6+9psOHD6uhoUH9+/fX9OnTdffdd+unP/2p3G73JY9yeOONN3Tw4EGFh4crMzNTSUlJkqSPPvpIW7ZskXTxUQ6TJ08O5GnhMsrKyvTv//7vuu2223yXOv/5n/9Zt99+O/0OQv/7v/+rlStXyuv1yrIsfetb39Jjjz2mmpoavfbaazp37pyGDRumZ599VmFhYWppadGKFSt0/Phx9e3bV1lZWRo0aJAkafPmzdq9e7fsdrtmzZqlv/3bvw3w2aEjpaWl2rp1q3Jzc+l1JwRlWAMAAAgWQXcZFAAAIJgQ1gAAAAxGWAMAADAYYQ0AAMBghDUAAACDEdYAAAAMRlgDAAAwGGENAADAYP8fHJf4fOgQmLgAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 720x360 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Qn4n1qOpOFMo"
      },
      "source": [
        "Observation:\r\n",
        "\r\n",
        "The most common bi-grams are those that do not give us any useful information. This also means that we need data cleaning."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1E0jFHPZN_EN"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4H6p2aWjOS3g"
      },
      "source": [
        "# Step 2: Data Cleaning"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aQln9WbuOYez"
      },
      "source": [
        "For this project, we plan to do some basic cleaning such as spelling correction, removing punctuations, removing html tags and emojis."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vyl0Cg3xOsFV"
      },
      "source": [
        "## 2.1 Remove URL's"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "C_o0fAfaOUnD"
      },
      "source": [
        "# Helper function to remove url from text\r\n",
        "def remove_URL(text):\r\n",
        "    url = re.compile(r'https?://\\S+|www\\.\\S+')\r\n",
        "    return url.sub(r'',text)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Sij6D8OhOzum"
      },
      "source": [
        "# Remove url's from text in both train and test set\r\n",
        "df_train['text'] = df_train['text'].apply(lambda x : remove_URL(x))\r\n",
        "df_test['text'] = df_test['text'].apply(lambda x : remove_URL(x))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "dT1EcqjuPHMY",
        "outputId": "83cb02b3-fe27-4d38-c50a-b44f827182a9"
      },
      "source": [
        "# Have a look at the result text\r\n",
        "df_test['text'].head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0                   Just happened a terrible car crash\n",
              "1    Heard about #earthquake is different cities, s...\n",
              "2    there is a forest fire at spot pond, geese are...\n",
              "3             Apocalypse lighting. #Spokane #wildfires\n",
              "4        Typhoon Soudelor kills 28 in China and Taiwan\n",
              "Name: text, dtype: object"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 48
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A64q-2qGPW4P"
      },
      "source": [
        "## 2.2 Remove HTML Tags"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UzbyAGpTPbCp"
      },
      "source": [
        "# Helper function to remove html tags from text\r\n",
        "def remove_html(text):\r\n",
        "    html=re.compile(r'<.*?>')\r\n",
        "    return html.sub(r'',text)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Gfh9nhlQPhPc"
      },
      "source": [
        "# Remove html tags from text in both train and test set\r\n",
        "df_train['text'] = df_train['text'].apply(lambda x : remove_html(x))\r\n",
        "df_test['text'] = df_test['text'].apply(lambda x : remove_html(x))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "M6XgnX4gPp1c",
        "outputId": "fcde642b-35b3-4f05-f3c5-080c440a2b3d"
      },
      "source": [
        "df_test['text'].head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0                   Just happened a terrible car crash\n",
              "1    Heard about #earthquake is different cities, s...\n",
              "2    there is a forest fire at spot pond, geese are...\n",
              "3             Apocalypse lighting. #Spokane #wildfires\n",
              "4        Typhoon Soudelor kills 28 in China and Taiwan\n",
              "Name: text, dtype: object"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 52
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "j35YeozRPzxB"
      },
      "source": [
        "## 2.3 Remove Emojis"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jXQtAVOkP2zS"
      },
      "source": [
        "# Reference : https://gist.github.com/slowkow/7a7f61f495e3dbb7e3d767f97bd7304b\r\n",
        "# Helper function to remove emoji from text\r\n",
        "def remove_emoji(text):\r\n",
        "    emoji_pattern = re.compile(\"[\"\r\n",
        "                           u\"\\U0001F600-\\U0001F64F\"  # emoticons\r\n",
        "                           u\"\\U0001F300-\\U0001F5FF\"  # symbols & pictographs\r\n",
        "                           u\"\\U0001F680-\\U0001F6FF\"  # transport & map symbols\r\n",
        "                           u\"\\U0001F1E0-\\U0001F1FF\"  # flags (iOS)\r\n",
        "                           u\"\\U00002702-\\U000027B0\"\r\n",
        "                           u\"\\U000024C2-\\U0001F251\"\r\n",
        "                           \"]+\", flags=re.UNICODE)\r\n",
        "    return emoji_pattern.sub(r'', text)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5Gw7doM_P9gv"
      },
      "source": [
        "# Remove emoji from text in both train and test set\r\n",
        "df_train['text'] = df_train['text'].apply(lambda x : remove_emoji(x))\r\n",
        "df_test['text'] = df_test['text'].apply(lambda x : remove_emoji(x))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XvYqbYqbQLuK"
      },
      "source": [
        "## 2.4 Remove Punctuations"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ajU5HlokQOB6"
      },
      "source": [
        "# Helper function to remove punctuations from text\r\n",
        "def remove_punct(text):\r\n",
        "    table=str.maketrans('','',string.punctuation)\r\n",
        "    return text.translate(table)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "U_YO7cLxQTKd"
      },
      "source": [
        "# Remove puctuations from text in both train and test set\r\n",
        "df_train['text'] = df_train['text'].apply(lambda x : remove_punct(x))\r\n",
        "df_test['text'] = df_test['text'].apply(lambda x : remove_punct(x))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Xyz3J0x_Qh6Q"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FlUIc8OEQqLf"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lM4YwN5ERDOx"
      },
      "source": [
        "# Step 3: Check Cleaning Result"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "okOLT27lRM4H"
      },
      "source": [
        "We check the reuslt of cleaning by performing again the bi-gram analysis."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WfobR6fWRGw_"
      },
      "source": [
        "# Helper function\r\n",
        "def get_top_tweet_bigrams(corpus, n=None):\r\n",
        "    vec = CountVectorizer(ngram_range=(2, 2)).fit(corpus)\r\n",
        "    bag_of_words = vec.transform(corpus)\r\n",
        "    sum_words = bag_of_words.sum(axis=0) \r\n",
        "    words_freq = [(word, sum_words[0, idx]) for word, idx in vec.vocabulary_.items()]\r\n",
        "    words_freq =sorted(words_freq, key = lambda x: x[1], reverse=True)\r\n",
        "    return words_freq[:n]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 337
        },
        "id": "eF_WhbxARIuo",
        "outputId": "e8d224a5-bfbb-408b-b94b-e2242b223ac5"
      },
      "source": [
        "plt.figure(figsize=(10,5))\r\n",
        "top_tweet_bigrams=get_top_tweet_bigrams(df_train['text'])[:10]\r\n",
        "x,y=map(list,zip(*top_tweet_bigrams))\r\n",
        "sns.barplot(x=y,y=x)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<matplotlib.axes._subplots.AxesSubplot at 0x7f9e85c82d30>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 58
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmkAAAEvCAYAAAAemFY+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3df3RU9Z3/8ddMfhJJCEkEDcFCDJKgpVnkh6Lyyyl6Us7u+mNdZWGVrcv6jeCCIVg8ZYMHUqIpiBx/VCrkuKIc2QrrqhHoINRSZE2lgkZJMCTbUCDDEEqEZEgyc79/eDoWSQgwM8xnhufjr8yd+7nznve5R1987nzutVmWZQkAAABGsYe7AAAAAJyNkAYAAGAgQhoAAICBCGkAAAAGIqQBAAAYiJAGAABgIEIaAACAgWLDXUCoHDp0KNwlRJ2MjAy53e5wlxF16Gto0NfQobehQV9DIxL6mpmZ2eV2ZtIAAAAMFLUzaXFvfxDuEqLOCUlx4S4iCtHX0KCvoUNvQ4O+hsbF9rXj7yYFu5QLxkwaAACAgQhpAAAABiKkAQAAGIiQBgAAYCBCGgAAgIEIaQAAAAYKWkj76U9/ekH7b9++Xc3Nzf7Xjz76qFpaWoJVDgAAQEQLWkhbsmTJBe2/fft2HT9+PFgfDwAAEFWCdjPb6dOn67XXXlN1dbX+67/+S8nJyWpsbFR2drZmz54tm83m33fXrl2qq6vTypUrFR8fr9LSUknSpk2b9Mknn6izs1OPP/64BgwYII/HozVr1qixsVFer1f/8A//oFGjRgWrbAAAACOF5Ddp9fX1euihh7R8+XI1NTWppqbmjPdvuukmXXvttXrsscdUXl6u+Ph4SVJycrKefvppTZ48We+8844kacOGDbrhhhu0dOlSlZSUaO3atfJ4PKEoGwAAwBgheSxUTk6O0tPTJUmDBg2Sy+VSbm5uj+PGjBkjScrOztbHH38sSdq7d68++eQTf2hrb2+X2+1WVlbWGWOdTqecTqckqaysLGjfBQAAIBxCEtLi4r59SpbdbpfP5zu/YmJj/WO8Xq8kybIsFRUVdfuE+L9wOBxyOBwXWTEAAIBZwnYLjsTERLW1tfW43w9+8AO9//77sixL0jeXUgEAAKJd2ELahAkT9Mtf/lLFxcVqb2/vdr97771XXq9X8+bN0+OPP64333zzElYJAAAQHjbrL1NUUeboS2vDXQIAAIhQHX836ZJ9Vnc/6eKJAwAAAAYipAEAABiIkAYAAGAgQhoAAICBCGkAAAAGCsnNbE1wKVdlXC4yMjLkdrvDXUbUoa+hQV9Dh96GBn0NjUjuKzNpAAAABiKkAQAAGIiQBgAAYCBCGgAAgIEIaQAAAAaK2tWdLRv/X7hLiDot4S4gStHX0KCvoUNvQ+N8+5py10shrQPmYCYNAADAQIQ0AAAAAxHSAAAADERIAwAAMBAhDQAAwECENAAAAAOFNKRVVlZq7ty5Wrly5RnbGxoatHv3bv/r9evX63/+539CWQoAAEBECel90rZs2aKFCxcqPT39jO0NDQ2qq6vTiBEjQvnxAAAAESsoIe3dd9/Vtm3bJEmTJk3Sj370I61atUpNTU362c9+pokTJ2rKlCmSpM7OTr355ptqb2/Xvn37dNddd0mSDh48qEWLFsntdqugoEAFBQWSpA8//FDvv/++Ojs7NWTIED388MOy27lKCwAAolvAIe3AgQPatm2bSktLJUlPPvmkhg0bppkzZ2rPnj0qKSlRSkrKtx8YG6t//Md/VF1dnX784x9L+uZy56FDh1RSUqK2tjbNmTNHkydP1pEjR7Rz504tXrxYsbGxeuWVV/Tb3/5W48ePD7RsAAAAowUc0vbt26fRo0crMTFRkjR69Gh9+eWXGjx48AUdZ8SIEYqLi1NcXJz69OmjEydO6PPPP1d9fb0WLFggSWpvbz8j8P01p9Mpp9MpSSorKwvgGwEAAISfMc/ujI39thS73S6v1yvLsjR+/HhNnTq1x/EOh0MOhyOUJQIAAFwyAf+4Kzc3V1VVVTp9+rQ8Ho+qqqqUl5d3zjGJiYlqa2vr8djf//73tWvXLp04cUKSdPLkSR09ejTQkgEAAIwX8Exadna2JkyYoCeffFLSNwsHerrUecMNN+jtt99WcXGxf+FAV7KysnT//fdryZIlsixLMTEx+vGPf6wrr7wy0LIBAACMZrMsywp3EaGw74W/C3cJAAAEXcpdL4W7hIiSkZEht9sd7jLOKTMzs8vt3MsCAADAQIQ0AAAAAxHSAAAADERIAwAAMBAhDQAAwECENAAAAAMZ88SBYGOJcvBFwjLmSERfQ4O+hg69DQ36iu9iJg0AAMBAhDQAAAADEdIAAAAMREgDAAAwUNQuHNi5+V/CXQJwWRl7x5pwlwAAUYWZNAAAAAMR0gAAAAxESAMAADAQIQ0AAMBAhDQAAAADEdIAAAAMFLaQ9vHHH+vgwYP+14sWLVJdXV24ygEAADBK2EJaVVXVGSENAAAA3wroZrbvvvuutm3bJkmaNGmSfvSjH8nlcmnp0qUaOnSoamtrlZaWpvnz5ys+Pt4/rqamRr///e/1xRdf6K233lJRUZEk6aOPPtIrr7yi1tZWPfLII8rLy5PP59Prr7+uL774Qh0dHbrjjjv0wx/+MJCyAQAAjHfRM2kHDhzQtm3bVFpaqtLSUm3dulX19fWSpMOHD+vOO+/U8uXLlZSUpF27dp0xdujQoRo5cqSmT5+u8vJyXXXVVZIkn8+npUuX6sEHH9SvfvUrSdIHH3ygpKQkLV26VEuXLtXWrVvlcrkutmwAAICIcNEzafv27dPo0aOVmJgoSRo9erS+/PJLjRw5Uv369dOgQYMkSdnZ2Tp69Oh5HXP06NH+MX8JYnv27NEf//hHf9BrbW3V4cOH1a9fvzPGOp1OOZ1OSVJZWdnFfi0AAAAjhOTZnXFxcf6/7Xa72tvbL2ic3W6Xz+eTJFmWpRkzZig/P/+cYx0OhxwOx0VWDAAAYJaLvtyZm5urqqoqnT59Wh6PR1VVVcrLyzvv8b169VJbW1uP++Xn52vLli3q7OyUJB06dEgej+diywYAAIgIFz2Tlp2drQkTJujJJ5+U9M3CgcGDB5/378XGjh2rl19+We+//74ef/zxbvebNGmSXC6XnnjiCUlSSkqKiouLL7ZsAACAiGCzLMsKdxGh8KuKO8NdAnBZGXvHmnCXcIaMjAy53e5wlxGV6G1o0NfQiIS+ZmZmdrmdJw4AAAAYiJAGAABgIEIaAACAgQhpAAAABiKkAQAAGIiQBgAAYKCQPHHABKbdDiAaRMIy5khEXwEAXWEmDQAAwECENAAAAAMR0gAAAAxESAMAADBQ1C4cWLVjRrhLAKLOzFsrwl0CAFw2mEkDAAAwECENAADAQIQ0AAAAAxHSAAAADERIAwAAMBAhDQAAwEABh7RTp05p8+bNAY2prq5WWVlZoKUAAABEjaCEtC1btoR8DAAAwOUk4JvZvvHGGzpy5IiKi4s1fPhwTZs2TWvXrtWnn34qSbrnnns0duzYc44ZMWKEPB6Pli1bpsbGRmVnZ2v27Nmy2Ww6cOCAXn31VXk8HqWkpKiwsFB9+/YNtGwAAACjBRzSpk6dqsbGRpWXl0uSdu3apYaGBpWXl6ulpUULFixQXl7eGcHqu2Oqq6tVX1+v5cuXq2/fvlq4cKFqamqUk5OjNWvWaP78+UpJSdHOnTu1bt06FRYWBlo2AACA0YL+WKh9+/bplltukd1uV2pqqoYNG6a6ujqNHDnynONycnKUnp4uSRo0aJBcLpeSkpLU2NioxYsXS5J8Pl+3s2hOp1NOp1OS+H0bAACIeMY8uzMuLs7/t91ul8/nkyRlZWWptLS0x/EOh0MOhyNk9QEAAFxKAS8c6NWrl9ra2vyv8/Ly9NFHH8nn86mlpUVffvmlcnJyzjmmO5mZmWppaVFtba0kqbOzU42NjYGWDAAAYLyAZ9KSk5M1dOhQFRUVKT8/X9OmTVNtba2Ki4slSdOmTVNqauo5x4wYMaLr4mJjVVRUpIqKCrW2tsrr9aqgoEADBw4MtGwAAACj2SzLssJdRCgsWn9HuEsAos7MWyvCXcJ5y8jIkNvtDncZUYnehgZ9DY1I6GtmZmaX23niAAAAgIEIaQAAAAYipAEAABiIkAYAAGAgQhoAAICBCGkAAAAGMuaJA8EWSbcKiBSRsIw5EtFXAEBXmEkDAAAwECENAADAQIQ0AAAAAxHSAAAADBS1Cwce/t814S4BCKlXxvxLuEsAAIQQM2kAAAAGIqQBAAAYiJAGAABgIEIaAACAgQhpAAAABiKkAQAAGIiQBgAAYKCgh7RTp05p8+bNFzTG5XKpqKgo2KUAAABErJCEtC1btgT7sAAAAJeVoD9x4I033tCRI0dUXFys4cOHa9q0aVq7dq0+/fRTSdI999yjsWPHnjXO6/Vq5cqVqq+vV1ZWlmbNmqWEhAQdOHBAr776qjwej1JSUlRYWKi+ffsGu2wAAACjBD2kTZ06VY2NjSovL5ck7dq1Sw0NDSovL1dLS4sWLFigvLy8s4LWoUOH9Mgjjyg3N1cvvviiNm/erIKCAq1Zs0bz589XSkqKdu7cqXXr1qmwsPCsz3U6nXI6nZKksrKyYH8tAACASyrkz+7ct2+fbrnlFtntdqWmpmrYsGGqq6vTyJEjz9gvPT1dubm5kqRx48apsrJS+fn5amxs1OLFiyVJPp+v21k0h8Mhh8MR2i8DAABwiRjzgHWbzdbl66ysLJWWloajJAAAgLAJ+sKBXr16qa2tzf86Ly9PH330kXw+n1paWvTll18qJyfnrHFut1u1tbWSpB07dig3N1eZmZlqaWnxb+/s7FRjY2OwSwYAADBO0GfSkpOTNXToUBUVFSk/P1/Tpk1TbW2tiouLJUnTpk1TamrqWeMyMzO1adMmvfTSSxowYIAmT56s2NhYFRUVqaKiQq2trfJ6vSooKNDAgQODXTYAAIBRbJZlWeEuIhQKNi4JdwlASL0y5l/CXYLRMjIy5Ha7w11GVKK3oUFfQyMS+pqZmdnldp44AAAAYCBCGgAAgIEIaQAAAAYipAEAABiIkAYAAGAgY25mG2ysfAu+SFghE4noKwCgK8ykAQAAGIiQBgAAYCBCGgAAgIEIaQAAAAYipAEAABgoald3zty5LdwlAGdZNXZiuEsAAEQIZtIAAAAMREgDAAAwECENAADAQIQ0AAAAAxHSAAAADERIAwAAMNBFhbTKykrNnTtXK1euvKgPdblc2rFjh//19u3btXr16os6FgAAQDS6qPukbdmyRQsXLlR6evp57e/1ehUTE+N/ffToUe3YsUO33nrrxXw8AABA1LvgkLZq1So1NTXpZz/7mSZOnKgJEyboxRdflMvlUkJCgmbOnKnvfe97Wr9+vZqamuRyuZSenq45c+b4j/HGG2/o4MGDKi4u1vjx49W7d28dP35cpaWlampq0ujRozVt2jRJ0p49e7R+/Xp1dnaqf//+KiwsVGJiYvA6AAAAYKALDmkzZ87Unj17VFJSopSUFK1Zs0aDBw/W/Pnz9fnnn+v5559XeXm5JOngwYNavHix4uPjzzjG1KlT9c477+gnP/mJpG8udzY0NOiZZ55RbGys5syZozvvvFPx8fHasGGDFi5cqMTERP33f/+33n33Xd17771n1eV0OuV0OiVJZWVlF9wIAAAAkwT8WKh9+/apqKhIknTDDTfo5MmTam1tlSSNHDnyrIDWnRtuuEFJSUmSpKysLLndbp06dUoHDx7UwoULJUmdnZ267rrruhzvcDjkcDgC/ToAAABGCOmzOxMSEs5737i4OP/fdrtdXq9XlmXp+9///hmXSgEAAC4HAd+CIzc3V7/97W8lSdXV1UpOTvbPiHWnV69eamtr6/HY1113nWpqanTkyBFJksfj0aFDhwItGQAAwHgBz6Tdd999evHFFzVv3jwlJCTo0Ucf7XHMNddcI7vdfsbCga6kpKTo0Ucf1XPPPaeOjg5J0v3336/MzMxAywYAADCazbIsK9xFhMKUX70e7hKAs6waO/GsbRkZGXK73WGoJrrR19Cht6FBX0MjEvra3eQTTxwAAAAwECENAADAQIQ0AAAAAxHSAAAADERIAwAAMFBIb2YbTl2tokNgImGFDAAA0YKZNAAAAAMR0gAAAAxESAMAADAQIQ0AAMBAhDQAAAADRe3qzkc/OhDuEqLQ5dXTF27ODncJAIDLGDNpAAAABiKkAQAAGIiQBgAAYCBCGgAAgIEIaQAAAAYipAEAABgopCFtw4YN5/Wey+VSUVFRKEsBAACIKCENaRs3bryo9wAAAC53QbmZ7TPPPKNjx46po6NDBQUFcjgcev3119Xe3q7i4mINHDhQjz32mH//7753//33y+fz6Re/+IVqa2uVlpam+fPnKz4+XkeOHNHq1avV0tKihIQE/du//ZsGDBgQjLIBAACMFZSQVlhYqN69e6u9vV0LFizQmDFj9E//9E/atGmTysvLz9r/u++5XC4dPnxY//7v/65HHnlEy5cv165duzRu3DitWrVK//qv/6qrr75a+/fv1yuvvKKSkpJglA0AAGCsoIS0yspKVVVVSZLcbrcOHz6s5OTkCzpGv379NGjQIElSdna2jh49Ko/Ho5qaGi1fvty/X2dnZ5fjnU6nnE6nJKmsrOwivgUAAIA5Ag5p1dXV+uyzz7RkyRIlJCRo0aJF6ujouODjxMXF+f+22+1qb2+Xz+fTFVdc0eVs3Hc5HA45HI4L/lwAAAATBbxwoLW1VVdccYUSEhL0pz/9Sfv37/e/Fxsb2+3M17ne+4ukpCT169dPH330kSTJsiw1NDQEWjIAAIDxAp5Jy8/P169//WvNnTtXV199tYYMGeJ/7/bbb1dxcbEGDx58xsKB7753//33d3v8xx57TL/85S+1YcMGdXZ26pZbbvFfFgUAAIhWNsuyrHAXEQp3vbUj3CUgwr1wc/Yl+ZyMjAy53e5L8lmXE/oaOvQ2NOhraERCXzMzM7vczhMHAAAADERIAwAAMBAhDQAAwECENAAAAAMR0gAAAAxESAMAADBQUB4LZaJLdfuEy0kkLGMGACBaMJMGAABgIEIaAACAgQhpAAAABiKkAQAAGChqFw5s/N+YcJcQhY5Livy+3jXGG+4SAADoETNpAAAABiKkAQAAGIiQBgAAYCBCGgAAgIEIaQAAAAYipAEAABgo5CFt+vTpZ207deqUNm/e7H9dXV2tsrKyUJcCAAAQMcIyk3bq1Clt2bIlHB8NAAAQEXq8me0zzzyjY8eOqaOjQwUFBXI4HJK+mSErKCjQ7t27FR8fr+LiYqWmpsrlcum5556Tx+PRqFGjujzmG2+8oSNHjqi4uFjDhw/XiBEj5PF4tGzZMjU2Nio7O1uzZ8+WzWbTgQMH9Oqrr8rj8SglJUWFhYXq27dvcLsAAABgmB5DWmFhoXr37q329nYtWLBAY8aMUXJysk6fPq0hQ4bogQce0Nq1a7V161bdc889qqio0OTJkzV+/Hht2rSpy2NOnTpVjY2NKi8vl/TN5c76+notX75cffv21cKFC1VTU6OcnBytWbNG8+fPV0pKinbu3Kl169apsLAwuF0AAAAwTI8hrbKyUlVVVZIkt9utw4cPKzk5WbGxsbrxxhslSdnZ2dq7d68kqaamRkVFRZKkcePG6fXXXz+vQnJycpSeni5JGjRokFwul5KSktTY2KjFixdLknw+X7ezaE6nU06nU5L4fRsAAIh45wxp1dXV+uyzz7RkyRIlJCRo0aJF6ujokCTFxMTIZrNJkux2u7zeb5+H+JftFyIuLs7/t91ul8/nkyRlZWWptLS0x/EOh8N/KRYAACDSnXPhQGtrq6644golJCToT3/6k/bv39/jAYcOHarf/e53kqQdO3Z0uU+vXr3U1tbW47EyMzPV0tKi2tpaSVJnZ6caGxt7HAcAABDpzjmTlp+fr1//+teaO3eurr76ag0ZMqTHA86YMUPPPfec3n777W4XDiQnJ2vo0KEqKipSfn6+RowY0XVxsbEqKipSRUWFWltb5fV6VVBQoIEDB57HVwMAAIhcNsuyrHAXEQovbGwKdwkw1F1jvD3vdAllZGTI7XaHu4yoQ19Dh96GBn0NjUjoa2ZmZpfbeeIAAACAgQhpAAAABiKkAQAAGIiQBgAAYCBCGgAAgIEIaQAAAAbq8bFQkcq02yxEg0hYxgwAQLRgJg0AAMBAhDQAAAADEdIAAAAMREgDAAAwUNQuHGiqjNqvFjZN+rMu9SnTv6Dzkn4eAACmYCYNAADAQIQ0AAAAAxHSAAAADERIAwAAMBAhDQAAwECENAAAAAMFNaT99Kc/Pe9933vvPZ0+fdr/evr06cEsBQAAIKIFNaQtWbLkvPetrKw8I6QBAADgW0G9M+n06dP12muv6fjx41qxYoVaW1vl8/n08MMPKy8vz79fZWWlmpub9dRTTyklJUUlJSWSpHXr1mn37t2Kj49XcXGxUlNT1dLSolWrVunYsWOSpAcffFC5ubnBLBsAAMA4Ibl9/I4dO/SDH/xAd999t3w+31kzZgUFBXrvvfdUUlKilJQUSdLp06c1ZMgQPfDAA1q7dq22bt2qe+65RxUVFZoyZYpyc3PldrtVWlqqZ599NhRlAwAAGCMkIe3aa6/VSy+9pM7OTo0ePVqDBg3quZDYWN14442SpOzsbO3du1eS9Nlnn+ngwYP+/VpbW+XxeJSYmHjGeKfTKafTKUkqKysL0jcBAAAIj5CEtGHDhumpp57S7t279cILL2jKlCkaP378OcfExMTIZrNJkux2u7xeryTJsiyVlpYqPj7+nOMdDoccDkdwvgAAAECYheQWHEePHlVqaqocDoduv/121dfXn7VPYmKiPB5Pj8caPny4Nm3a5H/d0NAQzFIBAACMFJKZtOrqar3zzjuKiYlRYmKiZs2addY+DodDpaWlSktL8y8c6MqMGTO0evVqzZs3T16vV3l5eZo5c2YoygYAADCGzbIsK9xFhMIfXnGFuwQEQf+CznCXEHIZGRlyu93hLiPq0NfQobehQV9DIxL6mpmZ2eV2njgAAABgIEIaAACAgQhpAAAABiKkAQAAGIiQBgAAYCBCGgAAgIFCcp80E1wOt2641CJhGTMAANGCmTQAAAADEdIAAAAMREgDAAAwECENAADAQFG7cCBm/eFwlxB1juuwYi7B53jvu/oSfAoAAGZjJg0AAMBAhDQAAAADEdIAAAAMREgDAAAwECENAADAQIQ0AAAAA110SHO5XCoqKrroD37vvfd0+vRp/+vp06df9LEAAACiTdhm0iorK88IaQAAAPhWQDez9Xq9Wrlyperr65WVlaVZs2Zp//79qqys1Pz58yVJe/fu1ebNm1VcXOwfV1lZqebmZj311FNKSUlRSUmJJGndunXavXu34uPjVVxcrNTUVLW0tGjVqlU6duyYJOnBBx9Ubm5uIGUDAAAYL6CZtEOHDmny5Ml69tln1atXL23evFnXX3+9Dh06pJaWFknStm3bNHHixDPGFRQUKC0tTSUlJf6Advr0aQ0ZMkTl5eXKy8vT1q1bJUkVFRWaMmWKli5dqqKiIr388suBlAwAABARAppJS09P989qjRs3TpWVlfrbv/1bjRs3Th9++KEmTpyo2tpazZo1q+dCYmN14403SpKys7O1d+9eSdJnn32mgwcP+vdrbW2Vx+NRYmLiGeOdTqecTqckqaysLJCvBQAAEHYBhTSbzdbl6wkTJujpp59WfHy8br75ZsXE9PzEx5iYGP94u90ur9crSbIsS6WlpYqPjz/neIfDIYfDcTFfAwAAwDgBXe50u92qra2VJO3YscM/q5aWlqa+ffvqrbfe0oQJE7ocm5iYKI/H0+NnDB8+XJs2bfK/bmhoCKRkAACAiBBQSMvMzNSmTZs0d+5cnTx5UpMnT/a/d9tttykjI0NZWVldjnU4HCotLdVTTz11zs+YMWOG6urqNG/ePM2dO1dbtmwJpGQAAICIYLMsywrFgVevXq3Bgwdr0qRJoTh8j5pWfBKWz0XgvPddHe4SLqmMjAy53e5wlxF16Gvo0NvQoK+hEQl9zczM7HJ7SO6T9sQTT+iPf/yjbrvttlAcHgAAIOoFtHCgO08//XQoDgsAAHDZ4NmdAAAABiKkAQAAGIiQBgAAYCBCGgAAgIFCsnDABJfbbRwuhUhYxgwAQLRgJg0AAMBAhDQAAAADheyJAwAAALh4UTmT9pOf/CTcJUQl+hoa9DU06Gvo0NvQoK+hEcl9jcqQBgAAEOkIaQAAAAaKWbRo0aJwFxEK2dnZ4S4hKtHX0KCvoUFfQ4fehgZ9DY1I7SsLBwAAAAzE5U4AAAADRdUTBz799FNVVFTI5/Pp9ttv19///d+Hu6SI9eijjyoxMVF2u10xMTEqKyvTyZMn9eyzz+ro0aO68sorNXfuXPXu3TvcpRrvxRdf1O7du9WnTx8tW7ZMkrrtpWVZqqio0B/+8AclJCSosLAwYqfpQ62rvq5fv15bt25VSkqKJOmBBx7QiBEjJEkbN27UBx98ILvdrhkzZig/Pz9stZvM7XbrhRde0J///GfZbDY5HA4VFBRwzgaou75yzgauvb1dJSUl6uzslNfr1U033aT77rtPLpdLK1as0Ndff63s7GzNnj1bsbGx6ujo0PPPP68DBw4oOTlZc+bMUb9+/cL9NbpmRQmv12vNmjXLOnLkiNXR0WHNmzfPamxsDHdZEauwsNA6ceLEGdtee+01a+PGjZZlWdbGjRut1157LRylRZzq6mqrrq7Oevzxx/3buuvlJ598YpWWllo+n8+qqamxFixYEJaaI0FXfX3zzTett99++6x9GxsbrXnz5lnt7e1WU1OTNWvWLMvr9V7KciNGc3OzVVdXZ1mWZbW2tlqPPfaY1djYyDkboO76yjkbOJ/PZ7W1tVmWZVkdHR3WggULrJqaGmvZsmXWjh07LMuyrJdfftnavHmzZVmWtWnTJuvll1+2LMuyduzYYS1fvjw8hZ+HqLnc+dVXX+mqq65S//79FRsbq7Fjx6qqqsbOvMQAAASESURBVCrcZUWVqqoqjR8/XpI0fvx4+nuehg0bdtaMY3e9/P3vf69x48bJZrPpuuuu06lTp3T8+PFLXnMk6Kqv3amqqtLYsWMVFxenfv366aqrrtJXX30V4gojU9++ff0zYb169dKAAQPU3NzMORug7vraHc7Z82ez2ZSYmChJ8nq98nq9stlsqq6u1k033SRJmjBhwhnn7IQJEyRJN910kz7//HNZhv48P2oudzY3Nys9Pd3/Oj09Xfv37w9jRZGvtLRUkvTDH/5QDodDJ06cUN++fSVJqampOnHiRDjLi2jd9bK5uVkZGRn+/dLT09Xc3OzfFz3bvHmzPvzwQ2VnZ+uf//mf1bt3bzU3N2vIkCH+fdLS0s75P0h8w+Vyqb6+Xjk5OZyzQfTXfd23bx/nbBD4fD498cQTOnLkiO644w71799fSUlJiomJkXRm//46L8TExCgpKUlff/21/5KzSaImpCG4Fi9erLS0NJ04cUJLlixRZmbmGe/bbDbZbLYwVRdd6GXwTJ48Wffee68k6c0339R//ud/qrCwMMxVRSaPx6Nly5bpoYceUlJS0hnvcc5evO/2lXM2OOx2u8rLy3Xq1Cn9/Oc/16FDh8JdUlBEzeXOtLQ0HTt2zP/62LFjSktLC2NFke0vvevTp49GjRqlr776Sn369PFfxjh+/LiR/+qIFN31Mi0tTW63278f5/GFSU1Nld1ul91u1+233666ujpJZ//3obm5mb6eQ2dnp5YtW6bbbrtNY8aMkcQ5Gwxd9ZVzNriuuOIKXX/99aqtrVVra6u8Xq+kM/v31731er1qbW1VcnJy2Go+l6gJaddee60OHz4sl8ulzs5O7dy5UyNHjgx3WRHJ4/Gora3N//fevXt1zTXXaOTIkfrNb34jSfrNb36jUaNGhbPMiNZdL0eOHKkPP/xQlmWptrZWSUlJXDa6AH/9W6iPP/5YAwcOlPRNX3fu3KmOjg65XC4dPnxYOTk54SrTaJZl6Re/+IUGDBigKVOm+Ldzzgamu75yzgaupaVFp06dkvTNSs+9e/dqwIABuv7667Vr1y5J0vbt2/2Z4MYbb9T27dslSbt27dL1119v7MxwVN3Mdvfu3Xr11Vfl8/k0ceJE3X333eEuKSI1NTXp5z//uaRv/pVx66236u6779bXX3+tZ599Vm63m1twXIAVK1boiy++0Ndff60+ffrovvvu06hRo7rspWVZWr16tfbs2aP4+HgVFhbq2muvDfdXMFJXfa2urlZDQ4NsNpuuvPJKzZw50x8YNmzYoG3btslut+uhhx7S3/zN34T5G5hp3759+o//+A9dc801/v9xPfDAAxoyZAjnbAC66+vvfvc7ztkA/d///Z9eeOEF+Xw+WZalm2++Wffee6+ampq0YsUKnTx5UoMHD9bs2bMVFxen9vZ2Pf/886qvr1fv3r01Z84c9e/fP9xfo0tRFdIAAACiRdRc7gQAAIgmhDQAAAADEdIAAAAMREgDAAAwECENAADAQIQ0AAAAAxHSAAAADERIAwAAMND/B1Vk19cdqalyAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 720x360 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2SoSTud5RcbI"
      },
      "source": [
        "Observation:\r\n",
        "\r\n",
        "It is happy to see that puctuations and html tags no longer exists in the most common bigram tables.\r\n",
        "\r\n",
        "Although in the picture above, the most common bi-gram after data cleaning are phrases made of stopwords, our BERT model in the next section will handle that easily."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "c3xLF9soRI8u"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rkYgeIsOR8jX"
      },
      "source": [
        "# Step 4: Save the Result"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hpxIJZztR_ny"
      },
      "source": [
        "# Write cleaned data into csv files for further use\r\n",
        "df_train.to_csv('train_cleaned.csv')\r\n",
        "df_test.to_csv('test_cleaned.csv')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "--Dw0sqVSVWt"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}