category_03_climate_environment/01_NLP_with_Disaster_Tweets/Clone_NLP_Getting_Started_Tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "shaped-foundation",
   "metadata": {},
   "source": [
    "This Notebook is Clone of 'NLP Getting Started Tutorial' kernel in Kaggle for nlp modeling practice."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "exterior-control",
   "metadata": {},
   "source": [
    "### NLP Tutorial\n",
    "\n",
    "NLP - or Natural Language Processing - is shorthand for a wide array of techniques designed to help machines learn from text. Natural Language Processing powers everything from chatbots to search engines, and is used in diverse tasks like sentiment analysis and machine translation.\n",
    "\n",
    "In this tutorial we'll look this competition's dataset, use a simple technique to process it, build a machine larning model, and submit predictions for a score!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "enhanced-conference",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np # linear algebra\n",
    "import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n",
    "from sklearn import feature_extraction, linear_model, model_selection, preprocessing"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "unsigned-defensive",
   "metadata": {},
   "source": [
    "### Step 1. Data Load and EDA"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "secret-piano",
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df = pd.read_csv('./data/train.csv')\n",
    "test_df = pd.read_csv('./data/test.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fifty-strap",
   "metadata": {},
   "source": [
    "#### A quick look at our data\n",
    "\n",
    "Let's look at our data... first, an example of what is NOT a disaster tweet."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "present-handle",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'I love fruits'"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df[train_df['target']==0]['text'].values[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "hourly-filling",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>keyword</th>\n",
       "      <th>location</th>\n",
       "      <th>text</th>\n",
       "      <th>target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Our Deeds are the Reason of this #earthquake M...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Forest fire near La Ronge Sask. Canada</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>5</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>All residents asked to 'shelter in place' are ...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>6</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>13,000 people receive #wildfires evacuation or...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>7</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Just got sent this photo from Ruby #Alaska as ...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   id keyword location                                               text  \\\n",
       "0   1     NaN      NaN  Our Deeds are the Reason of this #earthquake M...   \n",
       "1   4     NaN      NaN             Forest fire near La Ronge Sask. Canada   \n",
       "2   5     NaN      NaN  All residents asked to 'shelter in place' are ...   \n",
       "3   6     NaN      NaN  13,000 people receive #wildfires evacuation or...   \n",
       "4   7     NaN      NaN  Just got sent this photo from Ruby #Alaska as ...   \n",
       "\n",
       "   target  \n",
       "0       1  \n",
       "1       1  \n",
       "2       1  \n",
       "3       1  \n",
       "4       1  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Let's look this code more deeply\n",
    "train_df.head()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "driving-suffering",
   "metadata": {},
   "source": [
    "text column contains disaster tweet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "vocal-wheel",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>keyword</th>\n",
       "      <th>location</th>\n",
       "      <th>text</th>\n",
       "      <th>target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>23</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>What's up man?</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>24</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>I love fruits</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>25</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Summer is lovely</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>26</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>My car is so fast</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>28</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>What a goooooooaaaaaal!!!!!!</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    id keyword location                          text  target\n",
       "15  23     NaN      NaN                What's up man?       0\n",
       "16  24     NaN      NaN                 I love fruits       0\n",
       "17  25     NaN      NaN              Summer is lovely       0\n",
       "18  26     NaN      NaN             My car is so fast       0\n",
       "19  28     NaN      NaN  What a goooooooaaaaaal!!!!!!       0"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df[train_df['target']==0].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "headed-damages",
   "metadata": {},
   "source": [
    "this text isn't related to disaster tweet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "diverse-necklace",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([\"What's up man?\", 'I love fruits', 'Summer is lovely', ...,\n",
       "       'These boxes are ready to explode! Exploding Kittens finally arrived! gameofkittens #explodingkittens\\x89Û_ https://t.co/TFGrAyuDC5',\n",
       "       'Sirens everywhere!',\n",
       "       'I just heard a really loud bang and everyone is asleep great'],\n",
       "      dtype=object)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df[train_df['target']==0]['text'].values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "spoken-palace",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"What's up man?\""
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df[train_df['target']==0]['text'].values[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "metallic-adoption",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'I love fruits'"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df[train_df['target']==0]['text'].values[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "knowing-gabriel",
   "metadata": {},
   "source": [
    "OK. go to next step"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "stretch-virus",
   "metadata": {},
   "source": [
    "------------\n",
    "And one that is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "unlikely-solid",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Forest fire near La Ronge Sask. Canada'"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df[train_df['target']==1]['text'].values[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "focal-turkey",
   "metadata": {},
   "source": [
    "**Building vectors**\n",
    "\n",
    "The theory behind the model we'll build in this notebook is pretty simple : the words contained in each tweet are a good indicator of whether they're about a real disaster or not (this is not entirely correct, but it's a great place to start).\n",
    "\n",
    "We'll use scikit-learn's CountVectorizer to count the words in each tweet and turn them into data our machine learning model can process.\n",
    "\n",
    "NOte: a vector is, in this context, a set of numbers that a machine learning model can work with. We'll look at one in just a second."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "fuzzy-gateway",
   "metadata": {},
   "outputs": [],
   "source": [
    "count_vectorizer = feature_extraction.text.CountVectorizer()\n",
    "\n",
    "## let's get counts for the first 5 tweets in the data\n",
    "example_train_vectors = count_vectorizer.fit_transform(train_df['text'][0:5])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "jewish-traveler",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1, 54)\n",
      "[[0 0 0 1 1 1 0 0 0 0 0 0 1 1 0 0 0 0 1 0 0 0 0 0 0 1 0 0 0 1 0 0 0 0 1 0\n",
      "  0 0 0 1 0 0 0 0 0 0 0 0 0 1 1 0 1 0]]\n"
     ]
    }
   ],
   "source": [
    "## we use .todense() here because these vectors are 'sparse' (only non-zero elements are kept to save space)\n",
    "print(example_train_vectors[0].todense().shape)\n",
    "print(example_train_vectors[0].todense())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "tracked-macedonia",
   "metadata": {},
   "source": [
    "----------\n",
    "let's look more code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "vital-associate",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<1x54 sparse matrix of type '<class 'numpy.int64'>'\n",
       "\twith 13 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "example_train_vectors[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "revolutionary-south",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<5x54 sparse matrix of type '<class 'numpy.int64'>'\n",
       "\twith 61 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "example_train_vectors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "judicial-monitoring",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>keyword</th>\n",
       "      <th>location</th>\n",
       "      <th>text</th>\n",
       "      <th>target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Our Deeds are the Reason of this #earthquake M...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Forest fire near La Ronge Sask. Canada</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>5</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>All residents asked to 'shelter in place' are ...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>6</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>13,000 people receive #wildfires evacuation or...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>7</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Just got sent this photo from Ruby #Alaska as ...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7608</th>\n",
       "      <td>10869</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Two giant cranes holding a bridge collapse int...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7609</th>\n",
       "      <td>10870</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>@aria_ahrary @TheTawniest The out of control w...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7610</th>\n",
       "      <td>10871</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>M1.94 [01:04 UTC]?5km S of Volcano Hawaii. htt...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7611</th>\n",
       "      <td>10872</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Police investigating after an e-bike collided ...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7612</th>\n",
       "      <td>10873</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>The Latest: More Homes Razed by Northern Calif...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>7613 rows × 5 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         id keyword location  \\\n",
       "0         1     NaN      NaN   \n",
       "1         4     NaN      NaN   \n",
       "2         5     NaN      NaN   \n",
       "3         6     NaN      NaN   \n",
       "4         7     NaN      NaN   \n",
       "...     ...     ...      ...   \n",
       "7608  10869     NaN      NaN   \n",
       "7609  10870     NaN      NaN   \n",
       "7610  10871     NaN      NaN   \n",
       "7611  10872     NaN      NaN   \n",
       "7612  10873     NaN      NaN   \n",
       "\n",
       "                                                   text  target  \n",
       "0     Our Deeds are the Reason of this #earthquake M...       1  \n",
       "1                Forest fire near La Ronge Sask. Canada       1  \n",
       "2     All residents asked to 'shelter in place' are ...       1  \n",
       "3     13,000 people receive #wildfires evacuation or...       1  \n",
       "4     Just got sent this photo from Ruby #Alaska as ...       1  \n",
       "...                                                 ...     ...  \n",
       "7608  Two giant cranes holding a bridge collapse int...       1  \n",
       "7609  @aria_ahrary @TheTawniest The out of control w...       1  \n",
       "7610  M1.94 [01:04 UTC]?5km S of Volcano Hawaii. htt...       1  \n",
       "7611  Police investigating after an e-bike collided ...       1  \n",
       "7612  The Latest: More Homes Razed by Northern Calif...       1  \n",
       "\n",
       "[7613 rows x 5 columns]"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "normal-probability",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>keyword</th>\n",
       "      <th>location</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Just happened a terrible car crash</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Heard about #earthquake is different cities, s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>there is a forest fire at spot pond, geese are...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>9</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Apocalypse lighting. #Spokane #wildfires</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>11</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Typhoon Soudelor kills 28 in China and Taiwan</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3258</th>\n",
       "      <td>10861</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>EARTHQUAKE SAFETY LOS ANGELES ÛÒ SAFETY FASTE...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3259</th>\n",
       "      <td>10865</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Storm in RI worse than last hurricane. My city...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3260</th>\n",
       "      <td>10868</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Green Line derailment in Chicago http://t.co/U...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3261</th>\n",
       "      <td>10874</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>MEG issues Hazardous Weather Outlook (HWO) htt...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3262</th>\n",
       "      <td>10875</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>#CityofCalgary has activated its Municipal Eme...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3263 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         id keyword location  \\\n",
       "0         0     NaN      NaN   \n",
       "1         2     NaN      NaN   \n",
       "2         3     NaN      NaN   \n",
       "3         9     NaN      NaN   \n",
       "4        11     NaN      NaN   \n",
       "...     ...     ...      ...   \n",
       "3258  10861     NaN      NaN   \n",
       "3259  10865     NaN      NaN   \n",
       "3260  10868     NaN      NaN   \n",
       "3261  10874     NaN      NaN   \n",
       "3262  10875     NaN      NaN   \n",
       "\n",
       "                                                   text  \n",
       "0                    Just happened a terrible car crash  \n",
       "1     Heard about #earthquake is different cities, s...  \n",
       "2     there is a forest fire at spot pond, geese are...  \n",
       "3              Apocalypse lighting. #Spokane #wildfires  \n",
       "4         Typhoon Soudelor kills 28 in China and Taiwan  \n",
       "...                                                 ...  \n",
       "3258  EARTHQUAKE SAFETY LOS ANGELES ÛÒ SAFETY FASTE...  \n",
       "3259  Storm in RI worse than last hurricane. My city...  \n",
       "3260  Green Line derailment in Chicago http://t.co/U...  \n",
       "3261  MEG issues Hazardous Weather Outlook (HWO) htt...  \n",
       "3262  #CityofCalgary has activated its Municipal Eme...  \n",
       "\n",
       "[3263 rows x 4 columns]"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "curious-halifax",
   "metadata": {},
   "source": [
    "--------\n",
    "Ok. Next go!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "processed-rotation",
   "metadata": {},
   "source": [
    "The obove tells us that:\n",
    "\n",
    "1. There are 54 unique words(or 'tokens') in the first five tweets.\n",
    "2. The first tweet contains only some of those unique tokens - all of the non-zero counts above are the tokens that DO exist in the first tweet.\n",
    "\n",
    "Now let's create vectors for all of our tweets."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "greenhouse-rwanda",
   "metadata": {},
   "source": [
    "### Step 2. Text -> Vectorization(One hot Encoding)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "concrete-thailand",
   "metadata": {},
   "outputs": [],
   "source": [
    "train_vectors = count_vectorizer.fit_transform(train_df['text'])\n",
    "\n",
    "## note that we're NOT using .fit_transform() here. Using Just .transform() makes sure\n",
    "# that the tokens in the train vectors are the only ones mapped to the test vectores - \n",
    "# i. e. that the train and test vectors use same set of tokens.\n",
    "test_vectors = count_vectorizer.transform(test_df['text'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "swiss-radiation",
   "metadata": {},
   "source": [
    "**Our model**\n",
    "\n",
    "As we mentioned above, we think the words contained in each tweet are a good indicator of whether they're about a real disaster or not. The presence of particular word(or set of words) in a tweet might link directly to whether or not that tweet is real.\n",
    "\n",
    "What we're assuming here is a *linear* connection. So let's build a linear model and see!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "advance-incident",
   "metadata": {},
   "outputs": [],
   "source": [
    "## Our vectors are really big, so we want to push our model's weights\n",
    "## toward 0 without completely discounting different words - rigde regression\n",
    "## is a good way to do this.\n",
    "clf = linear_model.RidgeClassifier()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "unavailable-philip",
   "metadata": {},
   "source": [
    "Let's test our model and see how well it does on the training data. For this we'll use cross-validation, where we train on a portion of the known data, then validate it with te rest. If we do this several times (with different portions) we can get a good idea for how a particular model or method performs.\n",
    "\n",
    "The metric for this competition if F1, so let's use that here."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "removable-native",
   "metadata": {},
   "source": [
    "### Step 3. Cross Validation Check"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "behind-glass",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.59421842, 0.5642787 , 0.64082434])"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scores = model_selection.cross_val_score(clf, train_vectors, train_df['target'], cv=3, scoring='f1')\n",
    "scores"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "successful-respect",
   "metadata": {},
   "source": [
    "The above scores aren't terrible! It looks like our assumption will score roughly 0.65 on the leaderboard. There are lots of ways to potentially improve on this (TFIDF, LSA, LSTM / RNNs, the list is long!) - give any of them a shot!\n",
    "\n",
    "In the meantime, let's do predictions on our training set and build a submission for the competition."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "square-ethernet",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0 0 0 ... 0 0 0]]\n",
      "(1, 21637)\n"
     ]
    }
   ],
   "source": [
    "# check some data\n",
    "print(train_vectors[0].todense())\n",
    "print(train_vectors[0].todense().shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "parliamentary-williams",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<7613x21637 sparse matrix of type '<class 'numpy.int64'>'\n",
       "\twith 111497 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_vectors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "white-annotation",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1 1 1 ... 1 1 1]\n",
      "(7613,)\n"
     ]
    }
   ],
   "source": [
    "print(train_df['target'].values)\n",
    "print(train_df['target'].values.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "divine-stupid",
   "metadata": {},
   "source": [
    "same length -> 7613"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "smaller-dallas",
   "metadata": {},
   "source": [
    "### Step 4. Modeling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "vocational-forest",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RidgeClassifier()"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf.fit(train_vectors, train_df['target'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "stock-rebel",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 1, 1, ..., 1, 1, 0])"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf.predict(test_vectors)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "catholic-graph",
   "metadata": {},
   "source": [
    "### Step 5. Save / submission(kaggle)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "appointed-functionality",
   "metadata": {},
   "outputs": [],
   "source": [
    "sample_submission = pd.read_csv('./data/sample_submission.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "strange-manitoba",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>9</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>11</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   id  target\n",
       "0   0       0\n",
       "1   2       0\n",
       "2   3       0\n",
       "3   9       0\n",
       "4  11       0"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sample_submission.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "photographic-california",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0])"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sample_submission['target'].unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "saving-vertex",
   "metadata": {},
   "outputs": [],
   "source": [
    "sample_submission['target'] = clf.predict(test_vectors)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "veterinary-shakespeare",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>9</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>11</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   id  target\n",
       "0   0       0\n",
       "1   2       1\n",
       "2   3       1\n",
       "3   9       0\n",
       "4  11       1"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sample_submission.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "northern-tooth",
   "metadata": {},
   "outputs": [],
   "source": [
    "sample_submission.to_csv('submission.csv', index=False)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}