src/twitter-search_v3_snscrape.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "<h1 style=\"font-family:sans-serif; text-align:center\"> \n",
    "<!--     <span style='color: pink'> Twitter analysis of </span> -->\n",
    "    <span style='color: white; font-size:100%; text-shadow: 0px 0px 15px black'> Twitter analysis of </span>\n",
    "<!--     <span style='color:#00acee'> Twitter analysis of </span> -->\n",
    "<!--     <span style=\"-webkit-text-stroke\"> Twitter analysis of</span> -->\n",
    "<!--     <span class=\"hr3\" style='color:#e40843; letter-spacing: 4px; font-size:105%'> Canada</span> -->\n",
    "    <span class=\"hr3\" style='color:#e40843; font-size:120%; text-shadow: 0px 0px 30px pink'>Canada </span> <span class=\"hr3\" style='color:gray; font-size:100%; text-shadow: 0px 0px 30px pink'>response to Covid-19</span><br>\n",
    "</h1>\n",
    "\n",
    "#### — _Using snscrape_ —\n",
    "\n",
    "### ✅ This jupyter notebook works well!\n",
    "\n",
    "The aim of this notebook is to retrieve the tweets from March 1st until April 30th, to analyze the difference in sentiment analysis of tweets from people before and after Trudeau's [announcement of government policies facing impact of Covid-19](https://www.youtube.com/watch?v=1o-tV0A87l8&feature=youtu.be) to support small businesses and their employees.\n",
    "\n",
    "\n",
    "The **snscrape** allowed us to find old tweets (as opposed to the free version of the API from twitter, and the GetOldTweets3 library that is non-currently working). The way to download tweets with this library is well explained in Martin Beck's article _[How to Scrape Tweets With snscrape](https://medium.com/better-programming/how-to-scrape-tweets-with-snscrape-90124ed006af)_ at Medium.\n",
    "\n",
    "_Authors: Leo Cuspinera ([cuspime](https://github.com/cuspime)) and Victor Cuspinera ([vcuspinera](https://github.com/vcuspinera))_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# # Install development version of snscrape\n",
    "# !pip3 install git+https://github.com/JustAnotherArchivist/snscrape.git \n",
    "\n",
    "# General libraries\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import os\n",
    "import time\n",
    "from datetime import datetime, timedelta, date\n",
    "from pytz import timezone\n",
    "import json\n",
    "\n",
    "# Preprocess libraries\n",
    "import re\n",
    "import spacy\n",
    "import string\n",
    "import en_core_web_sm\n",
    "nlp = en_core_web_sm.load()\n",
    "\n",
    "# Ignore warnings\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "START = time.time()\n",
    "\n",
    "# dates\n",
    "today = datetime.now()\n",
    "init = date.fromisoformat('2020-02-01')\n",
    "\n",
    "my_dates = list()\n",
    "for d in range(0, 90, 1):\n",
    "    aux = init + timedelta(days=d)\n",
    "    my_dates.append(aux)\n",
    "\n",
    "# twitter accounts\n",
    "accounts = (\"@JustinTrudeau\", \"@CanadianPM\",\"@Canada\", \"@GovCanHealth\")\n",
    "\n",
    "# max number of results\n",
    "max_results = 100_000\n",
    "\n",
    "#folder to save information\n",
    "my_folder = \"../tweets/\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get and save tweets as `json` files\n",
    "⚠️ **Caution:** Just run this code chunk once, it takes so much time (more than couple hours) to download all the tweets. Additionally, this step downloads 244 JSON files, that in total weight 10.27 GB"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "# Retrieving tweets with `snscrape`, by using OS library to call CLI commands in Python.\n",
    "for ac in accounts:\n",
    "    for dt in my_dates:\n",
    "        next_day = dt + timedelta(days=1)\n",
    "        os.system(\"snscrape --jsonl --max-results \" + str(max_results) + \" --since \" + \n",
    "                  dt.strftime(\"%Y-%m-%d\") + \" twitter-search '\" + ac + \" until:\" + \n",
    "                  next_day.strftime(\"%Y-%m-%d\") + \"' > \" + my_folder + ac + \n",
    "                  \"_\" + dt.strftime(\"%Y-%m-%d\") + \".json\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bring and merge tweets by account"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Start with @JustinTrudeau files:\n",
      "   > date: 2020-02-01 time 0.3718\n",
      "   > date: 2020-02-02 time 0.3239\n",
      "   > date: 2020-02-03 time 0.2696\n",
      "   > date: 2020-02-04 time 0.199\n",
      "   > date: 2020-02-05 time 0.4545\n",
      "   > date: 2020-02-06 time 0.3254\n",
      "   > date: 2020-02-07 time 0.4981\n",
      "   > date: 2020-02-08 time 0.4872\n",
      "   > date: 2020-02-09 time 0.7022\n",
      "   > date: 2020-02-10 time 0.6538\n",
      "   > date: 2020-02-11 time 0.3519\n",
      "   > date: 2020-02-12 time 0.6593\n",
      "   > date: 2020-02-13 time 0.7202\n",
      "   > date: 2020-02-14 time 0.8659\n",
      "   > date: 2020-02-15 time 0.7752\n",
      "   > date: 2020-02-16 time 0.5921\n",
      "   > date: 2020-02-17 time 0.6629\n",
      "   > date: 2020-02-18 time 1.1729\n",
      "   > date: 2020-02-19 time 1.2126\n",
      "   > date: 2020-02-20 time 0.8324\n",
      "   > date: 2020-02-21 time 0.5746\n",
      "   > date: 2020-02-22 time 0.4616\n",
      "   > date: 2020-02-23 time 0.6444\n",
      "   > date: 2020-02-24 time 0.7931\n",
      "   > date: 2020-02-25 time 0.7965\n",
      "   > date: 2020-02-26 time 0.6174\n",
      "   > date: 2020-02-27 time 1.1044\n",
      "   > date: 2020-02-28 time 0.5402\n",
      "   > date: 2020-02-29 time 0.4527\n",
      "   > date: 2020-03-01 time 0.4701\n",
      "   > date: 2020-03-02 time 1.0176\n",
      "   > date: 2020-03-03 time 0.5478\n",
      "   > date: 2020-03-04 time 0.5576\n",
      "   > date: 2020-03-05 time 0.5814\n",
      "   > date: 2020-03-06 time 0.5594\n",
      "   > date: 2020-03-07 time 0.9579\n",
      "   > date: 2020-03-08 time 0.4936\n",
      "   > date: 2020-03-09 time 0.5633\n",
      "   > date: 2020-03-10 time 0.5492\n",
      "   > date: 2020-03-11 time 1.1731\n",
      "   > date: 2020-03-12 time 0.97\n",
      "   > date: 2020-03-13 time 2.006\n",
      "   > date: 2020-03-14 time 1.3732\n",
      "   > date: 2020-03-15 time 1.8516\n",
      "   > date: 2020-03-16 time 2.1375\n",
      "   > date: 2020-03-17 time 1.2273\n",
      "   > date: 2020-03-18 time 1.9268\n",
      "   > date: 2020-03-19 time 1.0499\n",
      "   > date: 2020-03-20 time 1.3493\n",
      "   > date: 2020-03-21 time 1.7991\n",
      "   > date: 2020-03-22 time 1.2746\n",
      "   > date: 2020-03-23 time 2.2531\n",
      "   > date: 2020-03-24 time 1.5546\n",
      "   > date: 2020-03-25 time 2.7206\n",
      "   > date: 2020-03-26 time 1.2066\n",
      "   > date: 2020-03-27 time 1.3289\n",
      "   > date: 2020-03-28 time 1.3406\n",
      "   > date: 2020-03-29 time 1.9571\n",
      "   > date: 2020-03-30 time 1.3479\n",
      "   > date: 2020-03-31 time 1.4208\n",
      "   > date: 2020-04-01 time 1.39\n",
      "   > date: 2020-04-02 time 2.2536\n",
      "   > date: 2020-04-03 time 1.2806\n",
      "   > date: 2020-04-04 time 1.259\n",
      "   > date: 2020-04-05 time 1.2234\n",
      "   > date: 2020-04-06 time 2.4976\n",
      "   > date: 2020-04-07 time 1.3935\n",
      "   > date: 2020-04-08 time 1.2146\n",
      "   > date: 2020-04-09 time 1.2144\n",
      "   > date: 2020-04-10 time 1.282\n",
      "   > date: 2020-04-11 time 1.0888\n",
      "   > date: 2020-04-12 time 2.5508\n",
      "   > date: 2020-04-13 time 1.2499\n",
      "   > date: 2020-04-14 time 1.3062\n",
      "   > date: 2020-04-15 time 1.3467\n",
      "   > date: 2020-04-16 time 1.3366\n",
      "   > date: 2020-04-17 time 2.5261\n",
      "   > date: 2020-04-18 time 1.326\n",
      "   > date: 2020-04-19 time 1.4103\n",
      "   > date: 2020-04-20 time 1.5366\n",
      "   > date: 2020-04-21 time 1.2535\n",
      "   > date: 2020-04-22 time 1.4579\n",
      "   > date: 2020-04-23 time 1.3417\n",
      "   > date: 2020-04-24 time 3.0792\n",
      "   > date: 2020-04-25 time 1.0097\n",
      "   > date: 2020-04-26 time 1.2295\n",
      "   > date: 2020-04-27 time 1.2802\n",
      "   > date: 2020-04-28 time 1.0\n",
      "   > date: 2020-04-29 time 1.3876\n",
      "   > date: 2020-04-30 time 1.3215\n",
      "@JustinTrudeau DB ready, time 102.749\n",
      "@JustinTrudeau tweets saved as .JSON file, time 102.7494\n",
      "- - - - - o - - - - -\n",
      "\n",
      "Start with @CanadianPM files:\n",
      "   > date: 2020-02-01 time 0.0799\n",
      "   > date: 2020-02-02 time 0.0298\n",
      "   > date: 2020-02-03 time 0.0274\n",
      "   > date: 2020-02-04 time 0.0237\n",
      "   > date: 2020-02-05 time 0.0247\n",
      "   > date: 2020-02-06 time 0.0311\n",
      "   > date: 2020-02-07 time 0.0328\n",
      "   > date: 2020-02-08 time 0.0457\n",
      "   > date: 2020-02-09 time 0.0436\n",
      "   > date: 2020-02-10 time 0.0322\n",
      "   > date: 2020-02-11 time 0.0321\n",
      "   > date: 2020-02-12 time 0.0313\n",
      "   > date: 2020-02-13 time 0.0347\n",
      "   > date: 2020-02-14 time 0.0454\n",
      "   > date: 2020-02-15 time 0.0387\n",
      "   > date: 2020-02-16 time 0.0336\n",
      "   > date: 2020-02-17 time 0.0548\n",
      "   > date: 2020-02-18 time 0.0573\n",
      "   > date: 2020-02-19 time 0.054\n",
      "   > date: 2020-02-20 time 0.0403\n",
      "   > date: 2020-02-21 time 0.0429\n",
      "   > date: 2020-02-22 time 0.0472\n",
      "   > date: 2020-02-23 time 0.0493\n",
      "   > date: 2020-02-24 time 0.0405\n",
      "   > date: 2020-02-25 time 0.0483\n",
      "   > date: 2020-02-26 time 0.0411\n",
      "   > date: 2020-02-27 time 0.039\n",
      "   > date: 2020-02-28 time 0.0346\n",
      "   > date: 2020-02-29 time 0.0468\n",
      "   > date: 2020-03-01 time 0.0388\n",
      "   > date: 2020-03-02 time 0.0419\n",
      "   > date: 2020-03-03 time 0.0362\n",
      "   > date: 2020-03-04 time 0.0451\n",
      "   > date: 2020-03-05 time 0.0573\n",
      "   > date: 2020-03-06 time 0.0375\n",
      "   > date: 2020-03-07 time 0.0428\n",
      "   > date: 2020-03-08 time 0.0326\n",
      "   > date: 2020-03-09 time 0.0423\n",
      "   > date: 2020-03-10 time 0.0425\n",
      "   > date: 2020-03-11 time 0.0549\n",
      "   > date: 2020-03-12 time 0.0595\n",
      "   > date: 2020-03-13 time 0.0528\n",
      "   > date: 2020-03-14 time 0.0658\n",
      "   > date: 2020-03-15 time 0.0763\n",
      "   > date: 2020-03-16 time 0.0782\n",
      "   > date: 2020-03-17 time 0.1034\n",
      "   > date: 2020-03-18 time 0.0934\n",
      "   > date: 2020-03-19 time 0.0763\n",
      "   > date: 2020-03-20 time 0.0695\n",
      "   > date: 2020-03-21 time 0.0679\n",
      "   > date: 2020-03-22 time 0.0876\n",
      "   > date: 2020-03-23 time 0.0841\n",
      "   > date: 2020-03-24 time 0.0806\n",
      "   > date: 2020-03-25 time 1.93\n",
      "   > date: 2020-03-26 time 0.1155\n",
      "   > date: 2020-03-27 time 0.0831\n",
      "   > date: 2020-03-28 time 0.0884\n",
      "   > date: 2020-03-29 time 0.1001\n",
      "   > date: 2020-03-30 time 0.0766\n",
      "   > date: 2020-03-31 time 0.0851\n",
      "   > date: 2020-04-01 time 0.0742\n",
      "   > date: 2020-04-02 time 0.079\n",
      "   > date: 2020-04-03 time 0.0812\n",
      "   > date: 2020-04-04 time 0.0813\n",
      "   > date: 2020-04-05 time 0.1029\n",
      "   > date: 2020-04-06 time 0.0946\n",
      "   > date: 2020-04-07 time 0.0814\n",
      "   > date: 2020-04-08 time 0.0798\n",
      "   > date: 2020-04-09 time 0.0796\n",
      "   > date: 2020-04-10 time 0.0795\n",
      "   > date: 2020-04-11 time 0.0729\n",
      "   > date: 2020-04-12 time 0.0699\n",
      "   > date: 2020-04-13 time 0.0778\n",
      "   > date: 2020-04-14 time 0.0789\n",
      "   > date: 2020-04-15 time 0.0938\n",
      "   > date: 2020-04-16 time 0.1052\n",
      "   > date: 2020-04-17 time 0.078\n",
      "   > date: 2020-04-18 time 0.0828\n",
      "   > date: 2020-04-19 time 0.0862\n",
      "   > date: 2020-04-20 time 0.0721\n",
      "   > date: 2020-04-21 time 0.1172\n",
      "   > date: 2020-04-22 time 0.1155\n",
      "   > date: 2020-04-23 time 0.0865\n",
      "   > date: 2020-04-24 time 0.0978\n",
      "   > date: 2020-04-25 time 0.089\n",
      "   > date: 2020-04-26 time 0.0861\n",
      "   > date: 2020-04-27 time 0.0947\n",
      "   > date: 2020-04-28 time 0.0929\n",
      "   > date: 2020-04-29 time 0.0795\n",
      "   > date: 2020-04-30 time 0.0757\n",
      "@CanadianPM DB ready, time 7.7284\n",
      "@CanadianPM tweets saved as .JSON file, time 7.7285\n",
      "- - - - - o - - - - -\n",
      "\n",
      "Start with @Canada files:\n",
      "   > date: 2020-02-01 time 0.0309\n",
      "   > date: 2020-02-02 time 0.0269\n",
      "   > date: 2020-02-03 time 0.026\n",
      "   > date: 2020-02-04 time 0.029\n",
      "   > date: 2020-02-05 time 0.0249\n",
      "   > date: 2020-02-06 time 0.0334\n",
      "   > date: 2020-02-07 time 0.0294\n",
      "   > date: 2020-02-08 time 0.0338\n",
      "   > date: 2020-02-09 time 0.0326\n",
      "   > date: 2020-02-10 time 0.0329\n",
      "   > date: 2020-02-11 time 0.029\n",
      "   > date: 2020-02-12 time 0.0303\n",
      "   > date: 2020-02-13 time 0.0266\n",
      "   > date: 2020-02-14 time 0.0308\n",
      "   > date: 2020-02-15 time 0.0323\n",
      "   > date: 2020-02-16 time 0.0295\n",
      "   > date: 2020-02-17 time 0.0288\n",
      "   > date: 2020-02-18 time 0.0255\n",
      "   > date: 2020-02-19 time 0.0386\n",
      "   > date: 2020-02-20 time 0.0318\n",
      "   > date: 2020-02-21 time 0.0307\n",
      "   > date: 2020-02-22 time 0.0244\n",
      "   > date: 2020-02-23 time 0.0237\n",
      "   > date: 2020-02-24 time 0.0251\n",
      "   > date: 2020-02-25 time 0.0254\n",
      "   > date: 2020-02-26 time 0.0282\n",
      "   > date: 2020-02-27 time 0.0287\n",
      "   > date: 2020-02-28 time 0.0369\n",
      "   > date: 2020-02-29 time 0.0283\n",
      "   > date: 2020-03-01 time 0.0255\n",
      "   > date: 2020-03-02 time 0.0301\n",
      "   > date: 2020-03-03 time 0.026\n",
      "   > date: 2020-03-04 time 0.0251\n",
      "   > date: 2020-03-05 time 0.0289\n",
      "   > date: 2020-03-06 time 0.0277\n",
      "   > date: 2020-03-07 time 0.0279\n",
      "   > date: 2020-03-08 time 0.0281\n",
      "   > date: 2020-03-09 time 0.0269\n",
      "   > date: 2020-03-10 time 0.0268\n",
      "   > date: 2020-03-11 time 0.0314\n",
      "   > date: 2020-03-12 time 0.0242\n",
      "   > date: 2020-03-13 time 0.035\n",
      "   > date: 2020-03-14 time 0.0347\n",
      "   > date: 2020-03-15 time 0.0431\n",
      "   > date: 2020-03-16 time 0.0465\n",
      "   > date: 2020-03-17 time 0.0378\n",
      "   > date: 2020-03-18 time 0.0351\n",
      "   > date: 2020-03-19 time 0.0347\n",
      "   > date: 2020-03-20 time 0.0366\n",
      "   > date: 2020-03-21 time 0.039\n",
      "   > date: 2020-03-22 time 0.0337\n",
      "   > date: 2020-03-23 time 0.0497\n",
      "   > date: 2020-03-24 time 0.0355\n",
      "   > date: 2020-03-25 time 0.0401\n",
      "   > date: 2020-03-26 time 0.0416\n",
      "   > date: 2020-03-27 time 0.0351\n",
      "   > date: 2020-03-28 time 0.0352\n",
      "   > date: 2020-03-29 time 0.0353\n",
      "   > date: 2020-03-30 time 0.0325\n",
      "   > date: 2020-03-31 time 0.0398\n",
      "   > date: 2020-04-01 time 0.0351\n",
      "   > date: 2020-04-02 time 0.0357\n",
      "   > date: 2020-04-03 time 0.0372\n",
      "   > date: 2020-04-04 time 0.0436\n",
      "   > date: 2020-04-05 time 0.0398\n",
      "   > date: 2020-04-06 time 0.039\n",
      "   > date: 2020-04-07 time 0.0347\n",
      "   > date: 2020-04-08 time 0.0412\n",
      "   > date: 2020-04-09 time 0.0318\n",
      "   > date: 2020-04-10 time 0.0323\n",
      "   > date: 2020-04-11 time 0.0375\n",
      "   > date: 2020-04-12 time 0.0357\n",
      "   > date: 2020-04-13 time 0.0363\n",
      "   > date: 2020-04-14 time 0.0342\n",
      "   > date: 2020-04-15 time 0.0462\n",
      "   > date: 2020-04-16 time 0.0399\n",
      "   > date: 2020-04-17 time 0.034\n",
      "   > date: 2020-04-18 time 0.0339\n",
      "   > date: 2020-04-19 time 0.0401\n",
      "   > date: 2020-04-20 time 0.036\n",
      "   > date: 2020-04-21 time 0.041\n",
      "   > date: 2020-04-22 time 0.0354\n",
      "   > date: 2020-04-23 time 0.033\n",
      "   > date: 2020-04-24 time 0.0345\n",
      "   > date: 2020-04-25 time 0.0371\n",
      "   > date: 2020-04-26 time 0.0468\n",
      "   > date: 2020-04-27 time 0.038\n",
      "   > date: 2020-04-28 time 0.0327\n",
      "   > date: 2020-04-29 time 0.0361\n",
      "   > date: 2020-04-30 time 0.0374\n",
      "@Canada DB ready, time 3.0325\n",
      "@Canada tweets saved as .JSON file, time 3.0325\n",
      "- - - - - o - - - - -\n",
      "\n",
      "Start with @GovCanHealth files:\n",
      "   > date: 2020-02-01 time 0.0217\n",
      "   > date: 2020-02-02 time 0.0184\n",
      "   > date: 2020-02-03 time 0.0204\n",
      "   > date: 2020-02-04 time 0.0191\n",
      "   > date: 2020-02-05 time 0.0187\n",
      "   > date: 2020-02-06 time 0.0171\n",
      "   > date: 2020-02-07 time 0.0179\n",
      "   > date: 2020-02-08 time 0.0164\n",
      "   > date: 2020-02-09 time 0.0152\n",
      "   > date: 2020-02-10 time 0.0161\n",
      "   > date: 2020-02-11 time 0.0182\n",
      "   > date: 2020-02-12 time 0.0254\n",
      "   > date: 2020-02-13 time 0.0216\n",
      "   > date: 2020-02-14 time 0.0174\n",
      "   > date: 2020-02-15 time 0.0162\n",
      "   > date: 2020-02-16 time 0.0179\n",
      "   > date: 2020-02-17 time 0.0179\n",
      "   > date: 2020-02-18 time 0.0179\n",
      "   > date: 2020-02-19 time 0.0183\n",
      "   > date: 2020-02-20 time 0.0197\n",
      "   > date: 2020-02-21 time 0.018\n",
      "   > date: 2020-02-22 time 0.0184\n",
      "   > date: 2020-02-23 time 0.0188\n",
      "   > date: 2020-02-24 time 0.0187\n",
      "   > date: 2020-02-25 time 0.0202\n",
      "   > date: 2020-02-26 time 0.0192\n",
      "   > date: 2020-02-27 time 0.0244\n",
      "   > date: 2020-02-28 time 0.023\n",
      "   > date: 2020-02-29 time 0.0203\n",
      "   > date: 2020-03-01 time 0.0204\n",
      "   > date: 2020-03-02 time 0.0242\n",
      "   > date: 2020-03-03 time 0.0241\n",
      "   > date: 2020-03-04 time 0.0245\n",
      "   > date: 2020-03-05 time 0.0301\n",
      "   > date: 2020-03-06 time 0.0273\n",
      "   > date: 2020-03-07 time 0.0214\n",
      "   > date: 2020-03-08 time 0.0213\n",
      "   > date: 2020-03-09 time 0.0255\n",
      "   > date: 2020-03-10 time 0.0296\n",
      "   > date: 2020-03-11 time 0.0564\n",
      "   > date: 2020-03-12 time 0.0486\n",
      "   > date: 2020-03-13 time 0.0507\n",
      "   > date: 2020-03-14 time 0.039\n",
      "   > date: 2020-03-15 time 0.0738\n",
      "   > date: 2020-03-16 time 0.0688\n",
      "   > date: 2020-03-17 time 0.08\n",
      "   > date: 2020-03-18 time 0.0562\n",
      "   > date: 2020-03-19 time 0.0576\n",
      "   > date: 2020-03-20 time 0.0682\n",
      "   > date: 2020-03-21 time 0.0546\n",
      "   > date: 2020-03-22 time 0.06\n",
      "   > date: 2020-03-23 time 0.0633\n",
      "   > date: 2020-03-24 time 0.0546\n",
      "   > date: 2020-03-25 time 0.0493\n",
      "   > date: 2020-03-26 time 0.0508\n",
      "   > date: 2020-03-27 time 0.0529\n",
      "   > date: 2020-03-28 time 0.0474\n",
      "   > date: 2020-03-29 time 0.0443\n",
      "   > date: 2020-03-30 time 0.0462\n",
      "   > date: 2020-03-31 time 0.0501\n",
      "   > date: 2020-04-01 time 0.0477\n",
      "   > date: 2020-04-02 time 0.047\n",
      "   > date: 2020-04-03 time 0.043\n",
      "   > date: 2020-04-04 time 0.0389\n",
      "   > date: 2020-04-05 time 0.0413\n",
      "   > date: 2020-04-06 time 0.0504\n",
      "   > date: 2020-04-07 time 0.045\n",
      "   > date: 2020-04-08 time 0.0479\n",
      "   > date: 2020-04-09 time 0.0435\n",
      "   > date: 2020-04-10 time 0.0485\n",
      "   > date: 2020-04-11 time 0.0419\n",
      "   > date: 2020-04-12 time 0.0351\n",
      "   > date: 2020-04-13 time 0.04\n",
      "   > date: 2020-04-14 time 0.0492\n",
      "   > date: 2020-04-15 time 0.0566\n",
      "   > date: 2020-04-16 time 0.0514\n",
      "   > date: 2020-04-17 time 0.0516\n",
      "   > date: 2020-04-18 time 0.0411\n",
      "   > date: 2020-04-19 time 0.0395\n",
      "   > date: 2020-04-20 time 0.0417\n",
      "   > date: 2020-04-21 time 0.045\n",
      "   > date: 2020-04-22 time 0.04\n",
      "   > date: 2020-04-23 time 0.0433\n",
      "   > date: 2020-04-24 time 0.035\n",
      "   > date: 2020-04-25 time 0.0345\n",
      "   > date: 2020-04-26 time 0.0372\n",
      "   > date: 2020-04-27 time 0.1033\n",
      "   > date: 2020-04-28 time 0.0994\n",
      "   > date: 2020-04-29 time 0.0723\n",
      "   > date: 2020-04-30 time 0.0693\n",
      "@GovCanHealth DB ready, time 3.4663\n",
      "@GovCanHealth tweets saved as .JSON file, time 3.4664\n",
      "- - - - - o - - - - -\n",
      "\n",
      "CPU times: user 1min 50s, sys: 15.7 s, total: 2min 6s\n",
      "Wall time: 2min 9s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "# Columns\n",
    "my_columns = ['account', 'date', 'content', 'user', 'replyCount', 'retweetCount',\n",
    "              'likeCount', 'quoteCount', 'lang', 'sourceLabel']\n",
    "\n",
    "dict_tot = {}\n",
    "\n",
    "# # Call and concatenate the data frames\n",
    "for ac in accounts:\n",
    "    # Create an empty pandas dataframe\n",
    "    df = pd.DataFrame(columns = my_columns)\n",
    "    t0 = time.clock()\n",
    "    print(\"Start with \" + ac + \" files:\")\n",
    "\n",
    "    # Call the JSON files of tweets\n",
    "    for d in my_dates:\n",
    "        t00 = time.clock()\n",
    "        aux = pd.read_json(my_folder + ac + '_' + str(d) + '.json', lines=True)\n",
    "        aux['account'] = ac\n",
    "        df = pd.concat([df, aux[my_columns]])\n",
    "        print(\"   > date: \" + str(d) + ' time ' + str(round(time.clock() - t00, 4)))\n",
    "\n",
    "    # Save a JSON file for each account\n",
    "    df.reset_index(drop=True, inplace=True)\n",
    "    dict_tot[ac] = df\n",
    "    print(ac + \" DB ready, time \" + str(round(time.clock() - t0, 4)))\n",
    "    # df.to_csv(my_folder + 'tweets_db_' + ac + '.csv')\n",
    "    print(ac + \" tweets saved as .JSON file, time \" + str(round(time.clock() - t0, 4)))\n",
    "    print(\"- - - - - o - - - - -\\n\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Merge all tweets in one `json` file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "⚠️ Just run this section once: the first time you run the notebook. If you previously run this cell and you are reopening this notebook, go to the next section _**Open Json file with all tweets**_."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 883 ms, sys: 685 ms, total: 1.57 s\n",
      "Wall time: 2.06 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "df_tot = pd.DataFrame()\n",
    "# Concatenate all tweets\n",
    "df_tot = pd.concat([dict_tot['@Canada'], \n",
    "                    dict_tot['@CanadianPM'], \n",
    "                    dict_tot['@GovCanHealth'], \n",
    "                    dict_tot['@JustinTrudeau']])\n",
    "df_tot.reset_index(drop=True, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "def unpack(df, column, fillna=None):\n",
    "    ret = None\n",
    "    if fillna is None:\n",
    "        ret = pd.concat([df, pd.DataFrame((d for idx, d in df[column].iteritems()))], axis=1)\n",
    "        del ret[column]\n",
    "    else:\n",
    "        ret = pd.concat([df, pd.DataFrame((d for idx, d in df[column].iteritems())).fillna(fillna)], axis=1)\n",
    "        del ret[column]\n",
    "    return ret"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 11.7 s, sys: 463 ms, total: 12.1 s\n",
      "Wall time: 12.1 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "# unpack information from user\n",
    "df_tot = unpack(df_tot, 'user')\n",
    "\n",
    "# select only the columns we wanted to save\n",
    "wanted_columns = ['account', 'date', 'content', 'replyCount', 'retweetCount', \n",
    "                  'likeCount', 'quoteCount', 'lang', 'sourceLabel', 'username', \n",
    "                  'followersCount', 'friendsCount', 'location']\n",
    "df_tot = df_tot[wanted_columns]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 9.54 s, sys: 1.87 s, total: 11.4 s\n",
      "Wall time: 14.1 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "# Save tweets as JSON file\n",
    "df_tot.to_json(my_folder + 'tweets_db.json')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Open `json` file with all tweets\n",
    "#### 👉 Use this in case you already run the previous chunks, and you are reopening the notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 11.8 s, sys: 2.88 s, total: 14.7 s\n",
      "Wall time: 15.7 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "# Open the file\n",
    "df_tot = pd.read_json(my_folder + 'tweets_db.json')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total time to run this notebook (until this point): 3.0 minutes.\n"
     ]
    }
   ],
   "source": [
    "print(\"Total time to run this notebook (until this point):\", np.round((time.time() - START)/60, 2), \"minutes.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preprocess tweets\n",
    "#### ⚠️ This process is very slow because it changes more than 70 thousand tweets, one by one. \n",
    "Instead of running this section, we recommend to use the python script __[preprocess.py](https://github.com/vcuspinera/Canada_response_covid/blob/master/src/preprocess.py)__ by running from the terminal the next code:  \n",
    "`$ python src/preprocess.py --input_dir=tweets/ --output_dir=tweets/`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "def preprocess(text, irrelevant_pos = ['SPACE'],\n",
    "              avoid_entities = ['ORG']):\n",
    "    \"\"\"\n",
    "    Function that identify sensible information and delete some of \n",
    "    these data as emails and urls.\n",
    "    Parameters\n",
    "    -------------\n",
    "    text : (list)\n",
    "        the list of text to be preprocessed\n",
    "    irrelevant_pos : (list)\n",
    "        a list of irrelevant 'pos' tags\n",
    "    avoid_entities : (list)\n",
    "        a list of entity labels to be avoided\n",
    "\n",
    "    Returns\n",
    "    -------------\n",
    "    (list) list of preprocessed text\n",
    "\n",
    "    Example\n",
    "    -------------\n",
    "    example = [\"Contact me at george23@gmail.com\",\n",
    "           \"@vcuspinera my webpage is https://vcuspinera.github.io\"]\n",
    "    preprocess(example)\n",
    "    (output:) ['contact me at',\n",
    "               'my webpage is']\n",
    "    \"\"\"\n",
    "    result = []\n",
    "\n",
    "    # function\n",
    "    for sent in text:\n",
    "        sent = str(sent).lower()\n",
    "\n",
    "        result_sent = []\n",
    "        doc = nlp(sent)\n",
    "        entities = [str(ent) for ent in doc.ents if ent.label_ in avoid_entities]\n",
    "        # This helps to detect names organization\n",
    "\n",
    "        for token in doc:            \n",
    "            if (token.like_email or\n",
    "                token.like_url or\n",
    "                token.pos_ in irrelevant_pos or\n",
    "                str(token) in entities\n",
    "               ):\n",
    "                continue\n",
    "            else:\n",
    "                if str(token) in string.punctuation:\n",
    "                    try:\n",
    "                        result_sent[-1] = str(result_sent[-1]) + str(token)\n",
    "                    except:\n",
    "                        result_sent.append(str(token))\n",
    "                else:\n",
    "                    result_sent.append(str(token))\n",
    "        result.append(\" \".join(result_sent))\n",
    "    return result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Preprocess tweets\n",
    "df_tot['tweet'] = preprocess(df_tot['content'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save file\n",
    "df_tot.drop('content', axis=1).to_json(output_dir + 'tweets_db_clean.json')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## **Basic analysis** and **EDA**\n",
    "\n",
    "#### 👉 [Click here](https://github.com/vcuspinera/Canada_response_covid/blob/master/src/eda.ipynb) to see the initial Data Analysis in the EDA jupyter notebook of this project."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "164.988px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}