src/twitter-search_v1_TwitterAPI.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Search for tweets\n",
    "\n",
    "#### — _Using Twitter API_ —\n",
    "\n",
    "### ⚠️ This notebook was not useful to retrieve historic tweets.\n",
    "\n",
    "Considering that Trudeau's announcement was done on March 27th, I was willing to get tweets from March 1st until April 30th, to look for messages before and after the announcement and compare how much the sentiment of people changed over this period.\n",
    "\n",
    "However, the standard twitter developer's account only gives access to search historic databases with a 7-day limit, which means that I was not able to find tweets for a date older than one week. \n",
    "\n",
    "For this reason, this notebook which uses the standard twitter developer's account is not useful for the objective of our project. However, this notebook can be used as reference; to use it you will need to get a Twitter developer's account. \n",
    "\n",
    "Instead, we invite you to look at the notebook **Canada_Twitter.ipynb**, located in the same [_src_ folder](https://github.com/vcuspinera/Canada_response_covid/blob/master/src), where we successfully download the historic tweets used for this project.\n",
    "\n",
    "_Author: Victor Cuspinera ([vcuspinera](https://github.com/vcuspinera))_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import twitter\n",
    "import time\n",
    "import collections\n",
    "from datetime import datetime, timedelta\n",
    "from pytz import timezone"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bring your twitter's keys"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Call your twitter keys\n",
    "keys = ! ../keys/twitter_config.py\n",
    "\n",
    "# Save keys to use them later\n",
    "api = twitter.Api(consumer_key = keys[0],\n",
    "                  consumer_secret = keys[1],\n",
    "                  access_token_key = keys[2],\n",
    "                  access_token_secret = keys[3],\n",
    "                  sleep_on_rate_limit=True #\n",
    "                 )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# twitter accounts\n",
    "accounts = (\"CanadianPM\", \"Canada\", \"OpenGovCan\", \"JustinTrudeau\")\n",
    "\n",
    "# dates\n",
    "# format YYYY-MM-DD, and has a 7-day limit (no tweets will be found \n",
    "# for a date older than one week.)\n",
    "today = datetime.now()#timezone('US/Pacific'))\n",
    "dates = list()\n",
    "for d in range(7, -2, -1):\n",
    "    aux = today - timedelta(days=d)\n",
    "    dates.append(aux.strftime(\"%Y-%m-%d\"))\n",
    "\n",
    "today = today.strftime(\"%Y-%m-%d\")\n",
    "\n",
    "# tweets\n",
    "num_tweets = 100\n",
    "max_tweets = 180\n",
    "time_epoc = 900 # 15 minutes = 900 seconds\n",
    "time_waiting = time_epoc / max_tweets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Restrictions\n",
    "The standard API rate limits described in the [Twitter developer's documents](https://developer.twitter.com/en/docs/tweets/search/api-reference/get-search-tweets) related with GET (read) endpoints are shown in the next table.  \n",
    "\n",
    "Note that endpoints not listed in the chart default to 15 requests per allotted user. All request windows are *15 minutes in length*.  These rate limits apply to the standard API endpoints only.\n",
    "\n",
    "| Endpoint | Resource family | Requests / window (user auth) | Requests / window (app auth) |\n",
    "|---|:--:|:---:|:---:|\n",
    "|GET search/tweets | search | 180 | 450 |\n",
    "\n",
    "Also, the search index has a 7-day limit, which means that no tweets will be found for a date older than one week."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Retrieve tweets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--- Retrieving tweets ---\n",
      "\n",
      "DATE       : ACCOUNT\n",
      "---------- : -------\n",
      "2020-08-21 : CanadianPM\n",
      "2020-08-22 : CanadianPM\n",
      "2020-08-23 : CanadianPM\n",
      "2020-08-24 : CanadianPM\n",
      "2020-08-25 : CanadianPM\n",
      "2020-08-26 : CanadianPM\n",
      "2020-08-27 : CanadianPM\n",
      "2020-08-28 : CanadianPM\n",
      "2020-08-21 : Canada\n",
      "2020-08-22 : Canada\n",
      "2020-08-23 : Canada\n",
      "2020-08-24 : Canada\n",
      "2020-08-25 : Canada\n",
      "2020-08-26 : Canada\n",
      "2020-08-27 : Canada\n",
      "2020-08-28 : Canada\n",
      "2020-08-21 : OpenGovCan\n",
      "2020-08-22 : OpenGovCan\n",
      "2020-08-23 : OpenGovCan\n",
      "2020-08-24 : OpenGovCan\n",
      "2020-08-25 : OpenGovCan\n",
      "2020-08-26 : OpenGovCan\n",
      "2020-08-27 : OpenGovCan\n",
      "2020-08-28 : OpenGovCan\n",
      "2020-08-21 : GovCanHealth\n",
      "2020-08-22 : GovCanHealth\n",
      "2020-08-23 : GovCanHealth\n",
      "2020-08-24 : GovCanHealth\n",
      "2020-08-25 : GovCanHealth\n",
      "2020-08-26 : GovCanHealth\n",
      "2020-08-27 : GovCanHealth\n",
      "2020-08-28 : GovCanHealth\n",
      "2020-08-21 : JustinTrudeau\n",
      "2020-08-22 : JustinTrudeau\n",
      "2020-08-23 : JustinTrudeau\n",
      "2020-08-24 : JustinTrudeau\n",
      "2020-08-25 : JustinTrudeau\n",
      "2020-08-26 : JustinTrudeau\n",
      "2020-08-27 : JustinTrudeau\n",
      "2020-08-28 : JustinTrudeau\n",
      "\n",
      "--- Completed ---\n"
     ]
    }
   ],
   "source": [
    "# dictionary of dictionaries\n",
    "results = collections.defaultdict(dict)\n",
    "print(\"--- Retrieving tweets ---\")\n",
    "print(\"\\nDATE       : ACCOUNT\")\n",
    "print(\"---------- : -------\")\n",
    "for ac in accounts:\n",
    "    for d in range(0, len(dates)-1):\n",
    "#     for da in dates:\n",
    "        print(dates[d] + \" : \" + ac)\n",
    "        results[ac][dates[d]] = api.GetSearch(\n",
    "            raw_query = \"q=\" + ac + \n",
    "                        \"%20&until=\" + dates[d+1] +\n",
    "                        \"&count=\" + str(num_tweets)\n",
    "        )\n",
    "        time.sleep(time_waiting)\n",
    "print(\"\\n--- Completed ---\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "accounts: ('CanadianPM', 'Canada', 'OpenGovCan', 'GovCanHealth', 'JustinTrudeau')\n",
      "dates: ['2020-08-21', '2020-08-22', '2020-08-23', '2020-08-24', '2020-08-25', '2020-08-26', '2020-08-27', '2020-08-28']\n"
     ]
    }
   ],
   "source": [
    "print(\"accounts:\", accounts)\n",
    "print(\"dates:\", dates[:-1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Uncomment and run this cell to see the retrieved results\n",
    "# results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Account: Canada\n",
      "Date: 2020-08-28\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Status(ID=1299319363930148865, ScreenName=durike, Created=Fri Aug 28 12:14:11 +0000 2020, Text='RT @ChukwudubemIgb1: @MaziNnamdiKanu https://t.co/GPVzTPEzuP @netanyahu @PMOIndia @JapanGov @CanadianPM @BrazilGovNews @BorisJohnson_MP @MF…')"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# First tweet by the public to the 'Canada' account on the last day of the period.\n",
    "print('Account: Canada')\n",
    "print('Date:', today)\n",
    "results['CanadianPM'][today][0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load dictionaries (if exist) and save tweets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load \n",
    "#  - retrieve dictionaries if they exist,\n",
    "#  - if they doesn't exist, it creates them,\n",
    "#  - finally adds the tweets from the last 7 days.\n",
    "results_all = collections.defaultdict(dict)\n",
    "for ac in accounts: \n",
    "    try:\n",
    "        results_all[ac] = np.load('../tweets/non-historic/tweets_' +ac+ '.npy',allow_pickle='TRUE').item()\n",
    "    except:\n",
    "        next\n",
    "    for d in range(0, len(dates)-1):\n",
    "        results_all[ac][dates[d]] = results[ac][dates[d]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Uncomment and run this cell to see all the retrieved results over time\n",
    "# results_all"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save\n",
    "for ac in accounts:\n",
    "    np.save('../tweets/non-historic/tweets_' + ac + '.npy', results_all[ac])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dict_keys(['2020-07-22', '2020-07-23', '2020-07-24', '2020-07-25', '2020-07-26', '2020-07-27', '2020-07-28', '2020-07-29', '2020-07-30', '2020-07-31', '2020-08-01', '2020-08-02', '2020-08-03', '2020-08-04', '2020-08-05', '2020-08-06', '2020-08-07', '2020-08-08', '2020-08-09', '2020-08-10', '2020-08-11', '2020-08-12', '2020-08-13', '2020-08-15', '2020-08-16', '2020-08-17', '2020-08-18', '2020-08-19', '2020-08-20', '2020-08-21', '2020-08-22', '2020-08-14', '2020-08-23', '2020-08-24', '2020-08-25', '2020-08-26', '2020-08-27', '2020-08-28'])\n",
      "dict_keys(['2020-07-22', '2020-07-23', '2020-07-24', '2020-07-25', '2020-07-26', '2020-07-27', '2020-07-28', '2020-07-29', '2020-07-30', '2020-07-31', '2020-08-01', '2020-08-02', '2020-08-03', '2020-08-04', '2020-08-05', '2020-08-06', '2020-08-07', '2020-08-08', '2020-08-09', '2020-08-10', '2020-08-11', '2020-08-12', '2020-08-13', '2020-08-15', '2020-08-16', '2020-08-17', '2020-08-18', '2020-08-19', '2020-08-20', '2020-08-21', '2020-08-22', '2020-08-14', '2020-08-23', '2020-08-24', '2020-08-25', '2020-08-26', '2020-08-27', '2020-08-28'])\n",
      "dict_keys(['2020-07-22', '2020-07-23', '2020-07-24', '2020-07-25', '2020-07-26', '2020-07-27', '2020-07-28', '2020-07-29', '2020-07-30', '2020-07-31', '2020-08-01', '2020-08-02', '2020-08-03', '2020-08-04', '2020-08-05', '2020-08-06', '2020-08-07', '2020-08-08', '2020-08-09', '2020-08-10', '2020-08-11', '2020-08-12', '2020-08-13', '2020-08-15', '2020-08-16', '2020-08-17', '2020-08-18', '2020-08-19', '2020-08-20', '2020-08-21', '2020-08-22', '2020-08-14', '2020-08-23', '2020-08-24', '2020-08-25', '2020-08-26', '2020-08-27', '2020-08-28'])\n",
      "dict_keys(['2020-07-22', '2020-07-23', '2020-07-24', '2020-07-25', '2020-07-26', '2020-07-27', '2020-07-28', '2020-07-29', '2020-07-30', '2020-07-31', '2020-08-01', '2020-08-02', '2020-08-03', '2020-08-04', '2020-08-05', '2020-08-06', '2020-08-07', '2020-08-08', '2020-08-09', '2020-08-10', '2020-08-11', '2020-08-12', '2020-08-13', '2020-08-15', '2020-08-16', '2020-08-17', '2020-08-18', '2020-08-19', '2020-08-20', '2020-08-21', '2020-08-22', '2020-08-14', '2020-08-23', '2020-08-24', '2020-08-25', '2020-08-26', '2020-08-27', '2020-08-28'])\n",
      "dict_keys(['2020-08-21', '2020-08-22', '2020-08-23', '2020-08-24', '2020-08-25', '2020-08-26', '2020-08-27', '2020-08-28'])\n"
     ]
    }
   ],
   "source": [
    "for ac in accounts:\n",
    "    print(results_all[ac].keys())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}