twitter_data_extraction.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Covid19 - Twitter data extraction\n",
    "by Victoria, Maha, Gopi\n",
    "\n",
    "## Table of contents\n",
    "- Introduction\n",
    "- Authenticatications\n",
    "    - Twitter\n",
    "    - Google sheets\n",
    "- Gathering data & storing\n",
    "\n",
    "\n",
    "## Introduction\n",
    "This notebook is part of the project developed for the FLT Big Data Hackathon, whose objective is to create interesting and trustworthy analyses and visualizations about the COVID19 situation and its correlation with the stock market. \n",
    "\n",
    "In this notebook we use the Twitter API to retrieve the tweets related to COVID19 hashtags and economic tags, to perform a sentimental analysis and store it programatically in a google sheets file. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package punkt to\n",
      "[nltk_data]     C:\\Users\\v.perez\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package punkt is already up-to-date!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Load important libraries\n",
    "import gspread \n",
    "from df2gspread import df2gspread as d2g\n",
    "from oauth2client.service_account import ServiceAccountCredentials\n",
    "import json\n",
    "import tweepy\n",
    "from textblob import TextBlob\n",
    "from tweepy import Stream\n",
    "from tweepy import StreamListener\n",
    "import pandas as pd\n",
    "import re\n",
    "import csv\n",
    "import nltk\n",
    "from  geopy.geocoders import Nominatim\n",
    "from datetime import datetime\n",
    "import pycountry\n",
    "nltk.download('punkt')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Authentication\n",
    "### Twitter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 175
    },
    "colab_type": "code",
    "id": "jqdQzdqytO88",
    "outputId": "b46f1541-50e8-4fe4-a091-00175482cd65"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Authentication OK\n",
      "printing tweets from timeline \n",
      " \n",
      "#AHORA Vuelven a extender la cuarentena obligatoria, ahora hasta el 7 de junio\n",
      "\n",
      "https://t.co/qb8rjD1xV2\n",
      "\n",
      "RT @normanesnombre: ojalá alguien me cuidara tanto como Alberto está cuidando a Larreta.\n",
      "\n",
      "RT @ElDedoAcusador: Harta de que hablen de una super cuarentena en Argentina cuando en muchisimos lugares del pais, definitivamente muchos…\n",
      "\n",
      "#urgente @alferdez confirmó que la cuarentena seguirá hasta el 7 de junio https://t.co/XF0Wx2ud1i\n",
      "\n",
      "RT @CharlotteBRF: Some intermediate &amp; advanced #JavaScript concepts very well explained in @LadybugPodcast 🐞 https://t.co/Iv9Zq6zWyw . This…\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Load twitter credentials\n",
    "with open(\"covid19-sentanalysis-twitter_credentials.json\") as datafile:\n",
    "  data = json.load(datafile)\n",
    "\n",
    "# Define the keys\n",
    "consumer_key= data['consumer_key'] #'API_CONSUMER_KEY_HERE'\n",
    "consumer_secret=  data['consumer_secret']#'CONSUMER_SECRET_HERE'\n",
    "\n",
    "access_token= data['access_token_key'] #'ACCESS_TOKEN_HERE'\n",
    "access_token_secret= data['access_token_secret'] #'ACCESS_TOKEN_SECRET_HERE'\n",
    "\n",
    "\n",
    "#Crate the auth object\n",
    "auth = tweepy.OAuthHandler(consumer_key, consumer_secret)\n",
    "auth.set_access_token(access_token, access_token_secret)\n",
    "\n",
    "# create API, set limits to avouid errors because of a timeout \n",
    "api = tweepy.API(auth, wait_on_rate_limit = True, wait_on_rate_limit_notify = True)\n",
    "\n",
    "try:\n",
    "    api.verify_credentials()\n",
    "    print(\"Authentication OK\")\n",
    "except:\n",
    "    print(\"Error during authentication\")\n",
    "\n",
    "#Print 5 tweets for testing purposes - Should be deleted afterwards\n",
    "home_tweets = api.home_timeline(count=5)\n",
    "print(\"printing tweets from timeline \\n \")\n",
    "for tweet in home_tweets:\n",
    "    print(tweet.text)\n",
    "    print(\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Google sheets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "scope = [\n",
    "   'https://spreadsheets.google.com/feeds',\n",
    "         'https://www.googleapis.com/auth/drive']\n",
    "\n",
    "#authenticate gsheets\n",
    "google_key_file = 'service_key.json'\n",
    "credentials = ServiceAccountCredentials.from_json_keyfile_name(google_key_file, scope)\n",
    "gc = gspread.authorize(credentials)\n",
    "\n",
    "# Define spreadsheet access\n",
    "spreadsheet_key = '1auoQ9XanosnM7RUInzqeZi9EIgwtCtmtubNpXrfF6OM' \n",
    "wks_name = 'sentimentAnalysis'\n",
    "\n",
    "# Open the file\n",
    "book = gc.open_by_key(spreadsheet_key) \n",
    "worksheet = book.worksheet(wks_name) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "DlNcN1cx0uOD"
   },
   "source": [
    "## Gathering data & storing\n",
    "**GET Twitter Stream and Do Sentiment Analysis in Real time**\n",
    "\n",
    "## Gathering and Storing Twitter data\n",
    "\n",
    "In order to perform an analysis we had to retrieve all the relevant twitter data. \n",
    "Our analysis has two dimensions:\n",
    "- Real time data: It gives us an idea of what the sentiments are, what are the words trending at the moment and the countries participating\n",
    "\n",
    "- Archive data: This data is focues entirely on the findings on the dow jones data, for the specific dates where we find some insights that caught our attention we dived more into how the sentiment was varying. \n",
    "\n",
    "### Real time analysis\n",
    "\n",
    "In this notebook we use the Twitter API to retrieve the tweets related to COVID19 hashtags and economic tags, to perform a sentimental analysis and store it programatically in a google sheets file. \n",
    "\n",
    "#### Technical overview: \n",
    "\n",
    "1.   Data Extraction\n",
    "Twitter live streams are extracted and the dataset is created. \n",
    "\n",
    "*   Twitter live streams are obtained using `tweepy StreamListner`. \n",
    "*   Retrieved tweets are preprocessed and the following fields are extracted or derived,\n",
    "\n",
    "\n",
    "| Field Name                                                      | Origin                                          | Transformations                                                                            |\n",
    "|-----------------------------------------------------------------|-------------------------------------------------|--------------------------------------------------------------------------------------------|\n",
    "| - id - user_id - Text - created_at - location - followers_count | Twitter API endpoints                           | No need                                                                                    |\n",
    "| - timestamp                                                     | Created_at twitter API endpoint                 | Transformation of the 'created_at` field using `datetime` library                          |\n",
    "| - latitude - longitude                                          | Twitter API endpoints: - coordinates - location | Whenever the tweet had no coordinates assigned the location was used to obtain the country |\n",
    "| - country - country_code                                        | Twitter API endpoints: - coordinates - location |                                                                                            |\n",
    "| - hashtags                                                      | StartFragmentTwitter API endpoints: EndFragment |                                                                                            |\n",
    "| - polarity - subjectivity                                       | Sentiment analysis                              | Sentiment analysis performed on the tweet text                                             |\n",
    "\n",
    "2. Data storing \n",
    "\n",
    "The data was collected and stored programatically in google sheets file, in order to have it available online.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "huknvBk9Y8bj"
   },
   "outputs": [],
   "source": [
    "header_name = ['id', 'user_id','Text','created_at','timestamp','location','latitude','longitude','country',\n",
    "               'country_code','followers_count','hashtags','polarity','subjectivity']\n",
    "\n",
    "class Listener(StreamListener):\n",
    "    \n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.max_tweets = 5\n",
    "        self.tweet_count = 0\n",
    "        self.geolocator = Nominatim()\n",
    "        self.tweet_list = []\n",
    "    \n",
    "    def on_data(self, data):\n",
    "        raw_twitts = json.loads(data)\n",
    "        try:\n",
    "            #  Fields we need: id, created_at, text, coordinates, author_id\n",
    "            full_tweets = raw_twitts.copy()\n",
    "            tweets = raw_twitts['text']\n",
    "            tweets = ' '.join(re.sub(\"(@[A-Za-z0-9]+) | ({*0-9A-Za-z \\t]) |] (\\wt:\\/\\/\\St+)\", \" \", tweets).split())\n",
    "            tweets = ' '.join(re.sub('RT',' ', tweets).split())  \n",
    "  \n",
    "  \n",
    "            blob = TextBlob(tweets.strip())\n",
    "            #global polarity \n",
    "            #global subjectivity \n",
    "            \n",
    "            polarity = 0\n",
    "            subjectivity = 0\n",
    "            \n",
    "            for sent in blob.sentences:\n",
    "                polarity += round(sent.sentiment.polarity,2)\n",
    "                subjectivity += round(sent.sentiment.subjectivity,2)\n",
    "\n",
    "            #get timestamp from created_at\n",
    "            time_created_at = raw_twitts['created_at']\n",
    "            t = time_created_at.split('+0000 ')\n",
    "            time = t[0] +t[1]\n",
    "            format_time = '%a %b %d %H:%M:%S %Y'\n",
    "            date_time = datetime.strptime(time,format_time)\n",
    "            ts = int(date_time.timestamp())\n",
    "            \n",
    "            #get lat, long from location\n",
    "            lat = None\n",
    "            long = None\n",
    "            if raw_twitts['user']['location']:\n",
    "                loc = self.geolocator.geocode(raw_twitts['user']['location'])\n",
    "                if loc:\n",
    "                    lat = loc.latitude\n",
    "                    long = loc.longitude\n",
    "                    location = self.geolocator.reverse(str(lat)+','+str(long))\n",
    "                    country = location.raw['address']['country']\n",
    "                    #country name may be official name or name. \n",
    "                    #So, try to get the country code using both, if couldn't find then set code to None.\n",
    "                    #Some countries have symbols, so better to keep None if not found using above method\n",
    "                    if country:\n",
    "                        country_official_name = pycountry.countries.get(official_name=country)\n",
    "                        if country_official_name:\n",
    "                            country_code = country_official_name.alpha_3\n",
    "                        else:\n",
    "                            country_name = pycountry.countries.get(name=country)\n",
    "                            if country_name:\n",
    "                                country_code = country_name.alpha_3\n",
    "                            else:\n",
    "                                country_code = None\n",
    "                    else:\n",
    "                        country_code = None\n",
    "                        \n",
    "        \n",
    "            if lat and long:\n",
    "                info = {'id':raw_twitts['id'],\n",
    "                            'user_id':raw_twitts['user']['id'], \n",
    "                            'Text':raw_twitts['text'],\n",
    "                            'created_at':raw_twitts['created_at'],\n",
    "                            'timestamp':ts,\n",
    "                            'location':raw_twitts['user']['location'],\n",
    "                            'latitude':lat,\n",
    "                            'longitude':long,\n",
    "                            'country': country,\n",
    "                            'country_code': country_code,\n",
    "                            'followers_count':raw_twitts['user']['followers_count'],\n",
    "                            'hashtags': ''.join(re.findall(r'\\B#\\w*[a-zA-Z]+\\w*', raw_twitts['text'])),\n",
    "                            'polarity': polarity,\n",
    "                            'subjectivity': subjectivity\n",
    "                       }\n",
    "                self.tweet_list.append(info)\n",
    "\n",
    "            #print (tweets,'\\n')    \n",
    "        except:\n",
    "            print('ERROR got')\n",
    "        else:\n",
    "            self.tweet_count+=1\n",
    "                # Once it reaches a fix limit the Write the data into gsheets\n",
    "            if(self.tweet_count==self.max_tweets):          \n",
    "                # save to a dataframe for eeasier file upload\n",
    "                df_tweet_list = pd.DataFrame(self.tweet_list, columns = header_name)\n",
    "            \n",
    "                values = df_tweet_list.values.tolist()\n",
    "                book.values_append(wks_name, {'valueInputOption': 'USER_ENTERED'}, {'values': values})\n",
    "                \n",
    "                 #d2g.upload(df_tweet_list, spreadsheet_key, wks_name, clean =True, credentials=credentials, row_names=False)\n",
    "            \n",
    "                print(\"completed\")\n",
    "                self.tweet_count = 0\n",
    "                #return(False)\n",
    "            else:\n",
    "                decoded = json.loads(data)\n",
    "\n",
    "        def on_error(self, status):\n",
    "            print(status)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "u-MAEzOxY9k9"
   },
   "outputs": [],
   "source": [
    "auth = tweepy.OAuthHandler(consumer_key, consumer_secret)\n",
    "auth.set_access_token(access_token, access_token_secret)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "colab_type": "code",
    "id": "jE5iBNqMY-D5",
    "outputId": "192cc8f0-3c22-4cd0-e3c2-ea65a512c153"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\v.perez\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:10: DeprecationWarning: Using Nominatim with the default \"geopy/1.22.0\" `user_agent` is strongly discouraged, as it violates Nominatim's ToS https://operations.osmfoundation.org/policies/nominatim/ and may possibly cause 403 and 429 HTTP errors. Please specify a custom `user_agent` with `Nominatim(user_agent=\"my-application\")` or by overriding the default `user_agent`: `geopy.geocoders.options.default_user_agent = \"my-application\"`. In geopy 2.0 this will become an exception.\n",
      "  # Remove the CWD from sys.path while we load stuff.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "completed\n",
      "completed\n",
      "completed\n",
      "completed\n",
      "ERROR got\n",
      "completed\n",
      "ERROR got\n",
      "ERROR got\n",
      "ERROR got\n",
      "completed\n",
      "completed\n",
      "ERROR got\n",
      "ERROR got\n"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-14-f6bb4b7d8e4e>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m      1\u001b[0m \u001b[0mtwitter_stream\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mStream\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mauth\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mListener\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 2\u001b[1;33m \u001b[0mtwitter_stream\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mfilter\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mtrack\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;33m[\u001b[0m\u001b[1;34m'covid'\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;34m'covid19'\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;34m'economic pandemic'\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;34m'chinese virus impact'\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;34m'lockdown'\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;34m'lockdown recession'\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mlanguages\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;34m'en'\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\tweepy\\streaming.py\u001b[0m in \u001b[0;36mfilter\u001b[1;34m(self, follow, track, is_async, locations, stall_warnings, languages, encoding, filter_level)\u001b[0m\n\u001b[0;32m    472\u001b[0m             \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mbody\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;34m'filter_level'\u001b[0m\u001b[1;33m]\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mfilter_level\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mencode\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mencoding\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    473\u001b[0m         \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msession\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mparams\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;33m{\u001b[0m\u001b[1;34m'delimited'\u001b[0m\u001b[1;33m:\u001b[0m \u001b[1;34m'length'\u001b[0m\u001b[1;33m}\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 474\u001b[1;33m         \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_start\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mis_async\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    475\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    476\u001b[0m     def sitestream(self, follow, stall_warnings=False,\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\tweepy\\streaming.py\u001b[0m in \u001b[0;36m_start\u001b[1;34m(self, is_async)\u001b[0m\n\u001b[0;32m    387\u001b[0m             \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_thread\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mstart\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    388\u001b[0m         \u001b[1;32melse\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 389\u001b[1;33m             \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_run\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    390\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    391\u001b[0m     \u001b[1;32mdef\u001b[0m \u001b[0mon_closed\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mresp\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\tweepy\\streaming.py\u001b[0m in \u001b[0;36m_run\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    287\u001b[0m                     \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msnooze_time\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msnooze_time_step\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    288\u001b[0m                     \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mlistener\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mon_connect\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 289\u001b[1;33m                     \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_read_loop\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mresp\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    290\u001b[0m             \u001b[1;32mexcept\u001b[0m \u001b[1;33m(\u001b[0m\u001b[0mTimeout\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mssl\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mSSLError\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;32mas\u001b[0m \u001b[0mexc\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    291\u001b[0m                 \u001b[1;31m# This is still necessary, as a SSLError can actually be\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\tweepy\\streaming.py\u001b[0m in \u001b[0;36m_read_loop\u001b[1;34m(self, resp)\u001b[0m\n\u001b[0;32m    349\u001b[0m             \u001b[0mnext_status_obj\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mbuf\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mread_len\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mlength\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    350\u001b[0m             \u001b[1;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mrunning\u001b[0m \u001b[1;32mand\u001b[0m \u001b[0mnext_status_obj\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 351\u001b[1;33m                 \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_data\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mnext_status_obj\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    352\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    353\u001b[0m             \u001b[1;31m# # Note: keep-alive newlines might be inserted before each length value.\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\tweepy\\streaming.py\u001b[0m in \u001b[0;36m_data\u001b[1;34m(self, data)\u001b[0m\n\u001b[0;32m    321\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    322\u001b[0m     \u001b[1;32mdef\u001b[0m \u001b[0m_data\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mdata\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 323\u001b[1;33m         \u001b[1;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mlistener\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mon_data\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mdata\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;32mis\u001b[0m \u001b[1;32mFalse\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    324\u001b[0m             \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mrunning\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;32mFalse\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    325\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m<ipython-input-5-04d79f1c403e>\u001b[0m in \u001b[0;36mon_data\u001b[1;34m(self, data)\u001b[0m\n\u001b[0;32m     97\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     98\u001b[0m                 \u001b[0mvalues\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mdf_tweet_list\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mvalues\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mtolist\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 99\u001b[1;33m                 \u001b[0mbook\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mvalues_append\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mwks_name\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m{\u001b[0m\u001b[1;34m'valueInputOption'\u001b[0m\u001b[1;33m:\u001b[0m \u001b[1;34m'USER_ENTERED'\u001b[0m\u001b[1;33m}\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m{\u001b[0m\u001b[1;34m'values'\u001b[0m\u001b[1;33m:\u001b[0m \u001b[0mvalues\u001b[0m\u001b[1;33m}\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    100\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    101\u001b[0m                  \u001b[1;31m#d2g.upload(df_tweet_list, spreadsheet_key, wks_name, clean =True, credentials=credentials, row_names=False)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\gspread\\models.py\u001b[0m in \u001b[0;36mvalues_append\u001b[1;34m(self, range, params, body)\u001b[0m\n\u001b[0;32m    162\u001b[0m         \"\"\"\n\u001b[0;32m    163\u001b[0m         \u001b[0murl\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mSPREADSHEET_VALUES_APPEND_URL\u001b[0m \u001b[1;33m%\u001b[0m \u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mid\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mquote\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mrange\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 164\u001b[1;33m         \u001b[0mr\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mclient\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mrequest\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'post'\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0murl\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mparams\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mparams\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mjson\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mbody\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    165\u001b[0m         \u001b[1;32mreturn\u001b[0m \u001b[0mr\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mjson\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    166\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\gspread\\client.py\u001b[0m in \u001b[0;36mrequest\u001b[1;34m(self, method, endpoint, params, data, json, files, headers)\u001b[0m\n\u001b[0;32m     65\u001b[0m             \u001b[0mdata\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mdata\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     66\u001b[0m             \u001b[0mfiles\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mfiles\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 67\u001b[1;33m             \u001b[0mheaders\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mheaders\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     68\u001b[0m         )\n\u001b[0;32m     69\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\requests\\sessions.py\u001b[0m in \u001b[0;36mpost\u001b[1;34m(self, url, data, json, **kwargs)\u001b[0m\n\u001b[0;32m    579\u001b[0m         \"\"\"\n\u001b[0;32m    580\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 581\u001b[1;33m         \u001b[1;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mrequest\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'POST'\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0murl\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mdata\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mdata\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mjson\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mjson\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    582\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    583\u001b[0m     \u001b[1;32mdef\u001b[0m \u001b[0mput\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0murl\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mdata\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;32mNone\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\google\\auth\\transport\\requests.py\u001b[0m in \u001b[0;36mrequest\u001b[1;34m(self, method, url, data, headers, max_allowed_time, timeout, **kwargs)\u001b[0m\n\u001b[0;32m    448\u001b[0m                 \u001b[0mheaders\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mrequest_headers\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    449\u001b[0m                 \u001b[0mtimeout\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mtimeout\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 450\u001b[1;33m                 \u001b[1;33m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    451\u001b[0m             )\n\u001b[0;32m    452\u001b[0m         \u001b[0mremaining_time\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mguard\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mremaining_timeout\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\requests\\sessions.py\u001b[0m in \u001b[0;36mrequest\u001b[1;34m(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json)\u001b[0m\n\u001b[0;32m    531\u001b[0m         }\n\u001b[0;32m    532\u001b[0m         \u001b[0msend_kwargs\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mupdate\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0msettings\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 533\u001b[1;33m         \u001b[0mresp\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msend\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mprep\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m**\u001b[0m\u001b[0msend_kwargs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    534\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    535\u001b[0m         \u001b[1;32mreturn\u001b[0m \u001b[0mresp\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\requests\\sessions.py\u001b[0m in \u001b[0;36msend\u001b[1;34m(self, request, **kwargs)\u001b[0m\n\u001b[0;32m    644\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    645\u001b[0m         \u001b[1;31m# Send the request\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 646\u001b[1;33m         \u001b[0mr\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0madapter\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msend\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mrequest\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    647\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    648\u001b[0m         \u001b[1;31m# Total elapsed time of the request (approximately)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\requests\\adapters.py\u001b[0m in \u001b[0;36msend\u001b[1;34m(self, request, stream, timeout, verify, cert, proxies)\u001b[0m\n\u001b[0;32m    447\u001b[0m                     \u001b[0mdecode_content\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;32mFalse\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    448\u001b[0m                     \u001b[0mretries\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mmax_retries\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 449\u001b[1;33m                     \u001b[0mtimeout\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mtimeout\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    450\u001b[0m                 )\n\u001b[0;32m    451\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\urllib3\\connectionpool.py\u001b[0m in \u001b[0;36murlopen\u001b[1;34m(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)\u001b[0m\n\u001b[0;32m    670\u001b[0m                 \u001b[0mbody\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mbody\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    671\u001b[0m                 \u001b[0mheaders\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mheaders\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 672\u001b[1;33m                 \u001b[0mchunked\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mchunked\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    673\u001b[0m             )\n\u001b[0;32m    674\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\urllib3\\connectionpool.py\u001b[0m in \u001b[0;36m_make_request\u001b[1;34m(self, conn, method, url, timeout, chunked, **httplib_request_kw)\u001b[0m\n\u001b[0;32m    419\u001b[0m                     \u001b[1;31m# Python 3 (including for exceptions like SystemExit).\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    420\u001b[0m                     \u001b[1;31m# Otherwise it looks like a bug in the code.\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 421\u001b[1;33m                     \u001b[0msix\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mraise_from\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0me\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;32mNone\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    422\u001b[0m         \u001b[1;32mexcept\u001b[0m \u001b[1;33m(\u001b[0m\u001b[0mSocketTimeout\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mBaseSSLError\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mSocketError\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    423\u001b[0m             \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_raise_timeout\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0merr\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0me\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0murl\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0murl\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mtimeout_value\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mread_timeout\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\urllib3\\packages\\six.py\u001b[0m in \u001b[0;36mraise_from\u001b[1;34m(value, from_value)\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\urllib3\\connectionpool.py\u001b[0m in \u001b[0;36m_make_request\u001b[1;34m(self, conn, method, url, timeout, chunked, **httplib_request_kw)\u001b[0m\n\u001b[0;32m    414\u001b[0m                 \u001b[1;31m# Python 3\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    415\u001b[0m                 \u001b[1;32mtry\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 416\u001b[1;33m                     \u001b[0mhttplib_response\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mconn\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mgetresponse\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    417\u001b[0m                 \u001b[1;32mexcept\u001b[0m \u001b[0mBaseException\u001b[0m \u001b[1;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    418\u001b[0m                     \u001b[1;31m# Remove the TypeError from the exception chain in\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\http\\client.py\u001b[0m in \u001b[0;36mgetresponse\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m   1342\u001b[0m         \u001b[1;32mtry\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1343\u001b[0m             \u001b[1;32mtry\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m-> 1344\u001b[1;33m                 \u001b[0mresponse\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mbegin\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m   1345\u001b[0m             \u001b[1;32mexcept\u001b[0m \u001b[0mConnectionError\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1346\u001b[0m                 \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mclose\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\http\\client.py\u001b[0m in \u001b[0;36mbegin\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    304\u001b[0m         \u001b[1;31m# read until we get a non-100 response\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    305\u001b[0m         \u001b[1;32mwhile\u001b[0m \u001b[1;32mTrue\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 306\u001b[1;33m             \u001b[0mversion\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mstatus\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mreason\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_read_status\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    307\u001b[0m             \u001b[1;32mif\u001b[0m \u001b[0mstatus\u001b[0m \u001b[1;33m!=\u001b[0m \u001b[0mCONTINUE\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    308\u001b[0m                 \u001b[1;32mbreak\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\http\\client.py\u001b[0m in \u001b[0;36m_read_status\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    265\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    266\u001b[0m     \u001b[1;32mdef\u001b[0m \u001b[0m_read_status\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 267\u001b[1;33m         \u001b[0mline\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mstr\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mfp\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mreadline\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0m_MAXLINE\u001b[0m \u001b[1;33m+\u001b[0m \u001b[1;36m1\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;34m\"iso-8859-1\"\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    268\u001b[0m         \u001b[1;32mif\u001b[0m \u001b[0mlen\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mline\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;33m>\u001b[0m \u001b[0m_MAXLINE\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    269\u001b[0m             \u001b[1;32mraise\u001b[0m \u001b[0mLineTooLong\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"status line\"\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\socket.py\u001b[0m in \u001b[0;36mreadinto\u001b[1;34m(self, b)\u001b[0m\n\u001b[0;32m    587\u001b[0m         \u001b[1;32mwhile\u001b[0m \u001b[1;32mTrue\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    588\u001b[0m             \u001b[1;32mtry\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 589\u001b[1;33m                 \u001b[1;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_sock\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mrecv_into\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mb\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    590\u001b[0m             \u001b[1;32mexcept\u001b[0m \u001b[0mtimeout\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    591\u001b[0m                 \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_timeout_occurred\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;32mTrue\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\urllib3\\contrib\\pyopenssl.py\u001b[0m in \u001b[0;36mrecv_into\u001b[1;34m(self, *args, **kwargs)\u001b[0m\n\u001b[0;32m    323\u001b[0m                 \u001b[1;32mraise\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    324\u001b[0m         \u001b[1;32mexcept\u001b[0m \u001b[0mOpenSSL\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mSSL\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mWantReadError\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 325\u001b[1;33m             \u001b[1;32mif\u001b[0m \u001b[1;32mnot\u001b[0m \u001b[0mutil\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mwait_for_read\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msocket\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msocket\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mgettimeout\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    326\u001b[0m                 \u001b[1;32mraise\u001b[0m \u001b[0mtimeout\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"The read operation timed out\"\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    327\u001b[0m             \u001b[1;32melse\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\urllib3\\util\\wait.py\u001b[0m in \u001b[0;36mwait_for_read\u001b[1;34m(sock, timeout)\u001b[0m\n\u001b[0;32m    144\u001b[0m     \u001b[0mReturns\u001b[0m \u001b[1;32mTrue\u001b[0m \u001b[1;32mif\u001b[0m \u001b[0mthe\u001b[0m \u001b[0msocket\u001b[0m \u001b[1;32mis\u001b[0m \u001b[0mreadable\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;32mor\u001b[0m \u001b[1;32mFalse\u001b[0m \u001b[1;32mif\u001b[0m \u001b[0mthe\u001b[0m \u001b[0mtimeout\u001b[0m \u001b[0mexpired\u001b[0m\u001b[1;33m.\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    145\u001b[0m     \"\"\"\n\u001b[1;32m--> 146\u001b[1;33m     \u001b[1;32mreturn\u001b[0m \u001b[0mwait_for_socket\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0msock\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mread\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;32mTrue\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mtimeout\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mtimeout\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    147\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    148\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\urllib3\\util\\wait.py\u001b[0m in \u001b[0;36mselect_wait_for_socket\u001b[1;34m(sock, read, write, timeout)\u001b[0m\n\u001b[0;32m     84\u001b[0m     \u001b[1;31m# thing.)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     85\u001b[0m     \u001b[0mfn\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mpartial\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mselect\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mselect\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mrcheck\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mwcheck\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mwcheck\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 86\u001b[1;33m     \u001b[0mrready\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mwready\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mxready\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0m_retry_on_intr\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mfn\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mtimeout\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     87\u001b[0m     \u001b[1;32mreturn\u001b[0m \u001b[0mbool\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mrready\u001b[0m \u001b[1;32mor\u001b[0m \u001b[0mwready\u001b[0m \u001b[1;32mor\u001b[0m \u001b[0mxready\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     88\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\lib\\site-packages\\urllib3\\util\\wait.py\u001b[0m in \u001b[0;36m_retry_on_intr\u001b[1;34m(fn, timeout)\u001b[0m\n\u001b[0;32m     41\u001b[0m     \u001b[1;31m# Modern Python, that retries syscalls by default\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     42\u001b[0m     \u001b[1;32mdef\u001b[0m \u001b[0m_retry_on_intr\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mfn\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mtimeout\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 43\u001b[1;33m         \u001b[1;32mreturn\u001b[0m \u001b[0mfn\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mtimeout\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     44\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     45\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "twitter_stream = Stream(auth, Listener())\n",
    "twitter_stream.filter(track = ['covid', 'covid19','economic pandemic','chinese virus impact', 'lockdown', 'lockdown recession'], languages=['en'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pycountry.countries.get(alpha_2='DE')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "twitter_data_extraction.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}