01_TwitterExtraction.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Twitter extraction\n",
    "\n",
    "This notebook aims to retrieve tweets, clean them and compute a sentiment in order to observe a correlation between crypto currencies and tweets' sentiments. The following steps are executed in this notebook :\n",
    "\n",
    "- Retrieve tweets with Twython API (Twitter API wrapper for python)\n",
    "- Extract the wanted data (tweet's text, #followers, #likes, etc.)\n",
    "- Clean the textual data (remove unnecessary elements like media, websites link, pseudos, ...)\n",
    "- Compute for each tweet a sentiment score with Vader (named compound) and a score linked to the popularity of the tweet and its compound\n",
    "\n",
    "This notebook is written using Python 3.6.\n",
    "\n",
    "## Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the currency\n",
    "#CURRENCY = \"zilliqa\"\n",
    "#CURRENCY_SYMBOL = \"ZIL\"\n",
    "#CURRENCY = \"nexo\"\n",
    "#CURRENCY_SYMBOL = \"NEXO\"\n",
    "CURRENCY = \"bitcoin\"\n",
    "CURRENCY_SYMBOL = \"BTC\"\n",
    "\n",
    "## personal config\n",
    "TWEETS_FOLDER    = f\"data/crypto/{CURRENCY_SYMBOL}\" # Relative path to historical data\n",
    "SEP_CHAR         = '~' # character seperating dates from and to in filename\n",
    "ENVS             = ['CRYPTO', 'LINE_COUNT', 'MOST_RECENT_FILE', 'MOST_RECENT_ID'] # Stored in var.csv\n",
    "MAX_ROW_PER_FILE = 20000 # Each file storing data has a maximum amount of rows\n",
    "\n",
    "#tweets_raw_file = f'data/twitter/{CURRENCY_SYMBOL}/{CURRENCY}_tweets_raw.csv'\n",
    "#tweets_clean_file = f'data/twitter/{CURRENCY_SYMBOL}/{CURRENCY}_tweets_clean.csv'\n",
    "query = f'#{CURRENCY} OR #{CURRENCY_SYMBOL}' ####TODO PUT BACK  OR {CURRENCY} OR ${CURRENCY} OR ${CURRENCY_SYMBOL}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Retrieve the tweets from Twitter API"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1 Import Twython\n",
    "We use the *twython* package as my Python interface with the Twitter API: https://twython.readthedocs.io/en/latest/usage/starting_out.html\n",
    "\n",
    "The twython package must be installed using *pip install twython* from the command line."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "from twython import Twython"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2 OAuth2 Authentication (*app* authentication)\n",
    "Here we use the method *OAuth2* along with the Twithon library to authenticate on the twitter API.\n",
    "\n",
    "OAuth1 will give you *user* access to the API, whereas OAuth2 will give the *app* access. For academic use the rate limits are generally better for *OAuth2* (app) authentication, with a few exceptions. For a chart showing the API limits for user and app authentication for the various parts of the Twitter API, see this chart: https://dev.twitter.com/rest/public/rate-limits\n",
    "\n",
    "Running the code block below shows that we now have a rate limit of 450 API calls. This means we can make 450 different calls to the API within the current 15-minute window. With the search API we can access 100 tweets per call. This means that, if we were downloading tweets with a specific hashtag, such as *#arnova16*, we could download 450 $\\times$ 100 or 45,000 tweets per window. This is much better than the 18,000 tweets we can access using the OAuth1 or user authentication."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'/search/tweets': {'limit': 450, 'remaining': 450, 'reset': 1527859898}}"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "APP_KEY = 'mPQKoRwd2Pb9qpQyQmyG5s8KR'\n",
    "APP_SECRET = 'HLvIhusvfzDLKaRXY8CnZGP143kp3E3f2KqQBIEMfVL5mOxZjq'\n",
    "twitter = Twython(APP_KEY, APP_SECRET, oauth_version=2)\n",
    "ACCESS_TOKEN = twitter.obtain_access_token()\n",
    "twitter = Twython(APP_KEY, access_token=ACCESS_TOKEN)\n",
    "twitter.get_application_rate_limit_status()['resources']['search']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3 Query the twitter API\n",
    "Here we query the twitter API to get the latest tweets about bitcoin. Then we transform it to store only the useful data inside a Pandas Dataframe.\n",
    "\n",
    "The following fields are retrieved from the response:\n",
    "\n",
    "- **id** (int) : unique identifier of the tweet\n",
    "- **text** (string) : UTF-8 textual content of the tweet, max 140 chars\n",
    "- user\n",
    "  - **name** (string) : twitter's pseudo of the user\n",
    "  - **followers_count** (int) : Number of followers the user has\n",
    "- **retweet_count** (int) : Number of times the tweet has been retweeted\n",
    "- **favorite_count** (int) : Number of likes\n",
    "- **created_at** (datetime) : creation date and time of the tweet\n",
    "\n",
    "Also, we wanted to retrieve the following fields but it is not possible with the standard free API, Enteprise or premium is needed (https://developer.twitter.com/en/docs/tweets/data-dictionary/overview/tweet-object.html):\n",
    "\n",
    "- reply_count (int) : Number of times the Tweet has been replied to\n",
    "\n",
    "The pandas package must be installed using *pip install pandas* from the command line.\n",
    "\n",
    "We used the search opertators that are explained here (https://lifehacker.com/search-twitter-more-efficiently-with-these-search-opera-1598165519) to not only search by hashtag but also the tweets that contain the currency name or that have the hashtag with the currency's abreviation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "from time import sleep\n",
    "import json\n",
    "import pandas as pd\n",
    "import io\n",
    "from tqdm import tqdm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "data/twitter/BTC/2018-05-29 12-20-53~2018-05-29 12-26-11.csv\n"
     ]
    }
   ],
   "source": [
    "NUMBER_OF_QUERIES = 450\n",
    "data = {\"statuses\": []}\n",
    "next_id = \"998511795781361665\"\n",
    "with open(tweets_raw_file,\"a+\", encoding='utf-8') as f:\n",
    "    if not next_id:\n",
    "        f.write(\"ID,Text,UserName,UserFollowerCount,RetweetCount,Likes,CreatedAt\\n\")\n",
    "    while(True):\n",
    "        twitter = Twython(APP_KEY, access_token=ACCESS_TOKEN)\n",
    "        last_size = 0\n",
    "        for i in tqdm(range(NUMBER_OF_QUERIES)):\n",
    "            if not next_id:\n",
    "                data = twitter.search(q=query, lang='en', result_type='recent', count=\"100\") # Use since_id for tweets after id\n",
    "            else:\n",
    "                data[\"statuses\"].extend(twitter.search(q=query, lang='en', result_type='mixed', count=\"100\", max_id=next_id)[\"statuses\"])\n",
    "            if len(data[\"statuses\"]) > 1:\n",
    "                next_id = data[\"statuses\"][len(data[\"statuses\"]) - 1]['id']\n",
    "            if last_size + 1 == len(data[\"statuses\"]):\n",
    "                break\n",
    "            else:\n",
    "                last_size = len(data[\"statuses\"])\n",
    "\n",
    "        print('Retrieved {0}, waiting for 15 minutes until next queries'.format(len(data[\"statuses\"])))\n",
    "        d = pd.DataFrame([[s[\"id\"], s[\"text\"].replace('\\n','').replace('\\r',''), s[\"user\"][\"name\"], s[\"user\"][\"followers_count\"], s[\"retweet_count\"], s[\"favorite_count\"], s[\"created_at\"]] for s in data[\"statuses\"]], columns=('ID', 'Text', 'UserName', \"UserFollowerCount\", 'RetweetCount', 'Likes', \"CreatedAt\"))\n",
    "        d.to_csv(f, mode='a', encoding='utf-8',index=False,header=False)\n",
    "        if last_size + 1 == len(data[\"statuses\"]):\n",
    "            print('No more new tweets, stopping...')\n",
    "            break\n",
    "        data[\"statuses\"] = []\n",
    "        \n",
    "        sleep(910)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preprocessing\n",
    "\n",
    "Now we will cleanup the data.\n",
    "\n",
    "We already filtered tweets in english in the call to the Twitter API.\n",
    "We will now filter links, @Pseudo, images, videos, unhashtag #happy -> happy.\n",
    "\n",
    "We won't transform to lower case because Vader take capital letters into consideration to emphasize sentiments.\n",
    "\n",
    "You must install `pip install tqdm`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 1120556/1120556 [17:30:30<00:00, 17.78it/s]  \n"
     ]
    }
   ],
   "source": [
    "import re # regular expressions\n",
    "from tqdm import tnrange, tqdm_notebook, tqdm\n",
    "\n",
    "d = pd.read_csv(tweets_raw_file)\n",
    "for i,s in enumerate(tqdm(d['Text'])):\n",
    "    text = d.loc[i, 'Text']\n",
    "    text = text.replace(\"#\", \"\")\n",
    "    text = re.sub('https?://(?:[-\\w.]|(?:%[\\da-fA-F]{2}))+', '', text, flags=re.MULTILINE)\n",
    "    text = re.sub('@\\\\w+ *', '', text, flags=re.MULTILINE)\n",
    "    d.loc[i, 'Text'] = text\n",
    "f = open(tweets_clean_file, 'a+', encoding='utf-8')\n",
    "d.to_csv(f, header=True, encoding='utf-8',index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>Text</th>\n",
       "      <th>UserName</th>\n",
       "      <th>UserFollowerCount</th>\n",
       "      <th>RetweetCount</th>\n",
       "      <th>Likes</th>\n",
       "      <th>CreatedAt</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1001439557504692224</td>\n",
       "      <td>RT : $btc bullish continuation</td>\n",
       "      <td>Tamekia Gay</td>\n",
       "      <td>0</td>\n",
       "      <td>548</td>\n",
       "      <td>0</td>\n",
       "      <td>Tue May 29 12:26:11 +0000 2018</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1001439550860873728</td>\n",
       "      <td>RT : The Woman Powering Bitcoin - Interview Wi...</td>\n",
       "      <td>PiranhadoAmor</td>\n",
       "      <td>53</td>\n",
       "      <td>375</td>\n",
       "      <td>0</td>\n",
       "      <td>Tue May 29 12:26:09 +0000 2018</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1001439550298828801</td>\n",
       "      <td>RT : Company Creates Decentralized VRWorld Whe...</td>\n",
       "      <td>🇻🇪 José Subero</td>\n",
       "      <td>2635</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>Tue May 29 12:26:09 +0000 2018</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1001439549178961920</td>\n",
       "      <td>RT : Zynga poker chips are a fun currency in T...</td>\n",
       "      <td>Bingaman Sarah</td>\n",
       "      <td>1</td>\n",
       "      <td>890</td>\n",
       "      <td>0</td>\n",
       "      <td>Tue May 29 12:26:09 +0000 2018</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1001439548877017088</td>\n",
       "      <td>RT Bitcoin: Bitcoin Cash is BitcoinVideo by Cr...</td>\n",
       "      <td>Crypto Br4in</td>\n",
       "      <td>9562</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>Tue May 29 12:26:08 +0000 2018</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    ID                                               Text  \\\n",
       "0  1001439557504692224                     RT : $btc bullish continuation   \n",
       "1  1001439550860873728  RT : The Woman Powering Bitcoin - Interview Wi...   \n",
       "2  1001439550298828801  RT : Company Creates Decentralized VRWorld Whe...   \n",
       "3  1001439549178961920  RT : Zynga poker chips are a fun currency in T...   \n",
       "4  1001439548877017088  RT Bitcoin: Bitcoin Cash is BitcoinVideo by Cr...   \n",
       "\n",
       "         UserName  UserFollowerCount  RetweetCount  Likes  \\\n",
       "0     Tamekia Gay                  0           548      0   \n",
       "1   PiranhadoAmor                 53           375      0   \n",
       "2  🇻🇪 José Subero               2635             4      0   \n",
       "3  Bingaman Sarah                  1           890      0   \n",
       "4    Crypto Br4in               9562             0      0   \n",
       "\n",
       "                        CreatedAt  \n",
       "0  Tue May 29 12:26:11 +0000 2018  \n",
       "1  Tue May 29 12:26:09 +0000 2018  \n",
       "2  Tue May 29 12:26:09 +0000 2018  \n",
       "3  Tue May 29 12:26:09 +0000 2018  \n",
       "4  Tue May 29 12:26:08 +0000 2018  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_clean = pd.read_csv(tweets_clean_file)\n",
    "df_clean.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df_clean.min(axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "f.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}