Using the Twitter API for Tweet Analysis.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Twitter data\n",
    "\n",
    "## Copyright and Licensing\n",
    "\n",
    "You are free to use or adapt this notebook for any purpose you'd like. However, please respect the [Simplified BSD License](https://github.com/ptwobrussell/Mining-the-Social-Web-2nd-Edition/blob/master/LICENSE.txt) that governs its use."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Twitter API Access\n",
    "\n",
    "Twitter implements OAuth 1.0A as its standard authentication mechanism, and in order to use it to make requests to Twitter's API, you'll need to go to https://dev.twitter.com/apps and create a sample application.\n",
    "\n",
    "Choose any name for your application, write a description and use `http://google.com` for the website.\n",
    "\n",
    "Under **Key and Access Tokens**, there are four primary identifiers you'll need to note for an OAuth 1.0A workflow: \n",
    "* consumer key, \n",
    "* consumer secret, \n",
    "* access token, and \n",
    "* access token secret (Click on Create Access Token to create those).\n",
    "\n",
    "Note that you will need an ordinary Twitter account in order to login, create an app, and get these credentials."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first time you execute the notebook, add all credentials so that you can save them in the `pkl` file, then you can remove the secret keys from the notebook because they will just be loaded from the `pkl` file.\n",
    "\n",
    "The `pkl` file contains sensitive information that can be used to take control of your twitter acccount, **do not share it**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pickle\n",
    "import os"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "if not os.path.exists('secret_twitter_credentials.pkl'):\n",
    "    Twitter={}\n",
    "    Twitter['Consumer Key'] = ''\n",
    "    Twitter['Consumer Secret'] = ''\n",
    "    Twitter['Access Token'] = ''\n",
    "    Twitter['Access Token Secret'] = ''\n",
    "    with open('secret_twitter_credentials.pkl','wb') as f:\n",
    "        pickle.dump(Twitter, f)\n",
    "else:\n",
    "    Twitter=pickle.load(open('secret_twitter_credentials.pkl','rb'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Install the `twitter` package to interface with the Twitter API"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "!pip install twitter"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example 1. Authorizing an application to access Twitter account data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import twitter\n",
    "\n",
    "auth = twitter.oauth.OAuth(Twitter['Access Token'],\n",
    "                           Twitter['Access Token Secret'],\n",
    "                           Twitter['Consumer Key'],\n",
    "                           Twitter['Consumer Secret'])\n",
    "\n",
    "twitter_api = twitter.Twitter(auth=auth)\n",
    "\n",
    "# Nothing to see by displaying twitter_api except that it's now a\n",
    "# defined variable\n",
    "\n",
    "print(twitter_api)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example 2. Retrieving trends"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Twitter identifies locations using the Yahoo! Where On Earth ID.\n",
    "\n",
    "The Yahoo! Where On Earth ID for the entire world is 1.\n",
    "See https://dev.twitter.com/docs/api/1.1/get/trends/place and\n",
    "http://developer.yahoo.com/geo/geoplanet/\n",
    "\n",
    "look at the BOSS placefinder here: https://developer.yahoo.com/boss/placefinder/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "WORLD_WOE_ID = 1\n",
    "US_WOE_ID = 23424977"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Look for the WOEID for [san-diego](http://woeid.rosselliot.co.nz/lookup/san%20diego%20%20ca)\n",
    "\n",
    "You can change it to another location."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "LOCAL_WOE_ID=2487889\n",
    "\n",
    "# Prefix ID with the underscore for query string parameterization.\n",
    "# Without the underscore, the twitter package appends the ID value\n",
    "# to the URL itself as a special case keyword argument.\n",
    "\n",
    "world_trends = twitter_api.trends.place(_id=WORLD_WOE_ID)\n",
    "us_trends = twitter_api.trends.place(_id=US_WOE_ID)\n",
    "local_trends = twitter_api.trends.place(_id=LOCAL_WOE_ID)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "world_trends[:2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "trends=local_trends\n",
    "print(type(trends))\n",
    "print(list(trends[0].keys()))\n",
    "print(trends[0]['trends'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example 3. Displaying API responses as pretty-printed JSON"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "print((json.dumps(us_trends[:2], indent=1)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example 4. Computing the intersection of two sets of trends"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "trends_set = {}\n",
    "trends_set['world'] = set([trend['name'] \n",
    "                        for trend in world_trends[0]['trends']])\n",
    "\n",
    "trends_set['us'] = set([trend['name'] \n",
    "                     for trend in us_trends[0]['trends']]) \n",
    "\n",
    "trends_set['san diego'] = set([trend['name'] \n",
    "                     for trend in local_trends[0]['trends']]) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "for loc in ['world','us','san diego']:\n",
    "    print(('-'*10,loc))\n",
    "    print((','.join(trends_set[loc])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print(( '='*10,'intersection of world and us'))\n",
    "print((trends_set['world'].intersection(trends_set['us'])))\n",
    "\n",
    "print(('='*10,'intersection of us and san-diego'))\n",
    "print((trends_set['san diego'].intersection(trends_set['us'])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example 5. Collecting search results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set the variable `q` to a trending topic, \n",
    "or anything else for that matter. The example query below\n",
    "was a trending topic when this content was being developed\n",
    "and is used throughout the remainder of this chapter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "q = '#MTVAwards' \n",
    "\n",
    "number = 100\n",
    "\n",
    "# See https://dev.twitter.com/docs/api/1.1/get/search/tweets\n",
    "\n",
    "search_results = twitter_api.search.tweets(q=q, count=number)\n",
    "\n",
    "statuses = search_results['statuses']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "len(statuses)\n",
    "print(statuses)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Twitter often returns duplicate results, we can filter them out checking for duplicate texts:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "all_text = []\n",
    "filtered_statuses = []\n",
    "for s in statuses:\n",
    "    if not s[\"text\"] in all_text:\n",
    "        filtered_statuses.append(s)\n",
    "        all_text.append(s[\"text\"])\n",
    "statuses = filtered_statuses     "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "len(statuses)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "[s['text'] for s in search_results['statuses']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Show one sample search result by slicing the list...\n",
    "print(json.dumps(statuses[0], indent=1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# The result of the list comprehension is a list with only one element that\n",
    "# can be accessed by its index and set to the variable t\n",
    "t = statuses[0]\n",
    "#[ status for status in statuses \n",
    "#          if status['id'] == 316948241264549888 ][0]\n",
    "\n",
    "# Explore the variable t to get familiarized with the data structure...\n",
    "\n",
    "print(t['retweet_count'])\n",
    "print(t['retweeted'])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example 6. Extracting text, screen names, and hashtags from tweets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "status_texts = [ status['text'] \n",
    "                 for status in statuses ]\n",
    "\n",
    "screen_names = [ user_mention['screen_name'] \n",
    "                 for status in statuses\n",
    "                     for user_mention in status['entities']['user_mentions'] ]\n",
    "\n",
    "hashtags = [ hashtag['text'] \n",
    "             for status in statuses\n",
    "                 for hashtag in status['entities']['hashtags'] ]\n",
    "\n",
    "# Compute a collection of all words from all tweets\n",
    "words = [ w \n",
    "          for t in status_texts \n",
    "              for w in t.split() ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Explore the first 5 items for each...\n",
    "\n",
    "print(json.dumps(status_texts[0:5], indent=1))\n",
    "print(json.dumps(screen_names[0:5], indent=1)) \n",
    "print(json.dumps(hashtags[0:5], indent=1))\n",
    "print(json.dumps(words[0:5], indent=1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example 7. Creating a basic frequency distribution from the words in tweets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from collections import Counter\n",
    "\n",
    "for item in [words, screen_names, hashtags]:\n",
    "    c = Counter(item)\n",
    "    print(c.most_common()[:10]) # top 10\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example 8. Create a prettyprint function to display tuples in a nice tabular format"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def prettyprint_counts(label, list_of_tuples):\n",
    "    print(\"\\n{:^20} | {:^6}\".format(label, \"Count\"))\n",
    "    print(\"*\"*40)\n",
    "    for k,v in list_of_tuples:\n",
    "        print(\"{:20} | {:>6}\".format(k,v))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "for label, data in (('Word', words), \n",
    "                    ('Screen Name', screen_names), \n",
    "                    ('Hashtag', hashtags)):\n",
    "    \n",
    "    c = Counter(data)\n",
    "    prettyprint_counts(label, c.most_common()[:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example 9. Finding the most popular retweets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "retweets = [\n",
    "            # Store out a tuple of these three values ...\n",
    "            (status['retweet_count'], \n",
    "             status['retweeted_status']['user']['screen_name'],\n",
    "             status['text'].replace(\"\\n\",\"\\\\\")) \n",
    "            \n",
    "            # ... for each status ...\n",
    "            for status in statuses \n",
    "            \n",
    "            # ... so long as the status meets this condition.\n",
    "                if 'retweeted_status' in status\n",
    "           ]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can build another `prettyprint` function to print entire tweets with their retweet count.\n",
    "\n",
    "We also want to split the text of the tweet in up to 3 lines, if needed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "row_template = \"{:^7} | {:^15} | {:50}\"\n",
    "def prettyprint_tweets(list_of_tuples):\n",
    "    print()\n",
    "    print(row_template.format(\"Count\", \"Screen Name\", \"Text\"))\n",
    "    print(\"*\"*60)\n",
    "    for count, screen_name, text in list_of_tuples:\n",
    "        print(row_template.format(count, screen_name, text[:50]))\n",
    "        if len(text) > 50:\n",
    "            print(row_template.format(\"\", \"\", text[50:100]))\n",
    "            if len(text) > 100:\n",
    "                print(row_template.format(\"\", \"\", text[100:]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Slice off the first 5 from the sorted results and display each item in the tuple\n",
    "\n",
    "prettyprint_tweets(sorted(retweets, reverse=True)[:10])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}