ipynb/Chapter 1 - Mining Twitter.ipynb

{
 "metadata": {
  "name": ""
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#Mining the Social Web, 2nd Edition\n",
      "\n",
      "##Chapter 1: Mining Twitter: Exploring Trending Topics, Discovering What People Are Talking About, and More\n",
      "\n",
      "This IPython Notebook provides an interactive way to follow along with and explore the numbered examples from [_Mining the Social Web (2nd Edition)_](http://bit.ly/135dHfs). The intent behind this notebook is to reinforce the concepts from the sample code in a fun, convenient, and effective way. This notebook assumes that you are reading along with the book and have the context of the discussion as you work through these exercises.\n",
      "\n",
      "In the somewhat unlikely event that you've somehow stumbled across this notebook outside of its context on GitHub, [you can find the full source code repository here](http://bit.ly/16kGNyb).\n",
      "\n",
      "## Copyright and Licensing\n",
      "\n",
      "You are free to use or adapt this notebook for any purpose you'd like. However, please respect the [Simplified BSD License](https://github.com/ptwobrussell/Mining-the-Social-Web-2nd-Edition/blob/master/LICENSE.txt) that governs its use."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "# Twitter API Access\n",
      "\n",
      "Twitter implements OAuth 1.0A as its standard authentication mechanism, and in order to use it to make requests to Twitter's API, you'll need to go to https://dev.twitter.com/apps and create a sample application. There are four primary identifiers you'll need to note for an OAuth 1.0A workflow: consumer key, consumer secret, access token, and access token secret. Note that you will need an ordinary Twitter account in order to login, create an app, and get these credentials.\n",
      "\n",
      "<img src=\"files/resources/ch01-twitter/images/Twitter-AppCredentials.png\" width=\"600px\">"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "If you are taking advantage of the virtual machine experience for this chapter that is powered by Vagrant, you should just be able to execute the code in this notebook without any worries whatsoever about installing dependencies. If you are running the code from your own development envioronment, however, be advised that these examples in this chapter take advantage of a Python package called [twitter](https://github.com/sixohsix/twitter) to make API calls. You can install this package in a terminal with [pip](https://pypi.python.org/pypi/pip) with the command `pip install twitter`, preferably from within a [Python virtual environment](https://pypi.python.org/pypi/virtualenv). "
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Once installed, you should be able to open up a Python interpreter (or better yet, your [IPython](http://ipython.org/) interpreter) and get rolling."
     ]
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Example 1. Authorizing an application to access Twitter account data"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import twitter\n",
      "\n",
      "# XXX: Go to http://dev.twitter.com/apps/new to create an app and get values\n",
      "# for these credentials, which you'll need to provide in place of these\n",
      "# empty string values that are defined as placeholders.\n",
      "# See https://dev.twitter.com/docs/auth/oauth for more information \n",
      "# on Twitter's OAuth implementation.\n",
      "\n",
      "CONSUMER_KEY = ''\n",
      "CONSUMER_SECRET =''\n",
      "OAUTH_TOKEN = ''\n",
      "OAUTH_TOKEN_SECRET = ''\n",
      "\n",
      "auth = twitter.oauth.OAuth(OAUTH_TOKEN, OAUTH_TOKEN_SECRET,\n",
      "                           CONSUMER_KEY, CONSUMER_SECRET)\n",
      "\n",
      "twitter_api = twitter.Twitter(auth=auth)\n",
      "\n",
      "# Nothing to see by displaying twitter_api except that it's now a\n",
      "# defined variable\n",
      "\n",
      "print twitter_api"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Example 2. Retrieving trends"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# The Yahoo! Where On Earth ID for the entire world is 1.\n",
      "# See https://dev.twitter.com/docs/api/1.1/get/trends/place and\n",
      "# http://developer.yahoo.com/geo/geoplanet/\n",
      "\n",
      "WORLD_WOE_ID = 1\n",
      "US_WOE_ID = 23424977\n",
      "\n",
      "# Prefix ID with the underscore for query string parameterization.\n",
      "# Without the underscore, the twitter package appends the ID value\n",
      "# to the URL itself as a special case keyword argument.\n",
      "\n",
      "world_trends = twitter_api.trends.place(_id=WORLD_WOE_ID)\n",
      "us_trends = twitter_api.trends.place(_id=US_WOE_ID)\n",
      "\n",
      "print world_trends\n",
      "print\n",
      "print us_trends"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Example 3. Displaying API responses as pretty-printed JSON"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import json\n",
      "\n",
      "print json.dumps(world_trends, indent=1)\n",
      "print\n",
      "print json.dumps(us_trends, indent=1)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Example 4. Computing the intersection of two sets of trends"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "world_trends_set = set([trend['name'] \n",
      "                        for trend in world_trends[0]['trends']])\n",
      "\n",
      "us_trends_set = set([trend['name'] \n",
      "                     for trend in us_trends[0]['trends']]) \n",
      "\n",
      "common_trends = world_trends_set.intersection(us_trends_set)\n",
      "\n",
      "print common_trends"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Example 5. Collecting search results"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# XXX: Set this variable to a trending topic, \n",
      "# or anything else for that matter. The example query below\n",
      "# was a trending topic when this content was being developed\n",
      "# and is used throughout the remainder of this chapter.\n",
      "\n",
      "q = '#MentionSomeoneImportantForYou' \n",
      "\n",
      "count = 100\n",
      "\n",
      "# See https://dev.twitter.com/docs/api/1.1/get/search/tweets\n",
      "\n",
      "search_results = twitter_api.search.tweets(q=q, count=count)\n",
      "\n",
      "statuses = search_results['statuses']\n",
      "\n",
      "\n",
      "# Iterate through 5 more batches of results by following the cursor\n",
      "\n",
      "for _ in range(5):\n",
      "    print \"Length of statuses\", len(statuses)\n",
      "    try:\n",
      "        next_results = search_results['search_metadata']['next_results']\n",
      "    except KeyError, e: # No more results when next_results doesn't exist\n",
      "        break\n",
      "        \n",
      "    # Create a dictionary from next_results, which has the following form:\n",
      "    # ?max_id=313519052523986943&q=NCAA&include_entities=1\n",
      "    kwargs = dict([ kv.split('=') for kv in next_results[1:].split(\"&\") ])\n",
      "    \n",
      "    search_results = twitter_api.search.tweets(**kwargs)\n",
      "    statuses += search_results['statuses']\n",
      "\n",
      "# Show one sample search result by slicing the list...\n",
      "print json.dumps(statuses[0], indent=1)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Note: Should you desire to do so, you can load the same set of search results that are illustrated in the text of _Mining the Social Web_ by executing the code below that reads a snapshot of the data and stores it into the same statuses variable as was defined above. Alternatively, you can choose to skip execution of this cell in order to follow along with your own data."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import json\n",
      "statuses = json.loads(open('resources/ch01-twitter/data/MentionSomeoneImportantForYou.json').read())\n",
      "\n",
      "# The result of the list comprehension is a list with only one element that\n",
      "# can be accessed by its index and set to the variable t\n",
      "t = [ status \n",
      "      for status in statuses\n",
      "          if status['id'] == 316948241264549888 ][0]\n",
      "\n",
      "# Explore the variable t to get familiarized with the data structure...\n",
      "\n",
      "print t['retweet_count']\n",
      "print t['retweeted_status']\n",
      "\n",
      "# Can you find the most retweeted tweet in your search results? Try do do it!"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Example 6. Extracting text, screen names, and hashtags from tweets"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "status_texts = [ status['text'] \n",
      "                 for status in statuses ]\n",
      "\n",
      "screen_names = [ user_mention['screen_name'] \n",
      "                 for status in statuses\n",
      "                     for user_mention in status['entities']['user_mentions'] ]\n",
      "\n",
      "hashtags = [ hashtag['text'] \n",
      "             for status in statuses\n",
      "                 for hashtag in status['entities']['hashtags'] ]\n",
      "\n",
      "# Compute a collection of all words from all tweets\n",
      "words = [ w \n",
      "          for t in status_texts \n",
      "              for w in t.split() ]\n",
      "\n",
      "# Explore the first 5 items for each...\n",
      "\n",
      "print json.dumps(status_texts[0:5], indent=1)\n",
      "print json.dumps(screen_names[0:5], indent=1) \n",
      "print json.dumps(hashtags[0:5], indent=1)\n",
      "print json.dumps(words[0:5], indent=1)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Example 7. Creating a basic frequency distribution from the words in tweets"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from collections import Counter\n",
      "\n",
      "for item in [words, screen_names, hashtags]:\n",
      "    c = Counter(item)\n",
      "    print c.most_common()[:10] # top 10\n",
      "    print"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Example 8. Using prettytable to display tuples in a nice tabular format"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from prettytable import PrettyTable\n",
      "\n",
      "for label, data in (('Word', words), \n",
      "                    ('Screen Name', screen_names), \n",
      "                    ('Hashtag', hashtags)):\n",
      "    pt = PrettyTable(field_names=[label, 'Count']) \n",
      "    c = Counter(data)\n",
      "    [ pt.add_row(kv) for kv in c.most_common()[:10] ]\n",
      "    pt.align[label], pt.align['Count'] = 'l', 'r' # Set column alignment\n",
      "    print pt"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Example 9. Calculating lexical diversity for tweets"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# A function for computing lexical diversity\n",
      "def lexical_diversity(tokens):\n",
      "    return 1.0*len(set(tokens))/len(tokens) \n",
      "\n",
      "# A function for computing the average number of words per tweet\n",
      "def average_words(statuses):\n",
      "    total_words = sum([ len(s.split()) for s in statuses ]) \n",
      "    return 1.0*total_words/len(statuses)\n",
      "\n",
      "print lexical_diversity(words)\n",
      "print lexical_diversity(screen_names)\n",
      "print lexical_diversity(hashtags)\n",
      "print average_words(status_texts)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Example 10. Finding the most popular retweets"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "retweets = [\n",
      "            # Store out a tuple of these three values ...\n",
      "            (status['retweet_count'], \n",
      "             status['retweeted_status']['user']['screen_name'],\n",
      "             status['text']) \n",
      "            \n",
      "            # ... for each status ...\n",
      "            for status in statuses \n",
      "            \n",
      "            # ... so long as the status meets this condition.\n",
      "                if status.has_key('retweeted_status')\n",
      "           ]\n",
      "\n",
      "# Slice off the first 5 from the sorted results and display each item in the tuple\n",
      "\n",
      "pt = PrettyTable(field_names=['Count', 'Screen Name', 'Text'])\n",
      "[ pt.add_row(row) for row in sorted(retweets, reverse=True)[:5] ]\n",
      "pt.max_width['Text'] = 50\n",
      "pt.align= 'l'\n",
      "print pt"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Example 11. Looking up users who have retweeted a status"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Get the original tweet id for a tweet from its retweeted_status node \n",
      "# and insert it here in place of the sample value that is provided\n",
      "# from the text of the book\n",
      "\n",
      "_retweets = twitter_api.statuses.retweets(id=317127304981667841)\n",
      "print [r['user']['screen_name'] for r in _retweets]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Example 12. Plotting frequencies of words"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "word_counts = sorted(Counter(words).values(), reverse=True)\n",
      "\n",
      "plt.loglog(word_counts)\n",
      "plt.ylabel(\"Freq\")\n",
      "plt.xlabel(\"Word Rank\")"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Example 13. Generating histograms of words, screen names, and hashtags"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "for label, data in (('Words', words), \n",
      "                    ('Screen Names', screen_names), \n",
      "                    ('Hashtags', hashtags)):\n",
      "\n",
      "    # Build a frequency map for each set of data\n",
      "    # and plot the values\n",
      "    c = Counter(data)\n",
      "    plt.hist(c.values())\n",
      "    \n",
      "    # Add a title and y-label ...\n",
      "    plt.title(label)\n",
      "    plt.ylabel(\"Number of items in bin\")\n",
      "    plt.xlabel(\"Bins (number of times an item appeared)\")\n",
      "    \n",
      "    # ... and display as a new figure\n",
      "    plt.figure()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Example 14. Generating a histogram of retweet counts"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Using underscores while unpacking values in\n",
      "# a tuple is idiomatic for discarding them\n",
      "\n",
      "counts = [count for count, _, _ in retweets]\n",
      "\n",
      "plt.hist(counts)\n",
      "plt.title(\"Retweets\")\n",
      "plt.xlabel('Bins (number of times retweeted)')\n",
      "plt.ylabel('Number of tweets in bin')\n",
      "\n",
      "print counts"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Note: This histogram gives you an idea of how many times tweets are retweeted with the x-axis defining partitions for tweets that have been retweeted some number of times and the y-axis telling you how many tweets fell into each bin. For example, a y-axis value of 5 for the \"15-20 bin\" on the x-axis means that there were 5 tweets that were retweeted between 15 and 20 times.\n",
      "\n",
      "Here's another variation that transforms the data using the (automatically imported from numpy) log function in order to improve the resolution of the plot."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Using underscores while unpacking values in\n",
      "# a tuple is idiomatic for discarding them\n",
      "\n",
      "counts = [count for count, _, _ in retweets]\n",
      "\n",
      "# Taking the log of the *data values* themselves can \n",
      "# often provide quick and valuable insight into the\n",
      "# underlying distribution as well. Try it back on\n",
      "# Example 13 and see if it helps.\n",
      "\n",
      "plt.hist(log(counts))\n",
      "plt.title(\"Retweets\")\n",
      "plt.xlabel('Log[Bins (number of times retweeted)]')\n",
      "plt.ylabel('Log[Number of tweets in bin]')\n",
      "\n",
      "print log(counts)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    }
   ],
   "metadata": {}
  }
 ]
}