Numbered Examples

Mining the Social Web, 2nd Edition - All Numbered Examples

This page provides a convenient catalog of all numbered examples that are included in the IPython Notebooks as part of this GitHub repository. Be sure to visit http://MiningTheSocialWeb.com for more updates, example code, and long-form content that didn't make it into the book...

Chapter 1 - Mining Twitter

• Example 1. Authorizing an application to access Twitter account data
• Example 2. Retrieving trends
• Example 3. Displaying API responses as pretty-printed JSON
• Example 4. Computing the intersection of two sets of trends
• Example 5. Collecting search results
• Example 6. Extracting text, screen names, and hashtags from tweets
• Example 7. Creating a basic frequency distribution from the words in tweets
• Example 8. Using prettytable to display tuples in a nice tabular format
• Example 9. Calculating lexical diversity for tweets
• Example 10. Finding the most popular retweets
• Example 11. Looking up users who have retweeted a status
• Example 12. Plotting frequencies of words
• Example 13. Generating histograms of words, screen names, and hashtags
• Example 14. Generating a histogram of retweet counts

Chapter 2 - Mining Facebook

• Example 1. Making Graph API requests over HTTP
• Example 2. Querying the Graph API with Python
• Example 3. Results for a Graph API query for Mining the Social Web
• Example 4. Querying the Graph API for Open Graph objects by their URLs
• Example 5. Comparing likes between Coke and Pepsi fan pages
• Example 6. Querying a page for its "feed" and "links" connections
• Example 7. Querying for all of your friends' likes
• Example 8. Calculating the most popular likes among your friends
• Example 9. Calculating the most popular categories for likes among your friends
• Example 10. Calculating the number of likes for each friend and sorting by frequency
• Example 11. Finding common likes between an ego and its friendships in a social network
• Example 12. Calculating the friends most similar to an ego in a social network
• Example 13. Constructing a graph of mutual friendships
• Example 14. Finding and analyzing cliques in a graph of mutual friendships
• Example 15. Serializing a NetworkX graph to a file for consumption by D3
• Example 16. Visualizing a mutual friendship graph with D3

Chapter 3 - Mining LinkedIn

• Example 1. Using LinkedIn OAuth credentials to receive an access token suitable for development and accessing your own data
• Example 2. Retrieving your LinkedIn connections and storing them to disk
• Example 3. Pretty-printing your LinkedIn connections' data
• Example 4. Displaying job position history for your profile and a connection's profile
• Example 5. Using field selector syntax to request additional details for APIs
• Example 6. Simple normalization of company suffixes from address book data
• Example 7. Standardizing common job titles and computing their frequencies
• Example 8. Geocoding locations with Microsoft Bing
• Example 9. Geocoding locations of LinkedIn connections with Microsoft Bing
• Example 10. Parsing out states from Bing geocoder results using a regular expression
• Example 11. Using NLTK to compute bigrams
• Example 12. Clustering job titles using a greedy heuristic
• Example 13. Clustering your LinkedIn professional network based upon the locations of your connections and emitting KML output for visualization with Google Earth

Chapter 4 - Mining Google+

• Example 1. Searching for a person with the Google+ API
• Example 2. Displaying Google+ avatars in IPython Notebook provides a quick way to disambiguate the search results and discover the person you are looking for
• Example 3. Fetching recent activities for a particular Google+ user
• Example 4. Cleaning HTML in Google+ content by stripping out HTML tags and converting HTML entities back to plain-text representations
• Example 5. Looping over multiple pages of Google+ activities and distilling clean text from notes
• Example 6. Sample data structures used in illustrations for the rest of this chapter
• Example 7. Running TF-IDF on sample data
• Example 8. Exploring Google+ data with NLTK
• Example 9. Querying Google+ data with TF-IDF
• Example 10. Finding similar documents using cosine similarity
• Example 11. Using NLTK to compute bigrams and collocations for a sentence
• Example 12. Using NLTK to compute collocations in a similar manner to the nltk.Text.collocations demo functionality

Chapter 5 - Mining Web Pages

• Example 1. Using boilerpipe to extract the text from a web page
• Example 2. Using feedparser to extract the text (and other fields) from an RSS or Atom feed
• Example 3. Pseudocode for a breadth-first search
• Example 4. Harvesting blog data by parsing feeds
• Example 5. Using NLTK’s NLP tools to process human language in blog data
• Example 6. A document summarization algorithm based principally upon sentence detection and frequency analysis within sentences
• Example 7. Visualizing document summarization results with HTML output
• Example 8. Extracting entities from a text with NLTK
• Example 9. Discovering interactions between entities
• Example 10. Visualizing interactions between entities with HTML output

Chapter 6 - Mining Mailboxes

• Example 1. Converting a toy mailbox to JSON
• Example 2. Converting the Enron corpus to a standardized mbox format
• Example 3. Converting an mbox to a JSON structure suitable for import into MongoDB
• Example 4. Getting the options for the mongoimport command from IPython Notebook
• Example 5. Using mongoimport to load data into MongoDB from IPython Notebook
• Example 6. Simulating a MongoDB shell that you can run from within IPython Notebook
• Example 7. Using PyMongo to access MongoDB from Python
• Example 8. Querying MongoDB by date/time range
• Example 9. Enumerating senders and receivers of messages
• Example 10. Analyzing senders and receivers with set operations
• Example 11. Finding senders and receivers of messages who were Enron employees
• Example 12. Counting sent/received messages for particular email addresses
• Example 13. Using MongoDB's data aggregation framework
• Example 14. Creating a text index on MongoDB documents with PyMongo
• Example 15. Aggregate querying for counts of messages by date/time range
• Example 16. Rendering time series results as a nicely displayed table
• Example 17. Connecting to Gmail with Xoauth
• Example 18. Query your Gmail inbox and store the results as JSON

Chapter 7 - Mining GitHub

• Example 1. Programmatically obtaining a personal API access token for accessing GitHub's API
• Example 2. Making direct HTTP requests to GitHub's API
• Example 3. Using PyGithub to query for stargazers of a particular repository
• Example 4. Constructing a trivial property graph
• Example 5. Constructing an ego graph of a repository and its stargazers
• Example 6. Introducing some handy graph operations
• Example 7. Calculating degree, betweenness, and closeness centrality measures on the Krackhardt kite graph
• Example 8. Adding additional interest edges to the graph through the inclusion of "follows" edges
• Example 9. Exploring the updated graph's "follows" edges
• Example 10. Snapshotting (pickling) the graph's state to disk
• Example 11. Applying centrality measures to the interest graph
• Example 12. Adding starred repositories to the graph
• Example 13. Exploring the graph after updates with additional starred repositories
• Example 14. Updating the graph to include nodes for programming languages
• Example 15. Sample queries for the final graph
• Example 16. Graph visualization of the social network for the original interest graph

Chapter 8 - Mining the Semantically Marked-Up Web

• Example 1. Extracting geo-microformatted data from a Wikipedia page
• Example 2. Displaying geo-microformats with Google Maps in IPython Notebook
• Example 3. Extracting hRecipe data from a web page
• Example 4. Parsing hReview-aggregate microformat data for a recipe

Chapter 9 - Twitter Cookbook

• Example 1. Accessing Twitter's API for development purposes
• Example 2. Doing the OAuth dance to access Twitter's API for production purposes
• Example 3. Discovering the trending topics
• Example 4. Searching for tweets
• Example 5. Constructing convenient function calls
• Example 6. Saving and restoring JSON data with flat-text files
• Example 7. Saving and accessing JSON data with MongoDB
• Example 8. Sampling the Twitter firehose with the Streaming API
• Example 9. Collecting time-series data
• Example 10. Extracting tweet entities
• Example 11. Finding the most popular tweets in a collection of tweets
• Example 12. Finding the most popular tweet entities in a collection of tweets
• Example 13. Tabulating frequency analysis
• Example 14. Finding users who have retweeted a status
• Example 15. Extracting a retweet's attribution
• Example 16. Making robust Twitter requests
• Example 17. Resolving user profile information
• Example 18. Extracting tweet entities from arbitrary text
• Example 19. Getting all friends or followers for a user
• Example 20. Analyzing a user's friends and followers
• Example 21. Harvesting a user's tweets
• Example 22. Crawling a friendship graph
• Example 23. Analyzing tweet content
• Example 24. Summarizing link targets
• Example 25. Analyzing a user's favorite tweets