Python package and tools for Twitter data analysis. Contact: SMaPP Lab, NYU.
smappPy is a Python "package" (a module of modules - basically, a structured collection of code) that addresses common tasks for programming with Tweets and analyzing Twitter data.
This includes:
- interfacing with Twitter to find tweets and user data (based on user, keyword, and/or location)
- accessing tweets and other twitter data stored in files or MongoDB databases
- getting information about tweets (contains links/mentions/hashtags/etc, is a retweet, contains location information or not, etc)
- mining tweet collections for interesting features (popular hashtags, links shared, images tweeted, etc)
- doing advanced computations on tweet collections, such as modeling topics or building networks
The package is a WIP (eg: eventually, we will include facebook tools). Existing and future functionality is defined below. Examples of how to use library functions to complete common tasks are also coming.
Contents:
We recommend using the Anaconda Python distribution, which contains almost all other prereqs for our code, and a bunch of useful python packages besides (numpy, scipy, matplotlib, networkx, etc).
We also use the pip package management tool, which is included in Anaconda python:
pip install tweepy
pip install pymongo
Note: this dependency section is not totally complete. Some functions required additional libraries. This section will be updated as dependencies are remembered. (EG: some URL utility functionality requires additional libraries)
keyword_tweets(oauth_file, keyword_list, limit_per_keyword)
user_tweets(oauth_file, userid_list, limit_per_user)
place_tweets(oauth_file, place_list, query, limit_per_location)
georadius_tweets(oauth_file, georadius_list, query, limit_per_location)
These methods query twitter via the REST interface (single-transaction. NOT STREAMING)
Note: calling these methods can incur a rate limit exception, in the case that too many requests have been made to the Twitter API. This is left for the user to handle. An example will be provided of how to do so.
Usage of place_tweets and georadius_tweets:
from smappPy.get_tweets import place_tweets, georadius_tweets
for tweet in place_tweets(api, place_list=["Manchester"], query="Coffee", limit=1):
print(tweet)
for tweet in place_tweets(api, place_list=["Glasgow", "Dublin"], query="Coffee", limit=1):
print(tweet)
for tweet in georadius_tweets(api, georadius_list=[[37.781157,-122.398720,"1mi"]], query="Coffee", limit=1):
print(tweet)
for tweet in georadius_tweets(api, georadius_list=[[32.781830, -96.795860,"1mi"], [37.781157,-122.398720,"1mi"]], query="Coffee", limit=1):
print(tweet)the "query" parameter is optional and can be omitted. If run with a query="Coffee", it will only pull tweets about query="Coffee" from the coordinates and radii you give it. If left blank I assume you will be pulling the top tweets or some sudo-random kinds of tweets.
stream_listener.SimpleFileListener # captures live tweets, stores in txt file
stream_listener.SimpleDBListener # captures live tweets, stores in MongoDB
Can capture "live" tweets based on keyword, user tweeting, and location. (This is a more complex and detailed system than fetching from the REST API. See the code for more.)
Can capture Twitter user data (including account info, picture, tweet numbers, friends, followers, etc)
get_user_data(oauth_file, userid_list) # returns a list of twitter user objects for each ID (if valid)
Note: twitter user object defined here
tweets_from_file(tweetfile) # returns a list of tweet objects (json/dict)
tweets_from_file_IT(tweetfile) # returns an iterator over tweet objects in file (json/dict)
tweets_from_db(server, port, user, password, database, collection, keywords, number)
db_tweets_by_date(server, port, user, password, database, collection, start, end, number)
tweets_to_file(tweets, tweet_file, ...) # stores tweets (in json) in a file
tweets_to_db(server, port, user, password, database, collection, tweets)
Note: for more complicated queries for getting tweets from a MongoDB instance, we recommend you read up on pymongo documentation and tweet structure
Note: When we store tweets to a MongoDB instance provided, we add two fields for easier data access: 'random_number' - a random decimal (0,1), making it easy to get a random sample of tweets; and 'timestamp' - a MongoDB Date object / python datetime object (depending on context), making tweets easier to query by datetime. See Tweet utilities section below.
Note: to use the 'tweets_to_db' function, the given username must have write privileges to the given database.
add_random_to_tweet(tweet) # adds random_number field to tweets
add_timestamp_to_tweet(tweet) # adds timestamp corresponding to 'created_at' to tweet
transform_collection(collection, create_indexes=True) # adds timestamp and random fields to a collection of tweets, and creates indexes on collection for faster access to tweets
is_retweet(tweet) # checks for all types of retweet (RT)
is_official_retweet(tweet) # checks for literal retweet via Twitter's retweet button
is_manual_retweet(tweet) # checks for manual "RT @someuser ..." type of RT
is_partial_retweet(tweet) # checks for manual RTs with additional text (may just be space)
get_user_retweeted(tweet) # returns a tuple (user ID, user screen name). If manual RT, id is None
split_manual_retweet(tweet) # splits a manual Rt into "pre, userRTed, post" text elements
is_MT(tweet) # checks for all cases of MT (modified tweets)
is_initial_MT(tweet) # if the MT is the whole tweet
is_interior_MT(tweet) # if the MT comes with additional tweeter commentary
split_MT(tweet) # splits an interior MT into "pre, userMTed, post" elements
remove_entities_from_text(tweet) # returns tweet text string with all entity strings removed
contains_mention(tweet) # returns True if tweet contains a mention
num_mentions(tweet) # returns number of mentions in tweet
get_users_mentioned(tweet) # returns a list of users mentioned in the tweet
contains_hashtag(tweet) # (same as for mentions)
num_hashtag(tweet)
get_hashtags(tweet)
contains_link(tweet) # returns True if tweet contains a link (url or media)
num_links(tweet)
contains_image(tweet) # returns True if tweet contains an image post (media)
get_image_urls(tweet) # returns a list of all image URLs contained in the tweet
save_web_image(url, filename) # given an image's URL, saves that image to filename
get_image_occurrences(tweets) # given an iterable of tweets, returns a dictionary of image urls and the number of times the occur in the tweet set.
urllib_get_html(url) # Uses urllib to download a webpage, returns the page's html as a string
requests_get_html(url) # (same as above, but with requests module instead of urllib)
get_html_text(html) # takes html of a webpage, returns clean plain text of website contents (good for articles!)
clean_whitespace(string, replacement=" ") # replaces all whitespace in a string with, by default, a single space
Note: get_html_text function works via readability and nltk
remove_punctuation(text) # translates most punctuation to single spaces
csv_safe(text) # replaces csv-breaking characters (comma, tab, and newline) with placeholders
translate_whitespace(text) # translates all whitespace chars to single spaces
translate_shorthand(text) # translate common shorthand to long form (eg: w/ to with)
translate_acronyms(text) # translates some common acronyms to long form.
translate_unicode(text) # translates unicode that breaks some software into ASCII
translate_contractions(text) # translates all unambiguous english language contractions into multiple words
Note: all translation and removal functions can (and should) be customized in the translation tables at the top of the text_clean.py code
language # includes python definitions for all Twitter-supported languages
date # date functions to translate twitter date strings to Python datetime objects
json_util # utilities for reading/writing JSON and MongoDB "bson" files
oauth # tools for reading and verifying oauth json files for Twitter authentication
autoRT # a tool to autoretweet any of a set of users' tweets during certain timeframes (to show your rowdy students who are tweeting during class that your twitter game is muy strong, and better than theirs)
build_retweet_network(tweets, internal_only=True) # given a collection of tweets, constructs and returns retweet network
display_retweet_network(network, outfile="default") # creates a basic display (via matplotlib) of the RT network
export_network(outfile?) # exports a network to Gephi format (for further processing and vis.)
Note: all network functionality is via the networkx package (included in Anaconda python).
IN PROGRESS
utilities.get_topic_string(model, topic_id, top_n) # Returns a string representing a topic from given model
utilities.get_short_topic_string(%) # Same, different representation
utilities.get_topic_strings(model, num_topics, top_n, ordered) # Returns a list of representative topic strings
utilities.get_doc_topic_strings(corpus, model. num_docs, topic_threshold, min_topics, topic_words)
# Prints documents in a corpus, the document's topics, and the topic's words
# All functions above can also be "print"changed, eg: utilities.print_topic_strings(...)
visualization.topic_barchart(corpus, model, topic_threshold, show, outfile, bar_width, trim)
# Creates, shows, and saves a barchart representing topic occurence in all corpus docs (if topic is >= threshold for that doc)
OAuth Files should be of the format:
[
{
"consumer_key": "YOUR_CONSUMER_KEY",
"consumer_secret": "YOUR_CONSUMER_SECRET",
"acces_token": "YOUR _ACCESS_TOKEN",
"access_token_secret": "YOUR_ACCESS_TOKEN_SECRET"
},
{
"consumer_key": "YOUR_CONSUMER_KEY",
"consumer_secret": "YOUR_CONSUMER_SECRET",
"acces_token": "YOUR _ACCESS_TOKEN",
"access_token_secret": "YOUR_ACCESS_TOKEN_SECRET"
}
]where there can be any number of JSON objects inside the array. The file above could be called "oauth-file.json" for extra clarity.
These two scripts represent our tunneler that creates a tunnel into HPC. To make the tunnel work you need to put a file in your home directory that contains your HPC password and username in the ~/.ssh directory like so:
/.ssh/hpclogin.txt ( is your home directory /User/YOUR_NAME)
that looks like so (username on line one, password on line two):
username password
IN PROGRESS. Basic scripts exist to scrape data from facebook pages.
These topics come from programming lectures given to SMaPP students. See Programmer Group for the full code.
- Plot the number of tweets per minute with co-occuring words "Obama" and "Syria"
- Get a collection of tweets (from DB or twitter), output CSV representation of all tweets with added indicator variables (eg: IsRetweet? HasImage?)
- Get a collection of tweets, go over all and compute aggregate statistics (eg: number of tweets, tweets/user, number of tweeters, tweets per day)
- Access DataScienceToolkit services, measure basic sentiment of tweet. Compute aggregate sentiment (basic measure) of tweets per keyword-topic.
- Plot sentiment-per-day of tweets on a certain topic (again, basic sentiment analysis)
- Collect streaming, real-time tweets by keywords, users, or geolocations - EG, collect all tweets coming from Kyiv (very specific location). Plot them on a map via OpenHeatMap
- Create "networks" from tweets - user retweet network, tweet-retweet network, etc
- Get the top N tweeted images in a certain date range (or any set of tweets). Same for hashtags, links, base-linked domains, etc.
- etc. Any functionality you can put together with these tools and other resources