No description, website, or topics provided.
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Failed to load latest commit information. Update Aug 31, 2017
formated_results_irma.xlsx updated results for Irma (Florida) Sep 10, 2017
tweets_for_hurricane_houston.csv Social ML for Houston's Hurricane Harvey Aug 31, 2017

Social ML

Wouldn't be great to use machine learning and artificial intelligence to help our neighbours, our friends who might be in danger? Who knows someone might be able to help. The web may be chaotic, yet contains significant information for those around us that might need our help. Machine learning can help us structure this data and hopefully make it more useful.

Hurricane Harvey

In this instance we scrape data from twitter including hashtags regrading Hurricane Harvey and classifiy/order tweets with serious or negative intent. We built this classification model using H2o's GBM and sklearn's tf-idf on a sample of the sentiment140 dataset.


python 2.7

Install python ml libraries

pip install sklearn
pip install numpy
pip install pandas
pip install scipy

Install h2o

pip install requests
pip install tabulate
pip install scikit-learn
pip install colorama
pip install future
pip uninstall h2o
pip install

Install TwitterSearch

pip install TwitterSearch

set up a twitter app

  • Create a Twitter user account if you do not already have one.
  • Go to and log in with your Twitter user account. This step gives you a Twitter dev account under the same name as your user account.
  • Click Create New App
  • Fill out the form, agree to the terms, and click Create your Twitter application
  • In the next page, click on Keys and Access Tokens tab, and copy your API key and API secret. Scroll down and click Create my access token, and copy your Access token and Access token secret.

source here


git clone

in add your twitter info in :

consumer_key = 'your consumer_key',
consumer_secret = 'your consumer_secret',
access_token = 'your access_token',
access_token_secret = 'your access_token_secret'

consider changing your search terms/tags in

        ['Houston', '#hurricane','#HoustonStrong','Harvey','help'],
        ['texas','Harvey' ,'help' ],



the produced tweets_for_hurricane_houston.csv will save have 3 comman separated fields [url, date, tweet]

Then run


This will build a classifier based on the sentiment data (sentiment_m140_.csv) and classify the tweets (low score= severe comment, high score= positive comment). the ranked results will be placed in ranked_tweets.csv . The ranked_houston_tweets.xls was created manually from ranked_tweets.csv to make the output more clear

Sample tweets

date tweet severity
Thu Aug 31 2017 Woke up feeling sad this morning for our neighbors in #texas. They need our help: 14.0%
Wed Aug 30 2017 RT @BrettFOX46: This is so sad! The storm may be out of Houston but people still need help across the Gulf Coast! 16.9%