Social ML

Wouldn't be great to use machine learning and artificial intelligence to help our neighbours, our friends who might be in danger? Who knows someone might be able to help. The web may be chaotic, yet contains significant information for those around us that might need our help. Machine learning can help us structure this data and hopefully make it more useful.

Hurricane Harvey

In this instance we scrape data from twitter including hashtags regrading Hurricane Harvey and classifiy/order tweets with serious or negative intent. We built this classification model using H2o's GBM and sklearn's tf-idf on a sample of the sentiment140 dataset.

requirements

python 2.7

Install python ml libraries

pip install sklearn
pip install numpy
pip install pandas
pip install scipy

Install h2o

pip install requests
pip install tabulate
pip install scikit-learn
pip install colorama
pip install future
pip uninstall h2o
pip install http://h2o-release.s3.amazonaws.com/h2o/master/4010/Python/h2o-3.15.0.4010-py2.py3-none-any.whl

Install TwitterSearch

pip install TwitterSearch

set up a twitter app

• Create a Twitter user account if you do not already have one.
• Go to https://apps.twitter.com/ and log in with your Twitter user account. This step gives you a Twitter dev account under the same name as your user account.
• Click Create New App
• Fill out the form, agree to the terms, and click Create your Twitter application
• In the next page, click on Keys and Access Tokens tab, and copy your API key and API secret. Scroll down and click Create my access token, and copy your Access token and Access token secret.

source here

Run

git clone https://github.com/h2oai/social_ml.git

in houston_hurricane.py add your twitter info in :

consumer_key = 'your consumer_key',
consumer_secret = 'your consumer_secret',
access_token = 'your access_token',
access_token_secret = 'your access_token_secret'

consider changing your search terms/tags in

sets_of_keywords=[
        ['Houston', '#hurricane','#HoustonStrong','Harvey','help'],
		['Houston','flood','help'],
        ['texas','Harvey' ,'help' ],
		...
		...
		]

run houston_hurricane.py

python houston_hurricane.py

the produced tweets_for_hurricane_houston.csv will save have 3 comman separated fields [url, date, tweet]

Then run model_sentiment.py

python model_sentiment.py

This will build a classifier based on the sentiment data (sentiment_m140_.csv) and classify the tweets (low score= severe comment, high score= positive comment). the ranked results will be placed in ranked_tweets.csv . The ranked_houston_tweets.xls was created manually from ranked_tweets.csv to make the output more clear

Sample tweets

date	tweet	severity
Thu Aug 31 2017	Woke up feeling sad this morning for our neighbors in #texas. They need our help: https://t.co/oNJgtr6uZL	14.0%
Wed Aug 30 2017	RT @BrettFOX46: This is so sad! The storm may be out of Houston but people still need help across the Gulf Coast! https://t.co/Pia1M3k2QW	16.9%