Wouldn't be great to use machine learning and artificial intelligence to help our neighbours, our friends who might be in danger? Who knows someone might be able to help. The web may be chaotic, yet contains significant information for those around us that might need our help. Machine learning can help us structure this data and hopefully make it more useful.
In this instance we scrape data from twitter including hashtags regrading Hurricane Harvey and classifiy/order tweets with serious or negative intent. We built this classification model using H2o's GBM and sklearn's tf-idf on a sample of the sentiment140 dataset.
Install python ml libraries
pip install sklearn pip install numpy pip install pandas pip install scipy
pip install requests pip install tabulate pip install scikit-learn pip install colorama pip install future pip uninstall h2o pip install http://h2o-release.s3.amazonaws.com/h2o/master/4010/Python/h2o-126.96.36.19910-py2.py3-none-any.whl
pip install TwitterSearch
set up a twitter app
- Create a Twitter user account if you do not already have one.
- Go to https://apps.twitter.com/ and log in with your Twitter user account. This step gives you a Twitter dev account under the same name as your user account.
- Click Create New App
- Fill out the form, agree to the terms, and click Create your Twitter application
- In the next page, click on Keys and Access Tokens tab, and copy your API key and API secret. Scroll down and click Create my access token, and copy your Access token and Access token secret.
git clone https://github.com/h2oai/social_ml.git
houston_hurricane.py add your twitter info in :
consumer_key = 'your consumer_key', consumer_secret = 'your consumer_secret', access_token = 'your access_token', access_token_secret = 'your access_token_secret'
consider changing your search terms/tags in
sets_of_keywords=[ ['Houston', '#hurricane','#HoustonStrong','Harvey','help'], ['Houston','flood','help'], ['texas','Harvey' ,'help' ], ... ... ]
the produced tweets_for_hurricane_houston.csv will save have 3 comman separated fields [url, date, tweet]
Then run model_sentiment.py
This will build a classifier based on the sentiment data (sentiment_m140_.csv) and classify the tweets (low score= severe comment, high score= positive comment). the ranked results will be placed in ranked_tweets.csv . The ranked_houston_tweets.xls was created manually from ranked_tweets.csv to make the output more clear
|Thu Aug 31 2017||Woke up feeling sad this morning for our neighbors in #texas. They need our help: https://t.co/oNJgtr6uZL||14.0%|
|Wed Aug 30 2017||RT @BrettFOX46: This is so sad! The storm may be out of Houston but people still need help across the Gulf Coast! https://t.co/Pia1M3k2QW||16.9%|