Skip to content
Disaster Management using Text mining.
Jupyter Notebook Python
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
automated_posting need to add api keys in twitter_stream.py Oct 21, 2018
scripts
README.md Update README.md Oct 9, 2018
requirements.txt .py successfully tested, just need to fix the twitter 280charecters p… Oct 22, 2018

README.md

Disaster Management using Text Mining from Twitter

Overview

This projects aims at extracting vital information from Twitter related to natural disasters in real-time. This can be achieved by following the given pipeline :

  • Collecting the tweets using Twitter’s Developer API.
  • Pre-processing the data. (spelling corrections, splitting joint hashtags, emoticons)
  • Feature extraction and selection (named entities- location, organization, money, temporal expressions using NLTK and additional concepts- earthquake, richter scale or magnitude using tf-idf)
  • Training the model using neural network and classifying disaster related tweets.
  • Developing a multi-class classifier to identify and remove less informative tweets such as prayers and sentiments (using NLTK or SpaCy).
  • Developing a real-time text summarization model on the situational tweets to have brief useful information related to the ongoing disaster crisis.

Goals and Motivations

  • During any natural disaster, people related to it directly or indirectly generate tremendous information via tweets. But due to the high velocity and massive amount of data on Twitter, it is overwhelming for a rescue operation to comb through. So we create a concise summary from real-time tweets related to any natural disaster.
  • Our aim is to generate summary that accurately represents the situation regarding the area affected by disaster and will include information related to relief, loss of life, injured lives, helpline contact numbers, volunteers, charity organisations etc.
  • Our aim also includes to develop an application for civilians to provide services like notifications (live updates), sos features, real-time status of a relatives or friends etc.

Specifications

Since Twitter API and AZURE’s cloud support allow only limited number of tweets, we plan to use open source datasets available on CRISIS NLP. The available datasets are already annotated and have classified tweets (disaster and others). We expect to get output summary using this data in batches to try and simulate a real time tweet stream.

You can’t perform that action at this time.