Deep Learning for Big Crisis Data
This repository will host Python implementation of a number of deep neural networks classifiers for the classification of crisis-related data on Twitter.
python 2.7 numpy, scikit-learn keras, tensorflow or theano backend
Dataset and Pre-process A sample of tweet data (data/sample.csv) is a .csv format with three columns
First, we need to pre-process tweets data: remove urls, special characters, lowercasing… - python data_helpers/preprocess.py data/sample.csv Split pre-processed data (data/sample_prccd.csv) into train, test and dev part. - python data_helpers/split_data.py data/sample_prccd.csv
Training a neural net model
To train a classifier we create a folder containing links to train, test and dev part (data/nn_data)
Folder embeddings/ includes word vector file, we provide our pre-trained crisis word vectors, we also can use Google word embedding here
Folder dnn_scrips/ contains all neural nets models: CNN, RNN_LSTM, MLP…
- bash run_cnn.sh to train a model with different parameters.
See the results and training process in .log file
Note that: if you do not have a GPU, you can run on CPU by change the theano flag to THEANO_FLAGS=mode=FAST_RUN,device=cpu,floatX=float32