The goal is to provide the public with a sequence of functions and files that would allow them in turn, with minimum changes, train test and evaluate their own datasets with state of the art deep learning approaches.
The main structure that we are going to follow is this:
|Datareader
|---->datareader.py
|ModelCreation
|----->|CNN
| |---->daphnetDatasetResultsAndConfiguration
| |---->opportunityDatasetResultsAndConfiguration
| |---->pamap2DatasetResultsAndConfiguration
| |---->cnn1d.py
|----->|DNN
| |---->daphnetDatasetResultsAndConfiguration
| |---->opportunityDatasetResultsAndConfiguration
| |---->pamap2DatasetResultsAndConfiguration
| |---->mlp.py
|----->|RNN
| |---->|FFLSTM
| | |---->daphnetDatasetResultsAndConfiguration
| | |---->opportunityDatasetResultsAndConfiguration
| | |---->pamap2DatasetResultsAndConfiguration
| | |---->fflstm.py
| |---->|BDLSTM
| | |---->daphnetDatasetResultsAndConfiguration
| | |---->opportunityDatasetResultsAndConfiguration
| | |---->pamap2DatasetResultsAndConfiguration
| | |---->bdlstm.py
Below is a small description of the process the practitioner should follow in order to run the code provided or further extend it to suit his/her own purposes.
PREPROCESS DATASETS STAGE
The Datareader folder contains a python script that follows the logic of this github repository : https://github.com/nhammerla/deepHAR
Its main purpose is to read a dataset and save it in a special file format known as hdf5 (https://support.hdfgroup.org/HDF5/), so that
in our main code we could extract the training and testing set of every different dataset just by changing the path and not anything else in the code. More datasets can be added by implementing additional if conditions inside the datareader.py .
The datasets that we are preprocessing are the following :
Opportunity dataset (it can be downloaded from : https://archive.ics.uci.edu/ml/datasets/opportunity+activity+recognition)
Daphnet Gait dataset (it can be downloaded from : https://archive.ics.uci.edu/ml/datasets/Daphnet+Freezing+of+Gait)
PAMAP2 dataset (it can be downloaded from : https://archive.ics.uci.edu/ml/datasets/PAMAP2+Physical+Activity+Monitoring)
Sphere dataset (it can be downloaded from : https://data.bris.ac.uk/data/dataset/8gccwpx47rav19vk8x4xapcog)
The way to do that is by downloading the dataset from the links provided above, adding the datareader.py script inside their folder and running the command in a terminal:
python datareader.py (dataset-three-first-initial-letters)
e.g. for Daphnet gait you would have to type:
python datareader.py dap
Note: For datasets with similar names, e.g. pamap and pamap2, we would use "pam" for pamap and "pa2" for pamap2.
After this stage is complete, a file named daphnet.h5 will have been created containing the training and testing sets that were decided to be used for this dataset in the datareader.py (Note: the training-test split we follow for the Opportunity, Daphnet and PAMAP2 datasets are described in Deep, Convolutional, and Recurrent Models for Human Activity Recognition using Wearables, by Nils Y. Hammerla, Shane Halloran, Thomas Ploetz (https://arxiv.org/abs/1604.08880). The split for the Sphere dataset is proposed by a Sphere researcher and it is files 1-8 are used as train set, and files 9-10 as test set.)
We can later on call the the dataset in our code by fixing the correct path in the line calling the os python function.
e.g. :
path = os.path.join(os.path.expanduser('~'), 'Downloads', 'OpportunityUCIDataset', 'opportunity.h5')
Note: This path function works both for windows and linux. The correct dataset with its train-test split is retrieved by simply typing :
f = h5py.File(path, 'r')
x_train = f.get('train').get('inputs')[()]
y_train = f.get('train').get('targets')[()]
x_test = f.get('test').get('inputs')[()]
y_test = f.get('test').get('targets')[()]
At this point you can run the code contained inside the ModelCreation folder to train a model. This can be done by going inside a specifc model's folder (e.g. CNN) and typing the following command :
python (executable) (three first letters of the dataset)
e.g.In order to train Daphnet Gait with a CNN you would have to type:
python cnn1d.py dap
Models
Initially, in all the following models except from the DNN-MLP, we use the datasets as is, and we split them in 1-5 second window segments as described in https://arxiv.org/abs/1604.08880 with 50% overlap. For the DNN, we flatten the input. E.g. if we have a dataset with 3 measurements x,y,z that contains 1000 samples and has a window size of 23 (ie 23 consecutive measurements, which, depending on the frequency of the measurements, it corresponds to 1-5 seconds of activity), the input dimension for the network will not be (1000,23,3) but it will be (1000,23*3) with the x,y,z having the formation x1,x2,..x23,y1,y2,..y23,z1,z2,...z23.
- Deep Feed Forward Neural Network - Multilayer Perceptron (DNN,MLP)
- Convolutional Neural Network (CNN)
- Long Short Term Memory Recurrent Neural Network (LSTM-RNN)
- Bi-Directional Long Short Term Memory Recurrent Neural Network (BD-LSTM-RNN)
A short description of this network is given here: https://en.wikipedia.org/wiki/Feedforward_neural_network The size of each layer depends on the dataset we use as an input, the smaller datasets contain fewer hidden layers, with fewer neurons each.
The goal was to replicate similar f-scores for the 3 datasets (Opportunity, Pamap2, Daphnet Gait) with the paper described here (https://arxiv.org/abs/1604.08880) so that we can examine the performance of these networks on the Sphere dataset and do some parameter and optimization function exploration.
There is a an if-else section in each algorithm which changes the parameters depending on which dataset (dap,opp,pam,sph) is given. In case another dataset name or a name is not given, it returns an error code. This part of the code could be changed as to include an additional dataset with different parameter configurations.
The network is described here : https://en.wikipedia.org/wiki/Convolutional_neural_network
We follow the same logic described in the paper mentioned above. After the training process both recall and precision is calculated. The confusion matrix is printed as well.
The network is described here : https://en.wikipedia.org/wiki/Long_short-term_memory
Same logic as above applies.
The bi directional concept of an RNN is described here : https://en.wikipedia.org/wiki/Bidirectional_recurrent_neural_networks
Same logic as above applies.
Basic Functions
There is a stucture followed throughout all the different files contained in this repository.
It goes this way:
- Read data from a certain path.
The reading of the data was already described and is an easy process due to datareader.py preprocessing step. - Depending on the dataset, segment the data to specific time windows.
The data segmentation is done via functions implemented called segment_(dataset initials), e.g. segment_opp.
There is also a function called windows that is responsible for the 50% overlap handling. - Write some initial parameters, (training epochs, number of labels/classes, number of features etc) with respect to the dataset.
This is done in each separate script provided, and the actual numbers were either hardcoded based on experimentations or suggestions of the paper of Hammerla et al. (https://arxiv.org/abs/1604.08880)
Other parameters can be retrieved by the input itself (e.g. number of samples) - Make the labels of the test/train set one-hot.
This seems as easy as a single function calling, i.e. pd.get_dummies but this is only correct when the train and test set contain the same labels. There has to be a different process for label mismatches between training-testing sets. - Define the model.
The definition of the models was done using initial skeleton codes.
For the DNN-MLP we used the MNIST tutorial on tensorflow website and adapted it to our needs.(https://github.com/aymericdamien/TensorFlow-Examples/blob/master/examples/3_NeuralNetworks/multilayer_perceptron.py)
For the CNN, we used a tutorial blog for new tensorflow users and adapted it accordingly.(https://aqibsaeed.github.io/2016-11-04-human-activity-recognition-cnn/)
For the LSTMs, we used the higher-level tensorflow libraries (contrib library) following a github repository which was under MIT licence, dealing with the HAR problem.
The FF-LSTM-RNN skeleton is from :https://github.com/guillaume-chevalier/LSTM-Human-Activity-Recognition
The BD-LSTM-RNN skeleton is from :https://github.com/guillaume-chevalier/HAR-stacked-residual-bidir-LSTMs - Define the optimization function.
We experimented with Adam Optimizer, Gradient Descent and Adagrad. - Run the training session.
- Print runntime and results for the test-set.