Exermote Preprocessing and Training
After collecting labeled data, a model needs to be trained!
There were a few preprocessing steps I made. Some of them are rooted in insights I had, when I was already training models:
- merging raw recordings to one file
- reducing total number of classes from 5 to 4, by converting "set break" to "break". I don't know what I was thinking when introducing two different break classes...
- converting the first and last two time steps of every squat repetition to "break". Earlier models often counted squats, when I actually didn't do anything. This fixed it.
Choosing a model
I intended to write my master thesis in human activity recognition (HAR), but I didn't find a supervisor. Anyway I could use some of the insights from my thesis proposal. The following table is an excerpt from this proposal.
As you can see in the last row DeepConvLSTM Neural Networks were already tested by Francisco Javier Ordóñez and Daniel Roggen for recognizing activities of daily living. Their approach and their results impressed me and so I decided to take their model and give it a try for my purpose. The model takes time sequences of raw sensor data and outputs the according exercise label. A simpliefied model represantation looks like this:
The actual model differs in terms of layer and channel (data feature) numbers. Furthermore a higher stride and a dropout layer were added for better generalization:
model = Sequential([ Conv1D(nodes_per_layer, filter_length, strides=2, activation='relu', input_shape=(timesteps, data_dim), name='accelerations'), Conv1D(nodes_per_layer, filter_length, strides=1, activation='relu'), LSTM(nodes_per_layer, return_sequences=True), LSTM(nodes_per_layer, return_sequences=False), Dropout(dropout), Dense(num_classes), Activation('softmax', name='scores'), ])
The whole training procedure took place in the google cloud, since I found this wonderful tutorial. The machine learning framework in use was Keras with TensorFlow as backend. Many thanks to Google for 300$ of free credits. After training hundreds of models there are still plenty left:
For training observation I used TensorBoard:
The (optimum) parameters shown below where determined during training.
timesteps defines the sliding window length, while
timesteps_in_future specifies which time step label should be characteristic for a sliding window. More
timesteps_in_future would mean a higher accuracy in recognition, while it would worsen live prediction experience.
# training parameters epochs = 50 batch_size = 100 validation_split = 0.2 # model parameters dropout = 0.2 timesteps = 40 timesteps_in_future = 20 nodes_per_layer = 32 filter_length = 3
While training models with various input combinations, it became clear that the benefit of using the mentioned Nearables is a smaller one. Therefore I gave up on using them any longer. Additional sensors might get interesting again for recognizing one-armed exercises or exercises where only your feet and/or legs are moving.
X = dataset[:, [ 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, # Device: xGravity, yGravity, zGravity, xAcceleration, yAcceleration, zAcceleration, pitch, roll, yaw, xRotationRate, yRotationRate, zRotationRate # 14,15,16,17, # Right Hand: rssi, xAcceleration, yAcceleration, zAcceleration # 18,19,20,21, # Left Hand: rssi, xAcceleration, yAcceleration, zAcceleration # 22,23,24,25, # Right Foot: rssi, xAcceleration, yAcceleration, zAcceleration # 26,27,28,29, # Left Foot: rssi, xAcceleration, yAcceleration, zAcceleration # 30,31,32,33, # Chest: rssi, xAcceleration, yAcceleration, zAcceleration # 34,35,36,37 # Belly: rssi, xAcceleration, yAcceleration, zAcceleration ]].astype(float) y = dataset[:, 0] # ExerciseType (Index 1 is ExerciseSubType)
The highest recognition accuray achieved on test data with only 12 data features was 95.56 %. Since mainly first or last timesteps of a repetition were confused for a break or the other way around, this accuracy is sufficient for recognizing and counting the mentioned exercises. The best model of a training procedure was saved to google cloud bucket. The model was also exported to .pb and .mlmodel format for later use on google cloud and on iPhone.