Recurrent Neural Networks

Introduction

Sequence labeling – process of transcribing data sequence to sequence of discrete labels
Applications
- Speech recognition
- Handwriting recognition
- Protein secondary structure prediction
Sequence labeling vs. pattern classification
- Correlations in input data and output data

Problem types

Pattern classification

-> T
Sequence classification

-> T
Segment classification
- Frame-wise labels
- Context
- Time windows
Temporal classification

-> Text
- Unsegmented labels

Unfolding RNN

Unfolding network along input sequence
No recurrent connections

Recurrent Neural Network (RNN)

MLP maps input vector to output vector
Recurrent connections allow ‘memory’ of previous inputs
RNN maps entire history of previous inputs to output vector

Forward pass

Almost the same as MLP, except inputs come from the hidden layer as well
- input to hidden unit h at time t
- output of hidden unit h at time t

Backward pass

Backpropagation through time (BPTT)

$\delta_h^t$