SSM vs RNN
One difference is the use of activation functions in RNN to help it pick on more complex connections between the input and output. While SSM just relies on linear transformation of the preceding state using a transformation matix A to predict the next state.
Another difference is in RNNs let's say we have a sequence of two data points say yesterday’s value and today’s value and we are trying to predict tomorrow's value, in RNNs first, yesterday’s value will be fed into the NN with let us say one hidden layer, then the value goes through the normal feed-forward steps (weights, biases, and finally an activation function) and there would be an output computed corresponding the prediction of today’s value.
Here instead of comparing the predicted value for today’s value with the actual value and backprop by computing error to optimize our parameters, RNN goes through what is called a feedback loop where that output is stored. The data for today’s value is fed into the same NN( the same neurons with the same weights). As it reaches the hidden layer (multiplied by the weight and bias added to it) before going through an activation function and sent to the output layer, the stored output value of yesterday’s value will be multiplied by another weight(feedback loop weight) before it is added with what was computed for the input corresponding to today’s input and that whole sum will go through the activation function and sent to the output neuron as tomorrow’s prediction. This way both yesterday’s and today’s values influence the prediction for tomorrow’s value The back-prop only happens once we have gone through every step and predicted the last value in the sequence. The problem with this is the vanishing gradient problem. This happens because we are using the same neuron(weights) as we predict for each step in the sequence, and the final prediction will inevitably involve the weights multiplied against themselves, and when computing the gradient(contribtion of a particular weight to the loss which is the prediction-actual value) which is done by taking the derivative, unlike regular NNs where the derivative with respect to the weight would just be the corresponding activation value of the preceding neuron since the power of the distinct weights is 1, in RNN’s case the power of distinct case is greater than 1(equivalent to our size of sequence we are considering), so if the initial value of the weight is between 0 and 1 which it usually is, the gradient( derivative) would be that value of the preceding neuron multiplied by the weight to the power of some number>1, and so the gradient would be very small this means it wouldn’t allow us to perform a proper gradient descent to optimze the values of the weights.
However, in SMMs, the tunable parameters are the entries of the matrices (A,B, C,D). A corresponds to how the preceding state transitions to the next state, and B corresponds to how the input affects each of the components of the state which are added to give the value of the next state,then the sum of the matrix multiplications with these to matrices would give the value for the next state. C corresponds to how our new state affects our custom output (which isn't necessarily just the state variable vector) and D corresponds to how the input directly affects our custom output.
During training from what I understand using the same example as last time. Yesterday’s value will be fed into the state equation accordingly and this will give us a prediction for today’s value, this output will be a stored output. Then today’s value will be fed into the state equation and give its prediction(output) for tomorrow’s value. Then when computing error, we will compute the error at each step using the prediction and actual value at each step, then update the matrices accordingly. This means the error would be the sum of the differences of the predicted value at a particular step and their corresponding actual value ((Ytoday-pred-Ytoday-actual) + (Ytomm-pred_Ytomm-actual)...) then backprop will happen, tuning the entries of the matrices.
The similarities lie in how an RNN as the weights multiply against each other as we go further down the sequence as every step is influenced by its preceding step, in SSMs the matrices multiply against themselves as we go down the sequence, as the prediction of the state of each step is the multiplication of its previous state and the matrices, but the previous state itself is also the product of its previous state with the same matrices so the matrices will end up getting multiplied against themselves as we go down the sequence.
Also, both classical SMMs and RNNs use a recurrent representation meaning the output of one time step will be the input for the next time step and so on for both training and inference processes. This sort of representation is good for inference as it has a linear time complexity, but because it can’t be parallelized, it is slow for training purposes.
S4 vs LSTM
S4 has 2 main features that differentiate them from classical SSMs. The first is while SSMs have a recurrence based representation of the sequences ,and although recurrence based representations are fast for inference purposes, they can't be parallelized so they are difficult to train. So S4 does something different.
Convolutions are much faster to compute as the process can be parallelized in contrast to recursive models so S4 exploits that for its training mode and switches to the recursive representation for its inference mode.
In addition, S4 has a mechanism like in attention networks they are able to put more weight on words that are more relevant to the context, S4 also does this where when taking in the current available context as a state it gives more importance to relevant words in the context than others. This is comparable to LSTMs as the influence of some steps in the sequence are still apparent further down the sequence by being placed in the long term memory by the use of the input gate, while the effects of some steps in the sequences are short lived and add a small value both to the long-term memory and the short term memory in the output gate.
- LSTM: Explicit Gating Mechanism LSTMs use gates (input, forget, and output gates) to explicitly control the flow of information across time steps. These gates allow the model to:
Forget irrelevant information (via the forget gate). Update the memory cell with new information (via the input gate). Control the output of the memory cell (via the output gate). This gating mechanism enables LSTMs to dynamically adjust the importance of past steps in the sequence, making them effective for handling long-term dependencies.
- S4: Implicit Long-Range Dependencies via State Space Models S4, on the other hand, uses a state-space model (SSM) to represent the sequence as a continuous-time dynamical system. It does not rely on explicit gates like LSTMs. Instead:
S4 parameterizes the state-space system to efficiently capture long-range dependencies across the sequence. This is due to the HiPPo which is a way for the model to keep a summary of what happened at all the previous timesteps so that old time steps can still have influence over the computation of the current state. The S4 model uses a learned convolution kernel derived from the SSM to determine how much influence each time step has on the output. This convolution kernel implicitly adjusts the importance of steps across the sequence, allowing S4 to model both short- and long-term dependencies without the need for explicit gating.
Key Difference in Adjusting Importance
LSTM: Adjusts importance explicitly through gates that are learned during training. S4: Adjusts importance implicitly through the learned convolution kernel derived from the