Attention explanation

[smolendawid]

Is there any paper or tutorial that describes exactly the same attention mechanism that is used in this repository? I mean the fact that attention values are added, not concatenated, the usage of LinearND, and the fact that there is a convolution. Is there any place with the theory given?
Thank you

[awni]

Here are two references you can take a look at:

• Attention-Based Models for Speech Recognition
• End-to-End Attention-based Large Vocabulary Speech Recognition

The attention (NNAttention) is mostly the same. The only difference is it doesn't do a linear multiply before the nonlinearity. I don't think that matrix multiply is necessary there as you can add more transformations to the encoded and decoded state in the encoder and decoder respectively. However, I haven't gotten around to testing this rigorously yet. (I expect it would be minor to no difference).

As for LinearND this is just a helper layer to do a linear transformation on something with the shape [batch, time, hidden dim] to reshape it to [batch*time, hidden dim] before the matrix operation.

[smolendawid]

I appreciate your help very much, thank you.