Input-feeding Approach

[TinaB19]

Thank you very much for the awesome work, I need a clarification in the decoder part of seq2seq-translation.
#Combine embedded input word and last context, run through RNN
rnn_input = torch.cat((word_embedded, last_context.unsqueeze(0)), 2)

Is the above code an implementation of Input-feeding Approach in the Effective Approaches to Attention-based Neural Machine Translation paper?

[spro]

Yes it is, though looking back at it I'm missing one layer between the context vector c_t and the softmax layer, to create the "attentional hidden state" ~h_t, which is what they use for input feeding.

[TinaB19]

It would be great if you add it to the tutorial later, thank you very much.

[TinaB19]

I just saw seq2seq-translation-batched.

concat_input = torch.cat((rnn_output, context), 1)
concat_output = F.tanh(self.concat(concat_input))

So I guess in this case we should concatenate concat_output with embedded in the next time step and then feed them to gru. Is this correct?

[spro]

Correct.