Stacked LSTMCells in time_sequence_prediction example

[balsulami]

In the time_sequence_prediction example, the two layers LSTM are defined and used as the following:

def forward(self, input, future = 0):
        outputs = []
        h_t = Variable(torch.zeros(input.size(0), 51).double(), requires_grad=False)
        c_t = Variable(torch.zeros(input.size(0), 51).double(), requires_grad=False)
        h_t2 = Variable(torch.zeros(input.size(0), 1).double(), requires_grad=False)
        c_t2 = Variable(torch.zeros(input.size(0), 1).double(), requires_grad=False)

        for i, input_t in enumerate(input.chunk(input.size(1), dim=1)):
            h_t, c_t = self.lstm1(input_t, (h_t, c_t))
            h_t2, c_t2 = self.lstm2(c_t, (h_t2, c_t2))
            outputs += [c_t2]
        for i in range(future):# if we should predict the future
            h_t, c_t = self.lstm1(c_t2, (h_t, c_t))
            h_t2, c_t2 = self.lstm2(c_t, (h_t2, c_t2))
            outputs += [c_t2]
        outputs = torch.stack(outputs, 1).squeeze(2)
        return outputs

Notice that the cell state , not the hidden state, in the 1st layer is used as in input for the 2nd layer. Is that correct? In my understanding of Stacked LSTM is that the hidden states of the lower layers are the input for higher layers. Am I missing something? Also, all the examples from tensorflow, chainer, and theano use the hidden state variables not the cell states as an input.

[dapengchen123]

I am also confusing about it.

[mlpanda]

I agree that this should be changed to:

    for i, input_t in enumerate(input.chunk(input.size(1), dim=1)):
        h_t, c_t = self.lstm1(input_t, (h_t, c_t))
        h_t2, c_t2 = self.lstm2(h_t, (h_t2, c_t2))
        h_t3, c_t3 = self.lstm3(h_t2, (h_t3, c_t3))
        outputs += [h_t3]

    for i in range(future):# if we should predict the future
        h_t, c_t = self.lstm1(h_t3, (h_t, c_t))
        h_t2, c_t2 = self.lstm2(h_t, (h_t2, c_t2))
        h_t3, c_t3 = self.lstm3(h_t2, (h_t3, c_t3))
        outputs += [h_t3]

[barnabytprowe]

Hi guys - I'm just exploring known issues here because I'm having some trouble with the LSTM example along the lines of #215, but is this particular issue not resolved by the merge in #222? (@dtolpin might be able to comment.)

The only difference seems to be the extra third layer you are advocating @mlpanda...

[mlpanda]

Hey @barnabytprowe, my difference was not the addition of a third layer (stupid of me to add that), but that they fed the cell state, c_t, into the next layer previously:

h_t2, c_t2 = self.lstm2(c_t, (h_t2, c_t2))

Instead that should be (as per my comment):

h_t2, c_t2 = self.lstm2(h_t, (h_t2, c_t2))

And yes, they have fixed that lately. As mentioned in #222 it works for the sine example either way, but it's correct to feed the hidden state, h_t, to the next layer.

[barnabytprowe]

Great, thanks for getting back to me @mlpanda ... I think that means this issue can be closed.

[Sodininkas]

Hello,

for i in range(future):# if we should predict the future
            h_t, c_t = self.lstm1(c_t2, (h_t, c_t))
            h_t2, c_t2 = self.lstm2(c_t, (h_t2, c_t2))
            outputs += [c_t2]

Is this correct because range(future) do nothing. Every time lstm1 gets same c_t2, that leads to same result?