ZoneoutLSTMCell incorrect output

[albertz]

Paper.

We currently have this code:

# Apply vanilla LSTM
output, new_state = self._cell(inputs, state, scope)

(prev_c, prev_h) = state
(new_c, new_h) = new_state

# apply zoneout
c = ...
h = ...

new_state = rnn_cell.LSTMStateTuple(c, h)

return output, new_state

output is the original output, and h the zoneout-transformed output.

But actually it should not return output but it should return h instead. At least this is my understanding of the paper.

Can someone confirm?

So, what do we do now? I think there are many existing setups using this. And just changing this would change the behavior and do sth different then, so it is not really compatible.

We could introduce a new option to switch between the incorrect and correct behavior. This flag default would use the correct behavior with a new behavior version.

[michelwi]

Can someone confirm?

If the h's are the same as the output's, then I agree that the current behavior is not the same as described in the paper. As described in the paper h should be the output which is a combination of prev_h and new_h according to a random binary mask.

[albertz]

I fixed this now. If you switch to behavior version 17, it will by default change to the new correct behavior. By staying at older behavior versions, nothing will change for you.

If you want to stay at your older behavior version, but want to see the effect of this fix, just explicitly set use_zoneout_output=True in the ZoneoutLSTM flags (unit_opts).

If you want to get the old incorrect behavior in a new behavior version, just explicitly set use_zoneout_output=False.