New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Does lstm in keras add peephole connection? Which paper do these codes refer to? #1717
Comments
@EderSantana @fchollet @dbonadiman Could you spend your valuable time solving this question? Thanks a lot. |
So here is how we calculate the activations of an LSTM https://github.com/fchollet/keras/blob/master/keras/layers/recurrent.py#L443 i = self.inner_activation(x_i + K.dot(h_tm1, self.U_i)) if I'm not wrong, the peephole should take a peek at the cell content and do something like: i = self.inner_activation(x_i + K.dot(h_tm1, self.U_i) + K.dot(c_tm1, self.U_i)) If I'm correct, the following class should do what you need: from keras.layers.recurrent import LSTM
class LSTMpeephole(LSTM):
def __init__(self, **kwargs):
super(LSTMpeephole, self).__init__(**kwargs)
def build(self):
super(LSTMpeephole, self).build()
self.P_i = self.inner_init((self.output_dim, self.output_dim))
self.P_f = self.inner_init((self.output_dim, self.output_dim))
self.P_c = self.inner_init((self.output_dim, self.output_dim))
self.P_o = self.inner_init((self.output_dim, self.output_dim))
self.trainable_weights += [self.P_i, self.P_f, self.P_o]
def step(self, x, states):
assert len(states) == 2
h_tm1 = states[0]
c_tm1 = states[1]
x_i = K.dot(x, self.W_i) + self.b_i
x_f = K.dot(x, self.W_f) + self.b_f
x_c = K.dot(x, self.W_c) + self.b_c
x_o = K.dot(x, self.W_o) + self.b_o
i = self.inner_activation(x_i + K.dot(h_tm1, self.U_i) + K.dot(c_tm1, self.P_i))
f = self.inner_activation(x_f + K.dot(h_tm1, self.U_f) + K.dot(c_tm1, self.P_f))
c = f * c_tm1 + i * self.activation(x_c + K.dot(h_tm1, self.U_c) + K.dot(c_tm1, self.P_c))
o = self.inner_activation(x_o + K.dot(h_tm1, self.U_o) + K.dot(c_tm1, self.P_o))
h = o * self.activation(c)
return h, [h, c] |
In addition you could take a look at this paper, and this Ph.D. thesis. |
@EderSantana Thanks a lot. By the way, does LSTM in keras stems from (Hochreiter and Schmidhuber, 1997) or a variant (Graves, 2013) ? |
@Imorton-zd We are using Graves 2013 (with forget gates with bias equal 1). |
@EderSantana |
@dinghaoyang Have you made this peephole structure work? Or anyone made it work? |
Here is a paper refering to peepholes. |
Hi, Did anyone able to find a solution for this? According to my understanding, Keras LSTM cell is similar to Tensorflow
However, I am looking for the implementation of Tensorflow
in Keras (LSTMCell : The class uses optional peep-hole connections, optional cell clipping, and I am wondering without writing a custom layer, is there is any way to use this Tensorflow LSTMCell in Keras or is it already available? |
I have the same question:
And it seems there's a peephole implementation in |
@zixia I don't see a peephole implementation in |
I guess lstm in keras doesn't implement peephole connection, according to the kera ducmentation, the lstm layer is based on the paper "Long Short-Term Memory layer - Hochreiter 1997", and i also check lassagne's documentation, it's lstm layer is based on "Graves, Alex: “Generating sequences with recurrent neural networks.” arXiv preprint arXiv:1308.0850 ", where peephole connection is added to the orgin model. And if you print the number of params of both implementation, you will find the params of keras is less the params of lassagne. |
The Unify RNN Interface RFC confirms that tf.keras's So if you want peepholes with tf.keras, you have to create a custom cell that wraps the |
In keras, the lstm neural network is efficient for text classification. However, I can't understand the detail codes and don't know if the lstm neural network adds peephole connection. Please give me some reference documents. Opinions/Views would be highly appreciated!
The text was updated successfully, but these errors were encountered: