Added implementation for GridRNN

[phvu]

As discussed here: #1453

The implementation is generic, users can specify the number of dimensions and various configurations for those dimensions (input/output/priority/non-recurrent). The type of the cells along dimensions can also be selected among LSTM, GRU, vanilla RNN.

Come with unittests for basic types: 2LSTM (tied weights, non-recurrent), 2BasicLSTM and 2RNN.

I made a simple test of Grid2LSTM for character-level language modeling: https://github.com/phvu/grid-lstm-tensorflow/tree/master/char-rnn.

[tensorflow-jenkins]

Can one of the admins verify this patch?

[ebrevdo]

Will take a look tomorrow.

[ebrevdo]

add newline

[phvu]

done

[ebrevdo]

Thanks for the hard work! Some comments.

[phvu]

@ebrevdo Thanks for the comments. I updated the code.

• Good to know about input_size and output_size, I updated the code to not depend on those.
  However as the LSTMCell is still accepting input_size, I still keep input_size as a parameter in the cell_fn callback. Should it be also removed?
• The tests are real. I computed the values using this. Since the initializer is fixed in the tests (weights are initialized to 0.5 or 0.2, depending on the tests; bias are initialized to zero), the output are deterministic. But of course I can switch to lightweight asserts if that is preferred.

[ebrevdo]

Without a unit test that calls tf.nn.rnn or tf.nn.dynamic_rnn, it's not clear that these cells interact correctly with those methods. Can you add such a test for your classes?

[phvu]

ok will do.

[phvu]

I added the tests for Grid1LSTM, Grid2LSTM, Grid3LSTM (with ReLU), trained with tf.nn.rnn. Can you take a look?

[ebrevdo]

Will look tomorrow-thanks!
On Apr 6, 2016 11:36 AM, "Vu Pham" notifications@github.com wrote:

I added the tests for Grid1LSTM, Grid2LSTM, Grid3LSTM (with ReLU),
trained with tf.nn.rnn. Can you take a look?

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#1665 (comment)

[ebrevdo]

Jenkins, test this please?

[gunan]

Can one of the admins verify this patch?

[ebrevdo]

Sorry for not reviewing earlier. Let's retest this and we can get it merged.

[ebrevdo]

@phvu can you rebase on HEAD and run the tests?

[vrv]

@ebrevdo: why does he have to rebase to HEAD? Do you think there will be a conflict in doing so?

Also, only we can trigger tests.

[vrv]

@tensorflow-jenkins: test this please

[phvu]

Cool, tests passed. So I don't need to rebase master for now?

[ebrevdo]

Excellent, we're good to go. Thanks for the contribution! If you have any example code using this we can consider adding it elsewhere in the repo (e.g. under models).

[phvu]

Cool, thanks for all the help.
Unfortunately I can't share what I am working on with this yet. I have an application of this to character-level modeling (https://github.com/phvu/grid-lstm-tensorflow/tree/master/char-rnn) but it isn't entirely my code so I put it in a separated repo.

I think at some point I can try reproducing some experiments in the paper. Will submit PRs by then.

[jstaker7]

Hi @phvu,

Thanks very much for your work. I'm a little confused as to how this can be applied generally. For example, in the paper a 3-LSTM network was applied to image patches of the MNIST dataset where each patch is c * m units long (which is equal to the depth dimension). So we have a 3D grid of c * m vectors. But in your implementation of __call__, we are expecting a tensor of size batch * c * m * num_dims. For MNIST, shouldn't it support b * c * m * dim1_size * dim2_size ? This would require input_dims to be something like [1, 2] (where dim 0 is batch); but I don't see any tests covering cases besides input_dim=0. Am I missing something?

[phvu]

Hi @jstaker7,
There are some nuances here.
First, the __call__ method expects the state tensor of batch_size * ( (c + m) * num_dims). When c=m=2 then c*m = c+m. Sorry for the bad test case, I should have used other values.

Second, in the MNIST case, (as far as I understand) I suppose you could use Grid3LSTMCell (https://github.com/tensorflow/tensorflow/blob/master/tensorflow/contrib/grid_rnn/python/ops/grid_rnn_cell.py#L242). This cells receives input and gives output in the first dimension (index 0), and uses LSTM in the other 2 dimensions.
In order to replicate Figure 11 in the paper, you will need to construct 4 3-LSTM cells, each cell will handle 1 scan direction of the input image. However the first hidden LSTM layer receives original pixels as input, while other hidden LSTM layers receives the output of the LSTM just below it.
How to do that is up to you. One simple way is to have a loop to scan the image in a given scan direction (say left-to-right top-to-bottom), which will give a sequence of vectors, and then feed that sequence into the corresponding 3-LSTM cell for that scan direction. For bigger images, you might want to use tf.scan()

So your assumption that input_dims should be [1,2] is not true. The cells should receive input at dimension 0. For me this is the only reasonable interpretation of the paper, which also take 1-LSTM and 2-LSTM cells into account. (unless the authors release their implementation so that we can do a comparison).

[jstaker7]

Hi @phvu,

Thank you, this helps quite a lot. Since each dimension will always have its own LSTM, does that mean we will never use an input_dims other than 0?

[phvu]

To replicate the experiments in the paper, yes. But I am not sure about that generally.

When we set dimension i to be input_dims, it simply means for each __call__, the cell expects an input tensor for dimension i. For non-input dimensions, the cell will take the recurrent values (extracted from the state tensor) as input.

So I guess we can be creative and feed inputs into more than one dimension. I used dimension 0 as input and output dimensions just as a convention. You can as well construct your own GridRNNCell with any configuration.

[jstaker7]

Perfect, thanks so much for the clarification. Super helpful and makes a lot of sense.

This might also be interesting to look into in the future. Grid LSTMs were mentioned and it looks like there was a recent merge: #2560