How to implement a multiDimensional RNN?

[ypxie]

Suppose the sequence are organized as a grid (2D) rather than a line. and each node recieve from it's predecessors in both row ,column and diagnal directions.
The following is an example, number means the sorted order of the grid feed into the LSTM.

For example, 6 will receive information from both 5 ,4 and 3. and 7 will recieve information from 6 and 3, and so on....

1 4 5

2 3 6

9 8 7

10 11 12

I plan to implement this using scan, but the problem is the tap is not fixed. It grows up.

I thought about using predecessorList to store the predecessor lists for every node. and pass the outputs of scan to scan. But I am not sure if scan supports that.

do you have any suggestions?

I'd like to provide more information to clarify the problem if you need.

Thank you!

@EderSantana

[EderSantana]

I am not sure how to do that, sorry! I would say that you should slice your inputs and pass each piece to an input. Also, can't you use a convolution to get the local neighborhoods you need?

[ypxie]

Thanks for your reply.

I am not sure what do you mean by slice the inputs. Is it about cropping the whole data (e.g. image) into many small squares?

I am a newbie to theano, one thing I am not sure is, if 'taps' has to be fixed value in scan? If so, I think it will be a very hard restriction for cases like this.

[ypxie]

@EderSantana

[EderSantana]

I'm not sure of what you mean by taps. But if you encountered a Theano restriction, that may be beyond of what I can do...

[ypxie]

Sorry for the confusion！
taps can be found in the following code snept. In this case, the taps are defined before the loops start. But in 2D case, I might need the taps can be changed over time.

    outputs, updates = theano.scan(
        self._step,
        sequences=[x, dict(
            input=padded_mask,
            taps=[(-i) for i in range(self.depth)]
        )],
        outputs_info=[dict(
            initial=initial,
            taps=[(-i-1) for i in range(self.depth)]
        )],

[EderSantana]

oh wow, I'm not familiar with taps, never used it before... I may have to learn that first.