Inefficient usage of resources

[zeryx]

I was exploring this project for a general purpose forecasting model project I was working on, and I realized that the performance of this model with multiple layers (say 5) is actually worse in both forward and backward passes than a standard pytorch GRU module with 5 layers.

https://gist.github.com/zeryx/c43fc53b4d3f71c4942dff44912aa3cb

From my understanding of the paper, the DRNN module should be at least as performant as the equivalent GRU module if not dramatically superior in both forward and backward pass compute time.

[kashif]

thanks @zeryx I'll also investigate...

[blythed]

Might have something to do with looping over layers in a native python loop.

[zeryx]

I did a little bit of digging as well, it looks like the hidden memory tensor is never actually updated. I made a local change that forced the DRNN layer to make use of the allocated hidden memory tensor, however, due to the python native List around the memory tensors (as each memory tensor has different dimensionality, you can't stack them conventionally) training requires the "retain_graph" variable set to true in the backward() function.
I can put my work in a PR but I have a feeling that a full rewrite might be required.

[kashif]

ah cool @zeryx please push your stuff then we can try to figure it out and fix it!

[zeryx]

^ that PR would expect the hidden tensor to be

drnn_h = [Variable(torch.zeros(2 ** i, 1, self.hidden_width)).cuda().float()
                    for i in range(self.depth)]

I haven't been able to figure out a way to convert that into torch.Variable, as each layer's hidden tensor has increasing dimensions