Recurrent layers: Accept layer for hid_init

[f0k]

The recurrent layers currently have some custom behaviour when hid_init is a TensorVariable rather than a shared variable, callable or numpy array: They assume hid_init is a tensor of one order higher than otherwise, to include the batch dimension, and they assume hid_init is not to be learned.

With #404, parameters can be arbitrary Theano expressions anyway, and overriding that behaviour to assume a different dimensionality is a bit awkward. As discussed in #11 (comment) and following, a better solution would be allowing hid_init to be a Layer, and assuming it would include the batch dimension in this case. This would also be a step towards supporting the encoder-decoder architecture discussed in #391 (comment) and following.

[craffel]

This seems reasonable to me. I think @skaae is better suited to comment, as he added that functionality for a specific use case (next character prediction) in mind. I think the new recurrent containers will be a better way to handle that use-case anyways.

[skaae]

Yes its better require ´hid_init` to be layer. For the "my" usecase you can just wrap the tensor in an input layer.

[craffel]

Ok, cool. @f0k can you assign this to me? Unless one of you want to do it.

[f0k]

You're welcome to tackle this! :) And I'll be glad to review.

[skaae]

Remember to return the parameters when hid_init/cell_init is a layer. We got a question about this on the mailing list.

[f0k]

Remember to return the parameters when hid_init/cell_init is a layer.

No need to do that manually, hid_init would become part of the layer's incoming layers (i.e., it would be passed to the MergeLayer super constructor). Otherwise it couldn't properly participate in get_output() anyway.

[skaae]

aah yes :)

[skaae]

@craffel: Do you plan to work on this soon? otherwise i can create a PR.

[f0k]

Do you plan to work on this soon? otherwise i can create a PR.

Look above your comment, there is a PR, #522! :)

You could start a PR on top of that which removes the TensorVariable special case, though. (I.e., do not base it on master, but base it on #522, as we're going to merge that anyway.)

[skaae]

missed that :) I'll remove the TensorVariable special case when #522 is merged.

[craffel]

@craffel: Do you plan to work on this soon? otherwise i can create a PR.

I was planning on it, but I'm glad someone else did it :)