Different input dimention compared to output dimension

[titu1994]

Hi, I'm trying to implement a naive version of this paper in Keras, and was wondering how is the case that - n_in != n_out handled.

I went through the code a few times, and couldn't understand the element wise multiplication of (1 - r_t) with x_t, if x_t is of a different shape than r_t.

[taolei87]

Hi,

When n_in != n_out, we simply add one more linear transform (say W) at the highway connection:
h'[t] = r[t] * h[t] + (1-r[t]) * (Wx[t])

Again, this multiplication can be batched together with other mm as well. that said, this (Wx) is being put into U. and I guess that's why you find the code confusing.

We should better document the implementation. Thanks for pointing this out!

[taolei87]

Also, in speech task, the (Wx) is always included, which is discussed in the appendix.

[titu1994]

Thanks for the quick reply!

Just for further clarification, this new W will infact be different W from equation (3) in the paper, and will not have any bias?

[taolei87]

yes, the new W will be different to (3). and no bias term for them.

only the neural gates have biases (4) (5)

[titu1994]

Fantastic. Thank you very much for the clarification, and fantastic work on this efficient implementation in CUDA.

[taolei87]

You are very welcome!