Standard interface of RNN units

[delta2323]

Currently, several types of RNN units are implemented (stateful/stateless, LSTM/GRU). And we also have PRs that implement other kind of RNN units (SGU, DSGU (#1115), MGU (#1101), peephole (#1207), stacked GRU and stacked LSTM (#1124) etc.).

It would be cumbersome if these RNN units had different APIs as we sometimes switch RNN units within the model without change the interfaces. So it is preferable for those RNN units to have standard interface that are common to them.

[unnonouno]

I think each RNN unit function f can be abstracted with four components. I assumes f is a Link object with parameter W (so called stateless XXX).

• input: x
• output: y
• hidden state: h
• parameters: W

Each unit f gets x and h, and it returns y and updated h: y, h' = f(x, h)
In some units, y is equivalent to h (such as GRU), or h contains y (such as LSTM).
When a user want to stack units, a second unit function gets output of a first unit function.

h1 = 0
h2 = 0
y1, h1' = f1(x, h1)
y2, h2' = f2(y1, h2)

A statefull unit s has an attribute h, and when __call__ is called, it updates its own hidden state.

def __call__(x):
    y, self.h = f(x, self.h)
    return y

For variable length inputs, we need return parts of hidden states as y. See #1183

The interface of cuDNN's RNN helps you because it supports many types of RNNs in one API.

[delta2323]

@unnonouno Thank you for your proposal. My vague idea was almost same as you. And also thank you for suggestion of cuDNN RNN's API. How do you think of the separation of Function and Link, as we did in LSTM?

[unnonouno]

Function is not mandatory. I mean, users can make RNN unit as a method with combination of existing Functions. I feel Link is enough though someone may need a method when they want to make parameter matrices from other functions. I don't have such a use case.

[delta2323]

I agree with you in that Function is not mandatory because for example, GRU is difficult to isolate "function" part.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 30 days if no further activity occurs. Thank you for your contributions.

[stale]

This issue is closed as announced. Feel free to re-open it if needed.