Wrapper to enable recurrent cells to handle sequential input

[tanmayb123]

In Flux (machine learning for Julia) you can wrap an RNN Cell in a Recur type so it can keep track of the hidden state for you. Here's an example from their code:

accum(h, x) = (h+x, x)
rnn = Flux.Recur(accum, 0)
rnn(2) # 2
rnn(3) # 3
rnn.state # 5
rnn.(1:10) # apply to a sequence
rnn.state # 60

Would it be valuable to have something similar with swift-apis?

[rxwei]

Interesting. I think so. Would you like to prototype it?

[tanmayb123]

Sure thing, let's get the cells merged in first though.

[jekbradbury]

I'm not sure this API is as good a fit for Swift as it is for Julia--we've tended to avoid mutation here. Instead it might make sense to expose a RNN<CellType> layer that folds its underlying cell over an input array or tensor and returns an output array or tensor.

[tanmayb123]

Thanks for the suggestion James! I'll submit the PR for that soon.

[rxwei]

I think it's better to start prototyping a sequence-to-sequence layer that's generic over RNN cells once Array conforms to Differentiable, before trying layers that mutate internal states. For example:

struct RNN<Cell: RNNCell>: Layer {
    @differentiable
    func applied(to input: [Cell.Input]) -> [Cell.Output] {
        return input.reduce(.zeroState) { ... }
    }
}

[rxwei]

Array now conforms to Differentiable thanks to work done by @eaplatanios and @marcrasi. We can go ahead and add an RNN layer now.

[tanmayb123]

Hooray! Should I also add the cells back with this PR? We can merge when TF-440 is resolved?

[rxwei]

It's already resolved and the cells are added back.

[tanmayb123]

Yeah just realized that sorry. Working on RNN now.

[tanmayb123]

Here's a quick sketch: https://gist.github.com/tanmayb123/33a337f5937cc588c2abc413b4e91343

I'd love some feedback on this!