Support for Recurrent Neural Networks

[jalving]

@davidstadem inquired about support for recurrent neural networks including support for LSTM nodes. I think one of the challenges with recurrence is representing the dynamic state in an optimization setting. For instance, the form of the problem would depend on the length of an input sequence. It will also be necessary to support general network architectures through ONNX. Currently, we always assume a single input layer.

[joshuahaddad]

I’ve thought about this a bit since we’ve talked about it in the past. I think we’d need to make an interface that takes the input data and generates the equations since the input sequence matters. If we know the length of the input sequence it shouldn’t be hard to do the LSTM equations.

One idea I had for this was to implement a “persistent” option where the user could change their input sequence length and we’d update the model.

From my use of OMLT though I think defining an input sequence length may be easy since we have the use define input variables.

[jalving]

I like the "persistent" option idea. We could try rebuilding an OmltBlock without re-creating the input and output variables. How about we start with something simple like a Jordan or Elman network? It would also be good to find out how ONNX stores simple recurrent networks.