Add RNN support for Pytorch

[JanFSchulte]

Adds support for RNN layers (GRU, LSTM, RNN) to the pytorch parser.

Caveat: We currently lack implementation for getitem operations, so we can currently not return the hidden state after the calculations

Caveat 2: We currently only support a single recurrent layers, whereas multiple within the same RNN instance are supported by pytorch

Caveat 3: We currently don't support the passing of non-zero initial values for the hidden states to the RNN

So this implementation is slightly hacky at the moment, but might serve as a starting point for discussion, and can be used by interested parties if they can life with the current limitations.

Also, this contains parts of #848 because I was inattentive.

Type of change

For a new feature or function, please create an issue first to discuss it
with us before submitting a pull request.

Note: Please delete options that are not relevant.

• New feature (non-breaking change which adds functionality)

Tests

Added pytests to confirm that the layers work.

Checklist

• I have read the guidelines for contributing.
• I have commented my code, particularly in hard-to-understand areas.
• I have made corresponding changes to the documentation.
• My changes generate no new warnings.
• I have installed and run pre-commit on the files I edited or added.
• I have added tests that prove my fix is effective or that my feature works.

[vloncar]

pre-commit.ci autofix

[jmitrevs]

The tests fail with:

FAILED test_pytorch_api.py::test_skipped_layers[io_parallel-Vivado] - TypeError: config_from_pytorch_model() got an unexpected keyword argument 'inputs_channel_last'
FAILED test_pytorch_api.py::test_skipped_layers[io_parallel-Quartus] - TypeError: config_from_pytorch_model() got an unexpected keyword argument 'inputs_channel_last'
FAILED test_pytorch_api.py::test_skipped_layers[io_stream-Vivado] - TypeError: config_from_pytorch_model() got an unexpected keyword argument 'inputs_channel_last'
FAILED test_pytorch_api.py::test_skipped_layers[io_stream-Quartus] - TypeError: config_from_pytorch_model() got an unexpected keyword argument 'inputs_channel_last'

[JanFSchulte]

All test failures the last time around seemed to be related to issues with the tests themselves, which I have mostly fixed. The only change I made was to add missing includes to some Quartus templates to fix compiliation errors when uint_8 was used.

There are currently still some remaining test failures with the case when activations are used in their nn.functionals implementation instead of as classes. Here I can't reproduce the failures in a standalone file, the exact same code that fails in the pytest works fine running in standalone python. Have not figured out how to debug it in those circumstances.

[JanFSchulte]

The tests here finally pass again and I think it is basically ready to merge. One thing to note is that this includes a change to the pytorch config interface giving more options on how to do the channels_last conversion so that it can be either full, transposing both inputs and internal layers, internal, assuming that inputs are already transposed and only transposing internal layers, or off. I developed this at some point, not sure if it was based on a discussion with Vladimir, and included it in here kinda by accident. Let me know if that's desired or should be removed.

[jmitrevs]

Very minor point: hls4ml seems to use single quotes more often (except for doc strings), so I would change "pytorch" to 'pytorch'. There are double quotes used in other places. But truthfully I am not convinced whether we need to change it and run the tests again, since this is so minor.

[jmitrevs]

Is it better to check that pytorch is in the keys or that it is defined and True? What if it is defined and set False? I wonder if using node.get_attr('pytorch') (returns None if not found) or node.get_attr('pytorch', False) is better.

[JanFSchulte]

Right now it will only be defined if a recurring layer is parsed in pytorch, and I can't really envision a situation where we would have code that sets this key that isn't part of the pytorch parser. But I still think you are right that this should be implemented in a more future-proof way and node.get_attr('pytorch', False) is probably the most stringent solution. I will implement it.