Error with nntool with keras recurrent models

[rallen10]

My issue is similar in theme to issue #247, however it results in a completely different error, thus I wanted to open a separate issue.

I am attempting port a custom keras model to GAP8 with nntool. For whatever it's worth, the model is a "neural circuit policy" built with the keras-ncp library which uses a modified AbstractRNNCell.

I can successfully convert my trained model to .tflite using a slightly modified version of h5_to_tflite.py. However, when I attempt to open the .tflite model with nntool, I get the error

Traceback (most recent call last):
  File "/home/ross/miniconda3/envs/hello_ncp/lib/python3.7/site-packages/cmd2/cmd2.py", line 1661, in onecmd_plus_hooks
    stop = self.onecmd(statement, add_to_history=add_to_history)
  File "/home/ross/miniconda3/envs/hello_ncp/lib/python3.7/site-packages/cmd2/cmd2.py", line 2081, in onecmd
    stop = func(statement)
  File "/home/ross/miniconda3/envs/hello_ncp/lib/python3.7/site-packages/cmd2/decorators.py", line 223, in cmd_wrapper
    return func(cmd2_app, args)
  File "/home/ross/Projects/AIIA/crazyflie/gap_sdk/tools/nntool/interpreter/commands/open.py", line 118, in do_open
    self.__open_graph(args)
  File "/home/ross/Projects/AIIA/crazyflie/gap_sdk/tools/nntool/interpreter/commands/open.py", line 92, in __open_graph
    G = create_graph(graph_file, opts=opts)
  File "/home/ross/Projects/AIIA/crazyflie/gap_sdk/tools/nntool/importer/importer.py", line 52, in create_graph
    graph = importer.create_graph(filename, opts)
  File "/home/ross/Projects/AIIA/crazyflie/gap_sdk/tools/nntool/importer/tflite2/tflite.py", line 103, in create_graph
    self._import_tflite_graph(G, model, opts)
  File "/home/ross/Projects/AIIA/crazyflie/gap_sdk/tools/nntool/importer/tflite2/tflite.py", line 154, in _import_tflite_graph
    self._provisional_outputs, opts)
  File "/home/ross/Projects/AIIA/crazyflie/gap_sdk/tools/nntool/importer/tflite2/tflite.py", line 245, in _import_nodes
    node, all_nodes=all_nodes, G=G, opts=opts, importer=self)
  File "/home/ross/Projects/AIIA/crazyflie/gap_sdk/tools/nntool/importer/tflite2/handlers/handler.py", line 65, in handle
    return ver_handle(node, **kwargs)
  File "/home/ross/Projects/AIIA/crazyflie/gap_sdk/tools/nntool/importer/tflite2/handlers/backend/fill.py", line 56, in version_1
    return cls._common(node, **kwargs)
  File "/home/ross/Projects/AIIA/crazyflie/gap_sdk/tools/nntool/importer/tflite2/handlers/backend/fill.py", line 40, in _common
    shape = list(cls._verify_constant(inputs[0]))
  File "/home/ross/Projects/AIIA/crazyflie/gap_sdk/tools/nntool/importer/tflite2/handlers/backend_handler.py", line 63, in _verify_constant
    raise ValueError("expected node %s to be constant input" % inp[0].name)
ValueError: expected node CONCATENATION_0_6 to be constant input
EXCEPTION of type 'ValueError' occurred with message: 'expected node CONCATENATION_0_6 to be constant input'

I am not sure if this is a bug I need to debug within the keras-ncp library or if it is something that needs to be debugged within nntool library, so I am asking in both places in the hopes someone might perceive a simple fix.

[rallen10]

Follow-up: I can produce the same error when trying to use a vanilla LSTM model:

lstm_model = keras.models.Sequential(
        [
            keras.layers.InputLayer(input_shape=(None, 2)),
            keras.layers.LSTM(1, return_sequences=True),
        ]
    )

Therefore the problem does not seem to be associated with the custom NCP model I was using. I am going to update the title of the post to reflect this.

Does nntool not work with LSTMs or RNNs?

[rallen10]

I've also tried SimpleRNN which also results in the same error.

[rallen10]

Further information, I've tried converting to .tflite using both TF2.x and tf.compat.v1 functions; both approaches producing the same error

[sousoux]

Could you send us a failing tflite file please. We support both LSTM and RNN import from TFLITE.

[rallen10]

hello_lstm.zip

Attached is a zip file that contains the tflite file: hello_lstm.tflite. For completeness, I also included the training data (hello_lstm_data.py), the training script (train_hello_lstm.py), the trained h5 model file prior to conversion to tflite (hello_lstm.h5), and the conversion script (h5_to_tflite.py); as well as a conda environment.yml file and resulting conda list (requirements.txt) in order to reproduce the environment I am using.

To reproduce (overwrite the existing .h5 and .tflite files):

conda env create -f environment.yml
conda activate hello_ncp
python train_hello_lstm.py
python h5_to_tflite.py hello_lstm.h5 hello_lstm.tflite True # True=tf.compat.v1, False=tf2
nntool
set debug true
open hello_lstm.tflite

[sousoux]

The TFLITE converter is not converting the model correctly. THe LSTM layer is not being converted to a fused TFLITE LSTM operator. Please take a look at https://www.tensorflow.org/lite/convert/rnn. Particularly the notebook referenced on the page. Using netron you should see no while operators in the TFLITE graph.

[rallen10]

Thank you! This seems to be the right direction. I've successfully converted an LSTM using nntool.