TF2 SavedModel to ONNX conversion fails for ConvLSTM2D layer

[rttus]

Describe the bug
I am trying to convert a TF2 SavedModel to ONNX using

python3 -m tf2onnx.convert --saved-model <model_dir_path> --opset 13 --output <output_file_name>

The conversion fails with the below stack trace

WARNING - ONNX Failed to infer shapes and dtypes for [functional_5/functional_3/conv_lst_m2d/while/TensorArrayV2Read/TensorListGetItem__17, type: Unsqueeze]
Traceback (most recent call last):
File "/.local/lib/python3.6/site-packages/tf2onnx/schemas.py", line 157, in infer_onnx_shape_dtype
inferred_model = shape_inference.infer_shapes(model_proto)
File "/.local/lib/python3.6/site-packages/onnx/shape_inference.py", line 35, in infer_shapes
inferred_model_str = C.infer_shapes(model_str, check_type)
RuntimeError: input 1 is out of bounds

ERROR - Failed to convert node 'functional_5/functional_3/conv_lst_m2d_2/while/BiasAdd_3' (fct=<bound method ConvOp.version_11 of <class 'tf2onnx.onnx_opset.nn.ConvOp'>>)
'OP=Conv\nName=functional_5/functional_3/conv_lst_m2d_2/while/BiasAdd_3\nInputs:\n\tfunctional_5/functional_3/conv_lst_m2d_2/while/TensorArrayV2Read/TensorListGetItem__72:0=Squeeze, None, 1\n\tfunctional_5/functional_3/conv_lst_m2d_2/while/split:3=Split, [1, 1, 128, 10], 1\n\tfunctional_5/functional_3/conv_lst_m2d_2/while/split_2:3=Split, [10], 1\nOutpus:\n\tfunctional_5/functional_3/conv_lst_m2d_2/while/BiasAdd_3:0=[1, 11, 20, 10], 1'
Traceback (most recent call last):
File "/.local/lib/python3.6/site-packages/tf2onnx/tfonnx.py", line 287, in tensorflow_onnx_mapping
func(g, node, **kwargs)
File "/.local/lib/python3.6/site-packages/tf2onnx/onnx_opset/nn.py", line 367, in version_11
cls.version_1(ctx, node, **kwargs)
File "~/.local/lib/python3.6/site-packages/tf2onnx/onnx_opset/nn.py", line 353, in version_1
if len(input_shape) == spatial + 1:
TypeError: object of type 'NoneType' has no len()

Urgency
None

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 18.04.5 LTS
• Tensorflow Version: 2.3.0
• ONNX Version: 1.8.0
• tf2onnx Version: 1.7.2/995bd6
• Python version: 3.6

To Reproduce
I will not be able to share the model or weights.

Expected behavior
Generate a ONNX file in the location given by --output param

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here. If the issue is about a particular model, please share the model details as well to facilitate debugging.

[TomWildenhain-Microsoft]

Hi @rttus, tf2onnx does not support opset 13 yet. Try using opset 12.

[TomWildenhain-Microsoft]

The other issue is that it looks like the input to your conv op has unknown rank. Can you share the model you are converting?

[rttus]

@TomWildenhain-Microsoft

Thanks for your response. I was dubious about opset=13 so i had tried different opsets before. However, with a different opset=12, i see similar error about dtype and shape during optimization loop but this time it creates a ONNX file at the end.
Wrt sharing the model, i wouldnt be able to share the actual model. However, i tried converting a sample code from https://keras.io/examples/vision/conv_lstm/

I still see very similar issue. Below is the stack trace

WARNING - ONNX Failed to infer shapes and dtypes for [trans_moved_from_loop_body__1030, type: Transpose]
Traceback (most recent call last):
File ".local/lib/python3.6/site-packages/tf2onnx/schemas.py", line 157, in infer_onnx_shape_dtype
inferred_model = shape_inference.infer_shapes(model_proto)
File ".local/lib/python3.6/site-packages/onnx/shape_inference.py", line 35, in infer_shapes
inferred_model_str = C.infer_shapes(model_str, check_type)
RuntimeError: Shape inference error(s): (op_type:Transpose, node name: trans_moved_from_loop_body__1030): [TypeInferenceError] Element type of input 0 unknown

WARNING - Failed to apply loop_optimizer
Traceback (most recent call last):
File ".local/lib/python3.6/site-packages/tf2onnx/optimizer/init.py", line 54, in optimize_graph
graph = opt.optimize(current) or graph
File ".local/lib/python3.6/site-packages/tf2onnx/optimizer/optimizer_base.py", line 42, in optimize
graph.update_proto()
File ".local/lib/python3.6/site-packages/tf2onnx/graph.py", line 814, in update_proto
node.update_proto(external_tensor_storage)
File ".local/lib/python3.6/site-packages/tf2onnx/graph.py", line 388, in update_proto
external_tensor_storage=external_tensor_storage)
File ".local/lib/python3.6/site-packages/tf2onnx/graph.py", line 1041, in make_graph
self.delete_unused_nodes(self.outputs)
File ".local/lib/python3.6/site-packages/tf2onnx/graph.py", line 1488, in delete_unused_nodes
self.reset_nodes(related_nodes)
File ".local/lib/python3.6/site-packages/tf2onnx/graph.py", line 728, in reset_nodes
raise ValueError("graph output " + o + " not exist")
ValueError: graph output sequential/conv_lst_m2d/while/Identity_4:0 not exist
INFO - After optimization: Cast -4 (28->24), Const -55 (142->87), Identity -93 (97->4), Reshape +5 (0->5), Transpose -33 (114->81)

Successfully converted TensorFlow model test_model to ONNX
INFO - ONNX model is saved at test.onnx

Now, the question is does ONNX support conversion for models with ConvLSTM2D layer?

[guschmue]

In general yes, The warning about the loop_optimizer is not expected and we should find out what ti complains about but if it fails it only impacts this specific optimizer.

[TomWildenhain-Microsoft]

Did the model work for you?

[rttus]

So, I use the ONNX converted model to generate a tensorrt engine. Now when i use this converted model, tensorrt optimization fails. So, i am not sure if it the ONNX file itself or tensorrt engine conversion that has problem.

In short, I havent gotten my model working yet.

[TomWildenhain-Microsoft]

Can you try the model in onnxruntime? That will help us determine if it is the model or TensorRT.

[rttus]

Thanks for the hint. So, i did try running the ONNX converted sample model from here https://keras.io/examples/vision/conv_lstm/
using ONNX runtime using below script

`import numpy
import onnxruntime as rt

sess = rt.InferenceSession("test.onnx")
input_name = sess.get_inputs()[0].name

pred_onx = sess.run(None, {input_name: numpy.random.rand(1, 1, 40, 40, 1).astype(numpy.float32)})[0]
print(pred_onx)`

gives me below error

[E:onnxruntime:, sequential_executor.cc:318 Execute] Non-zero status code returned while running Conv node. Name:'sequential/conv_lst_m2d_1/while/BiasAdd_1' Status Message: X num_dims does not match W num_dims. X: {1,40,1,40,1} W: {40,40,3,3}
[E:onnxruntime:, sequential_executor.cc:318 Execute] Non-zero status code returned while running Loop node. Name:'StatefulPartitionedCall/sequential/conv_lst_m2d_1/while_loop' Status Message: Non-zero status code returned while running Conv node. Name:'sequential/conv_lst_m2d_1/while/BiasAdd_1' Status Message: X num_dims does not match W num_dims. X: {1,40,1,40,1} W: {40,40,3,3}
Traceback (most recent call last):
File "test_onnx_inference.py", line 9, in
pred_onx = sess.run(None, {input_name: numpy.random.rand(1, 1, 40, 40, 1).astype(numpy.float32)})[0]
File "local/lib/python3.6/dist-packages/onnxruntime/capi/session.py", line 123, in run
return self._sess.run(output_names, input_feed, run_options)
onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : Non-zero status code returned while running Loop node. Name:'StatefulPartitionedCall/sequential/conv_lst_m2d_1/while_loop' Status Message: Non-zero status code returned while running Conv node. Name:'sequential/conv_lst_m2d_1/while/BiasAdd_1' Status Message: X num_dims does not match W num_dims. X: {1,40,1,40,1} W: {40,40,3,3}

[TomWildenhain-Microsoft]

Well at least onnxruntime gives a nicer error message... I'll take a look. There are a few pending issues involving loops.

[rttus]

Sounds good. Thank you very much.

[TomWildenhain-Microsoft]

Can you upload a copy of the saved model? That will save me some time.

[guschmue]

https://github.com/keras-team/keras-io/blob/master/examples/vision/conv_lstm.py

[TomWildenhain-Microsoft]

This PR should solve your issue:
#1244

I still need to look into the failing optimizers.