[ONNX] Fails to quantize dynamic input model

[peiwenhuang27]

Environment: Google Colab
LPOT Ver: 1.5.1
ONNX Ver: 1.10.1
ONNXRuntime Ver: 1.8.1

I originally built the model as Keras saved model, then I used tf2onnx to convert the saved model to ONNX format.

My model originally has two inputs of static shapes:

(64, 60, 257)
(64, 257, 60, 1)

which can be successfully quantized with LPOT.

But when I set some dimensions to be dynamic as follows:

(-1, -1, 257)
(-1, 257, -1, 1)

and use it to run quantization with LPOT, the program ends with an assertion error

2021-08-05 09:48:08.018023: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
2021-08-05 09:48:10 [INFO] Getting FP32 model baseline...
tcmalloc: large alloc 1073741824 bytes == 0x5599ab11e000 @  0x7fa8269b5b6b 0x7fa8269d5379 0x7fa75458534c 0x7fa7545822f4 0x7fa75453c7d1 0x7fa7545417b2 0x7fa7543b0b9f 0x7fa75456af3a 0x7fa75456c904 0x7fa754559d64 0x7fa75461e108 0x7fa7545294e7 0x7fa754535190 0x7fa75486e3c8 0x7fa75486eb75 0x7fa7548bd1e9 0x7fa7548e6b38 0x55991200cbf8 0x5599120806f2 0x55991207ac35 0x55991200d73a 0x55991207bd67 0x55991207b235 0x55991200d73a 0x55991207bd67 0x55991207b235 0x55991200d73a 0x55991207bb0e 0x55991200d65a 0x55991207bd67 0x55991200d65a
tcmalloc: large alloc 2147483648 bytes == 0x5599eb11e000 @  0x7fa8269b5b6b 0x7fa8269d5379 0x7fa75458534c 0x7fa7545822f4 0x7fa7545713a8 0x7fa7545290d7 0x7fa754535190 0x7fa75486e3c8 0x7fa75486eb75 0x7fa7548bd1e9 0x7fa7548e6b38 0x55991200cbf8 0x5599120806f2 0x55991207ac35 0x55991200d73a 0x55991207bd67 0x55991207b235 0x55991200d73a 0x55991207bd67 0x55991207b235 0x55991200d73a 0x55991207bb0e 0x55991200d65a 0x55991207bd67 0x55991200d65a 0x55991207bd67 0x55991200d65a 0x55991207bd67 0x55991200db99 0x559912050e79 0x55991200c7b2
2021-08-05 09:51:48 [INFO] Save tuning history to /content/drive/My Drive/Work/model_quantization/lpot_files/onnx/lpot_workspace/2021-08-05_09-47-51/./history.snapshot
2021-08-05 09:51:48 [INFO] FP32 baseline is: [accuracy: 0.2009, duration (seconds): 217.5703]
tcmalloc: large alloc 2147483648 bytes == 0x5599eb11e000 @  0x7fa8269b5b6b 0x7fa8269d5379 0x7fa75458534c 0x7fa7545822f4 0x7fa75453c7d1 0x7fa7545417b2 0x7fa7543b0b9f 0x7fa75456af3a 0x7fa75456c904 0x7fa754559c68 0x7fa7545e65ef 0x7fa7545294e7 0x7fa754535190 0x7fa75486e3c8 0x7fa75486eb75 0x7fa7548bd1e9 0x7fa7548e6b38 0x55991200cbf8 0x5599120806f2 0x55991207ac35 0x55991200d73a 0x55991207bd67 0x55991207b235 0x55991200d73a 0x55991207bd67 0x55991207ac35 0x55991200d73a 0x55991207bd67 0x55991207b235 0x55991200d73a 0x55991207bd67
tcmalloc: large alloc 4294967296 bytes == 0x559a6b11e000 @  0x7fa8269b5b6b 0x7fa8269d5379 0x7fa75458534c 0x7fa7545822f4 0x7fa75453c7d1 0x7fa7545417b2 0x7fa7543b0b9f 0x7fa75456af3a 0x7fa75456c904 0x7fa754559d64 0x7fa754642684 0x7fa75464a70b 0x7fa7545294e7 0x7fa754535190 0x7fa75486e3c8 0x7fa75486eb75 0x7fa7548bd1e9 0x7fa7548e6b38 0x55991200cbf8 0x5599120806f2 0x55991207ac35 0x55991200d73a 0x55991207bd67 0x55991207b235 0x55991200d73a 0x55991207bd67 0x55991207ac35 0x55991200d73a 0x55991207bd67 0x55991207b235 0x55991200d73a
Traceback (most recent call last):
  File "main.py", line 164, in <module>
    q_model = quantize() 
  File "/usr/local/lib/python3.7/dist-packages/lpot/experimental/quantization.py", line 177, in __call__
    self.strategy.traverse()
  File "/usr/local/lib/python3.7/dist-packages/lpot/strategy/strategy.py", line 310, in traverse
    tune_cfg, self.model, self.calib_dataloader, self.q_func)
  File "/usr/local/lib/python3.7/dist-packages/lpot/utils/utility.py", line 200, in fi
    res = func(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/lpot/adaptor/onnxrt.py", line 106, in quantize
    quantizer.quantize_model()
  File "/usr/local/lib/python3.7/dist-packages/lpot/adaptor/ox_utils/onnx_quantizer.py", line 204, in quantize_model
    op_quantizer.quantize()
  File "/usr/local/lib/python3.7/dist-packages/lpot/adaptor/ox_utils/operators/split.py", line 32, in quantize
    self.quantizer.quantize_inputs(node, [0])
  File "/usr/local/lib/python3.7/dist-packages/lpot/adaptor/ox_utils/onnx_quantizer.py", line 759, in quantize_inputs
    self.config[node.name]['activation']['dtype'])
  File "/usr/local/lib/python3.7/dist-packages/lpot/adaptor/ox_utils/onnx_quantizer.py", line 528, in _get_quantize_input_nodes
    of nodes to be quantized are required.".format(input_name))
ValueError: Quantization parameters are not specified for param StatefulPartitionedCall/model/32/Conv2D/SpaceToBatchND_reshape__132:0.In static mode quantization params for inputs and outputs

I also found out by comparing the static shape model and dynamic shape model, that the number of operations in the dynamic model is much higher than the static model, as it adds a lot of reshape/resize related nodes to help deal with the unknown dimension issue. The node that triggers the error StatefulPartitionedCall/model/32/Conv2D/SpaceToBatchND_reshape__132:0 seems to be one of those added for dynamic input purpose.