ONNX to HLS conversion tool

[vloncar]

Initial work on supporting ONNX.

Currently supported layers:

• Dense/Linear
• BatchNorm
• Conv1d and Conv2d
• Various activations

Tested with models converted from Keras and PyTorch

[vloncar]

This is now ready for review.

[nhanvtran]

Thanks @vloncar!

I assigned @thesps as the reviewer, but please let me know if this is okay -- we can get another reviewer instead/in addition

[thesps]

Environment

In onnx-to-hls.py:
from onnx import...
onnx is not in the hls4ml conda environment, add to the environment? (Which version?)
Same goes for onnxmltools in keras-to-onnx.py

Testing

onnx-to-hls

Ran onnx-to-hls with example config file and model.
The conversion ran successfully, and the produced HLS project simulates and synthesizes correctly.

keras-to-onnx

Not working for me. I try to run the conversion on one of the keras example models and get an error with the onnx optimizer step.
Trying:

path='../../keras-to-hls/example-keras-model-files/'
python keras-to-onnx.py -m $path/KERAS_3layer.json -w $path/KERAS_3layer_weights.h5 -o ./KERAS_3layer

The error is:

Traceback (most recent call last):
  File "keras-to-onnx.py", line 45, in <module>
    keras_to_onnx()
  File "keras-to-onnx.py", line 39, in keras_to_onnx
    onnx_model = optimizer.optimize(onnx_model, passes)
  File "/Users/sionisummers/miniconda3/envs/hls4ml-env-onnx/lib/python3.6/site-packages/onnx/optimizer.py", line 55, in optimize
    optimized_model_str = C.optimize(model_str, passes)
IndexError: unordered_map::at: key not found

Any ideas? Is this an onnx version/install problem?
Running, from Python:

from onnx import optimizer
optimizer.get_available_passes()

I see all of the three passes that should be performed. This seems like a problem in my setup or onnx rather than the converter...
I get the same behaviour with any of the appropriate keras example models.

[vloncar]

The optimizers we need are introduced in ONNX v1.4.0. conda install onnx complained about some version incompatibilities, and had only version 1.2.1 when I tested it. You can use pip within the conda environment to install the version you want. Version of onnxmltools should match the version of onnx package. That being said, onnxmltools version 1.4.x makes a mess out of Flatten layer which it didn't do in version 1.3.

[thesps]

Okay. I think we agree that the keras-to-onnx converter is not the primary responsibility of hls4ml, but it's a useful utility. More documentation, in the form of a recipe to translate a Keras model to Onnx would be useful there.

I tried to run onnx-to-hls on all of the example-onnx-model-files, and I find that for some models the conversion is unsuccessful.

===========================================================================================
===========================================================================================
Model: conv2d_keras
Loading configuration from TEST/onnx-config-conv2d_keras.yml
Traceback (most recent call last):
  File "onnx-to-hls.py", line 407, in <module>
    main()
  File "onnx-to-hls.py", line 213, in main
    raise Exception('ERROR: Unsupported operation type: {}'.format(operation.op_type))
Exception: ERROR: Unsupported operation type: MatMul
===========================================================================================
===========================================================================================

This one looks deliberate, the operation is not supported?

===========================================================================================
===========================================================================================
three_layer_pytorch
Loading configuration from TEST/onnx-config-three_layer_pytorch.yml
Input shape: [789444, 16]
Topology:
Layer name: Gemm1, layer type: Dense, current shape: [789444, 64]
Layer name: Relu2, layer type: Activation, current shape: [789444, 64]
Layer name: Gemm3, layer type: Dense, current shape: [789444, 32]
Layer name: Relu4, layer type: Activation, current shape: [789444, 32]
Layer name: Gemm5, layer type: Dense, current shape: [789444, 32]
Layer name: Relu6, layer type: Activation, current shape: [789444, 32]
Layer name: Gemm7, layer type: Dense, current shape: [789444, 5]
Layer name: Softmax8, layer type: Activation, current shape: [789444, 5]
Traceback (most recent call last):
  File "onnx-to-hls.py", line 407, in <module>
    main()
  File "onnx-to-hls.py", line 404, in main
    write_hls(hls_model)
  File "/Users/sionisummers/Work/hls4ml/hls4ml-onnx/onnx-to-hls/../hls-writer/hls_writer.py", line 327, in write_hls
    write_project_cpp(model)
  File "/Users/sionisummers/Work/hls4ml/hls4ml-onnx/onnx-to-hls/../hls-writer/hls_writer.py", line 144, in write_project_cpp
    func = layer.function_cpp()
  File "/Users/sionisummers/Work/hls4ml/hls4ml-onnx/onnx-to-hls/../hls-writer/hls_model.py", line 323, in function_cpp
    params = self._default_function_params()
  File "/Users/sionisummers/Work/hls4ml/hls4ml-onnx/onnx-to-hls/../hls-writer/hls_model.py", line 265, in _default_function_params
    params['input_t'] = self.get_input_variable().type
  File "/Users/sionisummers/Work/hls4ml/hls4ml-onnx/onnx-to-hls/../hls-writer/hls_model.py", line 193, in get_input_variable
    return self.model.get_layer_output_variable(self.inputs[0])
  File "/Users/sionisummers/Work/hls4ml/hls4ml-onnx/onnx-to-hls/../hls-writer/hls_model.py", line 89, in get_layer_output_variable
    return self.output_vars[output_name]
KeyError: '0'
===========================================================================================
===========================================================================================

All of the pytorch conversions exit with this same error. Is this an issue with the models or onnx-to-hls?

[vloncar]

The latest commits fix the issue with pytorch models. Thanks for reviewing @thesps!

[thesps]

Okay, sorry for the delay in my review. I have moved onto testing the HLS projects generated by onnx-to-hls.

I notice small differences in CSIM between three_layer_bn_keras (onnx model) and KERAS_3layer_batch_norm (original keras model). I just generated some random test vectors and ran them through CSIM for each model. I was using the same ap_fixed<16, 6> type for both as per the default settings.
For example, with input [1] I get:
onnx: [0.550781 , 0.123047 , 0.0898438, 0.125 , 0.121094 ]
keras: [0.550781 , 0.120117 , 0.0878906, 0.125977 , 0.120117 ]

Conversely using the same test data (it's 100 samples of 16 features) I get exact agreement with the three_layer_keras and KERAS_3layer (so the equivalent models without batch normalization). So it looks like there is a discrepancy in the conversion of the batch normalization layer. Indeed, it looks like the parameters for the batch normalization layers differ between the two. Notably the mean for every neuron is 0 for the onnx-to-hls converted model.

[1] input vector [-0.5862158763598992,-0.11333189695183554,-0.24251356953464542,-0.9882633084123655,0.7093189801948572,0.9702835717742708,0.1508570909459832,0.6719363341909363,-0.6062092074509522,0.055040551319794906,0.5682738089992081,0.10445195570702803,-0.73040007874425,0.3389607995537134,0.8240645541166507,-0.9642886120148126]

[thesps]

We talked some more offline, and I'm satisfied that the onnx model version of BatchNormalization is slightly optimized compared to the keras-to-hls version. When converting the keras model to onnx the mean / scale is folded into the bias. In the keras version this means some extra rounding is performed in the FPGA so the results can differ slightly. This optimization can be introduced to the keras-to-hls, and another PR will follow.