-
Notifications
You must be signed in to change notification settings - Fork 230
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
FINN does not handle a model with concatenation layer correctly #853
Comments
Thanks @williamliao28 for turning this into an issue, Could you also let me know which version of FINN you're using? |
It would be best to use the provided template: https://github.com/Xilinx/finn/blob/main/.github/ISSUE_TEMPLATE/bug_report.md |
@auphelia I have edited it using the template. I am using the |
Hi @williamliao28 , I have tested your use case with the current There was one line in the script I needed to change to make it run with You might have seen that we moved to QONNX export only (#850), with the current |
I've inspected the generated .onnx files in a bit more detail and actually the switch to The error is coming from the datatype inference for Concat. Concat is a standard onnx layer and so the FINN datatype doesn't get propagated correctly, I will look into a possible patch for that.
The problem is that for the transformation RoundAndClipThresholds the input datatype needs to be an integer. But for MultiThreshold_6 the input datatype doesn't have the correct type associated because the datatype inference doesn't get correctly propagated through the standard concat layer. After converting it to StreamingConcat, the data type inference works correctly and the floating point thresholds of MultiThreshold_6 can be rounded to integer values, which then corrects the rest of the finn-flow. Since I suggested to move to |
Hi @williamliao28 , Thanks for raising this issue! I will link this conversation in the GitHub discussion you started initially. We merged a patch in qonnx and updated the latest qonnx version in One remark because I had a look at your generated files, you should call a few additional streamlining functions to get rid of the transpose nodes (e.g. AbsorbConsecutiveTransposes) and also to make the necessary data layout conversions, like MakeMaxPoolNHWC. |
name: Bug report
about: Something isn't working as expected
title: 'FINN does not handle a model with concatenation layer correctly'
labels: bug
assignees: ''
Prerequisites
Quick summary
FINN does not handle a model with concatenation layer correctly.
Details
Any suggestions are welcomed. I have tried the following experiments for this issue.
First, I have looked into the source code of the
InferConcatLayer()
function (line 1678 to 1737 inconvert_to_hls_layers.py
). I think this function is used to convert theConcat
ONNX layer to the HLS layerStreamingConcat
. I found that it already calls InferDataTypes() on line 1736. So is it necessary to call anotherInferDataTypes()
again after callingInferConcatLayer()
?Second, I try to call the
InferConcatLayer()
function beforeInferBinaryMatrixVectorActivation()
andInferQuantizedMatrixVectorActivation()
. However, I get the following error message:AssertionError: Thresholds in MatrixVectorActivation_MatMul_4 can't be expressed with type INT10
. The traceback message is included in the screenshot posted below.Steps to Reproduce
I include two example python codes in the links below which may be useful for reproducing the error. The file
simple_test.py
contains the PyTorch module defining the model with Brevitas quantization. The filesimple_test_finn_synthesis.py
is the code I run inside the FINN docker and this one produce the error message described above. The filesimple_test_cifar10_8bit_2epoch.onnx
is a pertained model which can be used directly as the input forsimple_test_finn_synthesis.py
.simple_test.py
simple_test_finn_synthesis.py
simple_test_cifar10_8bit_2epoch.onnx
bash ./run_docker.sh
simple_test_cifar10_8bit_2epoch.onnx
under the directorybuild/quant_model/simple_test_cifar10/lr0.02
python simple_test_finn_synthesis.py
Expected behavior
The code
simple_test_finn_synthesis.py
should be run without any errors.Actual behavior
Running
python simple_test_finn_synthesis.py
generates the error described above.Originally posted by @williamliao28 in #845
The text was updated successfully, but these errors were encountered: