Reference implementation for onnx operators (python runtime for onnx)

[xadupre]

Description and Motivation

This PR addresses issue #4432. It proposes a reference implementation for onnx operators through a runtime following the same API as onnxruntime. This python runtime can also be used to test function body. Currently, there is no way to check the body of a function validates the backend tests. This can also be used to validate other runtime or add more backend tests.

This work comes from a python runtime I wrote to easily check the onnx model built with a converter. onnxruntime is a fast runtime but it cannot be easily use to debug a model, like display intermediate results. This is another goal of this runtime. The runtime only depends on python and numpy.

from onnx.reference import ReferenceEvaluator

sess = ReferenceEvaluator(onnx_proto)  # type of onnx_proto is ModelProto, FunctionProto, GraphProto or NodeProto.
results = sess.run(None, {"X": a, "A": a, "B": b})
print(results[0])  # display the first result

The current implementation covers 99.4% of the backend tests. It skips the following tests:

            # mismatches
            "test_center_crop_pad_crop_axes_hwc_expanded",  # shapes mismatch with the inlined operator
            "test_col2im_pads",  # mismatch by one value
            # deprecated
            "test_scan_sum",  # deprecated, opset 8 -> not implemented
            "test_scatter_with_axis",  # deprecated, scatter is removed
            "test_scatter_without_axis",  # deprecated, scatter is removed
            # not implemented
            "test__simple_gradient_of_add",  # gradient not implemented
            "test__simple_gradient_of_add_and_mul",  # gradient not implemented

[lgtm-com]

This pull request introduces 8 alerts when merging fce8bad into f2f1408 - view on LGTM.com

new alerts:

• 7 for Unused import
• 1 for Unused local variable

[lgtm-com]

This pull request introduces 7 alerts when merging 5018923 into f2f1408 - view on LGTM.com

new alerts:

• 6 for Unused import
• 1 for Unused local variable

[lgtm-com]

This pull request introduces 7 alerts when merging 4e247d7 into f2f1408 - view on LGTM.com

new alerts:

• 6 for Unused import
• 1 for Unused local variable

[lgtm-com]

This pull request introduces 7 alerts when merging 3682e11 into f2f1408 - view on LGTM.com

new alerts:

• 6 for Unused import
• 1 for Unused local variable

[lgtm-com]

This pull request introduces 7 alerts when merging 7b53b81 into f2f1408 - view on LGTM.com

new alerts:

• 6 for Unused import
• 1 for Unused local variable

[lgtm-com]

This pull request introduces 7 alerts when merging 452253b into f2f1408 - view on LGTM.com

new alerts:

• 6 for Unused import
• 1 for Unused local variable

[lgtm-com]

This pull request introduces 7 alerts when merging 4800f3d into f2f1408 - view on LGTM.com

new alerts:

• 6 for Unused import
• 1 for Unused local variable

[lgtm-com]

This pull request introduces 7 alerts when merging 4a9218a into f2f1408 - view on LGTM.com

new alerts:

• 6 for Unused import
• 1 for Unused local variable

[lgtm-com]

This pull request introduces 7 alerts when merging fd14c7c into f2f1408 - view on LGTM.com

new alerts:

• 6 for Unused import
• 1 for Unused local variable

[lgtm-com]

This pull request introduces 7 alerts when merging d5dd0ef into f2f1408 - view on LGTM.com

new alerts:

• 6 for Unused import
• 1 for Unused local variable

[lgtm-com]

This pull request introduces 7 alerts when merging 8707d24 into f2f1408 - view on LGTM.com

new alerts:

• 6 for Unused import
• 1 for Unused local variable

[lgtm-com]

This pull request introduces 7 alerts when merging b181f44 into f2f1408 - view on LGTM.com

new alerts:

• 6 for Unused import
• 1 for Unused local variable

[lgtm-com]

This pull request introduces 7 alerts when merging 7a68ae8 into f2f1408 - view on LGTM.com

new alerts:

• 6 for Unused import
• 1 for Unused local variable

[lgtm-com]

This pull request introduces 7 alerts when merging 176cbd0 into f2f1408 - view on LGTM.com

new alerts:

• 6 for Unused import
• 1 for Unused local variable

[lgtm-com]

This pull request introduces 6 alerts when merging 504aedd into f2f1408 - view on LGTM.com

new alerts:

• 6 for Unused import

[lgtm-com]

This pull request introduces 7 alerts when merging 4498dcb into f2f1408 - view on LGTM.com

new alerts:

• 7 for Unused import

[lgtm-com]

This pull request introduces 6 alerts when merging 8a1343c into f2f1408 - view on LGTM.com

new alerts:

• 6 for Unused import

[lgtm-com]

This pull request introduces 6 alerts when merging 4677bc4 into f2f1408 - view on LGTM.com

new alerts:

• 6 for Unused import

[lgtm-com]

This pull request introduces 6 alerts when merging 6273ab9 into f2f1408 - view on LGTM.com

new alerts:

• 6 for Unused import

[lgtm-com]

This pull request introduces 6 alerts when merging 1548e7a into f2f1408 - view on LGTM.com

new alerts:

• 6 for Unused import

[lgtm-com]

This pull request introduces 6 alerts when merging acd69ed into f2f1408 - view on LGTM.com

new alerts:

• 6 for Unused import

[lgtm-com]

This pull request introduces 6 alerts when merging 6534f54 into f2f1408 - view on LGTM.com

new alerts:

• 6 for Unused import

[lgtm-com]

This pull request introduces 9 alerts when merging cc8393f into f2f1408 - view on LGTM.com

new alerts:

• 8 for Unused import
• 1 for Unused local variable

[lgtm-com]

This pull request introduces 9 alerts when merging fc68aa5 into f2f1408 - view on LGTM.com

new alerts:

• 8 for Unused import
• 1 for Unused local variable

[lgtm-com]

This pull request introduces 9 alerts when merging 5315f33 into f2f1408 - view on LGTM.com

new alerts:

• 8 for Unused import
• 1 for Unused local variable

[lgtm-com]

This pull request introduces 11 alerts when merging 83163e0 into f2f1408 - view on LGTM.com

new alerts:

• 10 for Unused import
• 1 for Unused local variable

[lgtm-com]

This pull request introduces 15 alerts when merging d2827cd into f2f1408 - view on LGTM.com

new alerts:

• 11 for Unused import
• 3 for Conflicting attributes in base classes
• 1 for Unused local variable

[lgtm-com]

This pull request introduces 15 alerts when merging a4fd14e into f2f1408 - view on LGTM.com

new alerts:

• 11 for Unused import
• 3 for Conflicting attributes in base classes
• 1 for Unused local variable

[lgtm-com]

This pull request introduces 15 alerts when merging c31556f into f2f1408 - view on LGTM.com

new alerts:

• 11 for Unused import
• 3 for Conflicting attributes in base classes
• 1 for Unused local variable

[gramalingam]

nit: "specifications" => "specification"

[xadupre]

done

[jcwchen]

It seems that some tests in reference_evaluator_backend_test.py and reference_evaluator_test.py failed (see CI). One failed Windows CI uses x86 machine and another CI uses old NumPy version 1.16.6. Since other CIs are fine, is it because NumPy behaves slightly differently in Windows x86 machine and old NumPy version 1.16.6?

[xadupre]

It seems that some tests in reference_evaluator_backend_test.py and reference_evaluator_test.py failed (see CI). One failed Windows CI uses x86 machine and another CI uses old NumPy version 1.16.6. Since other CIs are fine, is it because NumPy behaves slightly differently in Windows x86 machine and old NumPy version 1.16.6?

I can either skip the test if numpy is too old and add in the documentation that the reference implementation only supports numpy >= 1.21.5 or fix the test one by one for numpy == 1.16.6. What do you prefer?

[jcwchen]

Although ideally ONNX should always support the minimum required NumPy version (for now it's 1.16.6), I believe sometimes NumPy behaves quite differently in different environments (hardware, version). It might take a lot of time to fix NumPy behavior gap one by one and it's pretty close to the code freeze date (11/18). Thus, skipping them for now is fine to me.

BTW, besides old NumPy version issue (1.16.6), NumPy package for Windows x86 seems to behave differently as well so the test failed in release _win CI for x86 machines.

[jcwchen]

Just speak aloud: to solve this kind of failure, is it possible that we can have loose tolerance when comparing output result from onnx reference and expected?

[xadupre]

We can. I had to do it. There are dictionaries I use to specify different tolerances for every test: https://github.com/onnx/onnx/blob/main/onnx/test/reference_evaluator_backend_test.py#L764.

[gramalingam]

Although ideally ONNX should always support the minimum required NumPy version (for now it's 1.16.6), I believe sometimes NumPy behaves quite differently in different environments (hardware, version). It might take a lot of time to fix NumPy behavior gap one by one and it's pretty close to the code freeze date (11/18). Thus, skipping them for now is fine to me.

BTW, besides old NumPy version issue (1.16.6), NumPy package for Windows x86 seems to behave differently as well so the test failed in release _win CI for x86 machines.

I agree that for this release, doing the simplest is best. Ok with me to see it currently supports only numpy >= 1.21.5, and add support for other numpy versions later on.

[xadupre]

See #4658, I disabled only three tests related to cast into/from bfloat16.

[belm0]

I tried using this, and get an error about ai.onnx.ml having no known functions. I tried compiling onnx with ONNX_ML=1, but it was the same. What am I missing?

>>> import onnx
>>> model = onnx.load_model('foo.onnx')
>>> from onnx.reference import ReferenceEvaluator
>>> sess = ReferenceEvaluator(model)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/.../site-packages/onnx/reference/reference_evaluator.py", line 227, in __init__
    self._init()
  File "/.../site-packages/onnx/reference/reference_evaluator.py", line 312, in _init
    cl = self._load_impl(node)
  File "/.../site-packages/onnx/reference/reference_evaluator.py", line 379, in _load_impl
    raise NotImplementedError(
NotImplementedError: Node type 'Scaler' from domain 'ai.onnx.ml' is unknown, known functions: [].

[gramalingam]

@xadupre

[xadupre]

Is it possible to share the file foo.onnx you used?

[belm0]

Is it possible to share the file foo.onnx you used?

foo.onnx.zip

I didn't expect that the model file was relevant, since the runtime is reporting known functions: [].

[xadupre]

I just tried with the main branch on github and it works. Which version of onnx are you using? The implementation for ml operators were introduced after the last published release on pypi.