ONNX release 1.12.0

[etiotto]

We are releasing ONNX 1.12.0. A release branch is created (https://github.com/onnx/onnx/tree/rel-1.12.0). The planned release date is the week of 2022-06-06.

It is important to integrate ONNX release branch into ORT ASAP so that any issues and incompatibilities can be detected and resolved before the ONNX release. Please follow instructions at (https://github.com/microsoft/onnxruntime/blob/master/docs/How_To_Update_ONNX_Dev_Notes.md) to integrate with the ONNX release branch. Please implement CPU kernels for new and updated ONNX ops. A list of new and updated ops can be found at (https://github.com/onnx/onnx/wiki/Logistics-for-ONNX-Release-1.12.0).

Changes in ONNX are documented in Logistics-for-ONNX-Release-1.12.0 wiki (or in the 1.12.0 release note).

In case a bug in ONNX is detected during integration of ONNX 1.12.0, please contact the ONNX Release Manager @etiotto so that the bug is fixed in the ONNX release branch for the Integration to continue.

[snnn]

The last time it took 3 months. I hope you don't need to wait on this.

[RyanUnderhill]

Just a note that the release branch/release date above is showing the previous release.

[garymm]

@snnn asking for help on these instructions from the "How to Update ONNX" page:

please tell Changming deploying the new test data along with other test models to our CI build machines

I'm telling you :-) LMK if there's anything I need to do.

manually queue a build for every packaging pipeline for your branch.

How do I do that?

[snnn]

Below are all the pipelines. You can find them at https://aiinfra.visualstudio.com/Lotus/_build

Pipeline name	filename
DML Nuget Pipeline	nuget/gpu-esrp-pipeline.yml
Linux GPU(cuda10.2) Validation Pipeline	linux-gpu-cuda-11-pipeline.yml
Linux Multi GPU CI Pipeline	linux-multi-gpu-ci-pipeline.yml
Linux Multi GPU TensorRT CI Pipeline	linux-multi-gpu-tensorrt-ci-pipeline.yml
Npm Packaging Pipeline	npm-packaging-pipeline.yml
Nuget WindowsAI Pipeline	nuget/templates/windowsai.yml
OLive-pipeline	py-package-build-pipeline.yml
ONNX Runtime Web Pipeline	web-packaging-pipeline.yml
Python Packaging Pipeline (Training CPU)	orttraining-py-packaging-pipeline-cpu.yml
Python Packaging Pipeline (Training Cuda 10.2)	orttraining-py-packaging-pipeline-cuda102.yml
Python Packaging Pipeline (Training Cuda 11.3)	orttraining-py-packaging-pipeline-cuda113.yml
Python Packaging Pipeline (Training Cuda 11.5)	orttraining-py-packaging-pipeline-cuda115.yml
Python packaging pipeline	py-packaging-pipeline.yml
Windows GPU(cuda10.2) Validation Pipeline	win-gpu-cuda-10-2-pipeline.yml
Zip-Nuget-Java-Nodejs Packaging Pipeline	c-api-noopenmp-packaging-pipelines.yml
onnxruntime-ios-packaging-pipeline	mac-ios-packaging-pipeline.yml

You can also use https://github.com/microsoft/azure-devops-python-api to write a script for doing this. I had one, but I couldn't find it anymore.

[garymm]

Finally got the tests to run. Lots of failures: https://dev.azure.com/onnxruntime/onnxruntime/_build/results?buildId=658069&view=logs&j=7536d2cd-87d4-54fe-4891-bfbbf2741d83&t=47129eb0-b626-5be0-6e77-09fcb6185efd

Will dig into them tomorrow.

[garymm]

Status update before I head off for the weekend:

I figured out how to run the ONNX conformance tests via the nodeJS API. The tests that were initially failing were all for newly added ops.

This is what I've been using to build and test:

./build.sh --cmake_generator Ninja --config Debug --skip_submodule_sync --parallel --enable_onnx_tests --enable_transformers_tool_test --build_nodejs --update --build
cd js/node
npm test -- --parallel --grep 'test_(blackmanwindow|blackmanwindow|dft|hammingwindow|hannwindow|layer_normalization|melweightmatrix|stft)'

After registering op implementations in #11733, the remaining test failures are in this file:
test-out.txt

Some of the actual vs expected are quite close and I am hoping we can configure the test runner to ignore the diffs.
Some of them are quite far and I fear there're bugs in either ORT or the function defs or the test data.

CC @xadupre @smk2007

[snnn]

@fs-eire any comment to the above?

[smk2007]

Status update before I head off for the weekend:

I figured out how to run the ONNX conformance tests via the nodeJS API. The tests that were initially failing were all for newly added ops.

This is what I've been using to build and test:

./build.sh --cmake_generator Ninja --config Debug --skip_submodule_sync --parallel --enable_onnx_tests --enable_transformers_tool_test --build_nodejs --update --build
cd js/node
npm test -- --parallel --grep 'test_(blackmanwindow|blackmanwindow|dft|hammingwindow|hannwindow|layer_normalization|melweightmatrix|stft)'

After registering op implementations in #11733, the remaining test failures are in this file: test-out.txt

Some of the actual vs expected are quite close and I am hoping we can configure the test runner to ignore the diffs. Some of them are quite far and I fear there're bugs in either ORT or the function defs or the test data.

CC @xadupre @smk2007

Hi, I took a look.

Window Functions Tests (Hann, Hamming, Blackman)
The issue here is that we added the periodic attribute to the function late in the CR process.
Now the tests have ONNX tests have 2 issues.

1. They are generating functions that do not take into account the periodic attribute in the function definition.
2. The default behavior in ONNX runtime has not been updated to match, and is missing the periodic attribute the contrib op definition always generated the "periodic" (window size instead of window size -1 ) window and not the "symmetric" window. The ONNX test has a few issues in the window functions (constants are incorrectly set).

DFT Tests
The DFT seems like it just needs to have precision in the tests updated. Where do I do that?

STFT Tests
The STFT test is also wrong because it hasnt taken into consideration that the default output will be onesided, and so it is made the expected dimension 16, instead of half+1 (9). I can update this test as well.

I added this PR to address: onnx/onnx#4249

[gramalingam]

@wschin : are the layer-normalization error differences expected? (Please see: https://github.com/microsoft/onnxruntime/files/8836128/test-out.txt ). I wonder if there is some subtle difference in the ONNX reference implementation/function and actual? Error margin is around 1% but more in one case. It also looks like there is a shape error in one case?

[garymm]

After incorporating onnx/onnx#4249, I still see these failures (there are other failures but I don't expect those to have been fixed by the PR):

test_blackmanwindow_symmetric_expanded
[ERR_ASSERTION]: Expected values to be strictly deep-equal:
+ actual - expected

  [
+   9
-   10
  ]
      + expected - actual

       [
      -  9
      +  10
       ]

test_hammingwindow_symmetric_expanded
[ERR_ASSERTION]: Expected values to be strictly deep-equal:
+ actual - expected

  [
+   9
-   10
  ]
      + expected - actual

       [
      -  9
      +  10
       ]

test_hannwindow_symmetric_expanded
[ERR_ASSERTION]: Expected values to be strictly deep-equal:
+ actual - expected

  [
+   9
-   10
  ]
      + expected - actual

       [
      -  9
      +  10
       ]
      
test_stft
[ERR_ASSERTION]: actual[269]=0.00008392586460104212, expected[269]=-12.363648414611816

@smk2007 @xadupre can you PTAL?

[gramalingam]

From what Sheil mentioned, the ORT implementation also needs a change, to support the newly added "periodic" attribute?

[garymm]

@gramalingam I don't think that should apply to the _expanded tests which expand the function body before passing it to ORT. Please correct me.

[smk2007]

@gramalingam I don't think that should apply to the _expanded tests which expand the function body before passing it to ORT. Please correct me.

@gramalingam @garymm
Window functions fix
Looked into it. For the symmetric case, the functions are using "Size_FP" when they should be using the original size passed in ("Periodic_Size_FP") to generate the range of values...

STFT fix
The error was in the test again, and is related to incorrect slicing of the input signal. Effectively the input signal was being sliced, and then the dft was being performed, rather than vice-verse. I have validated the output from the onnx test with the ORT implementation:

The fix for STFT has been added to & the fixes for the symmetric tests are here: onnx/onnx#4256

[garymm]

These are different tests right? test_dft vs test_stft?

Oops, deleted comment

[garymm]

There's currently no way to specify a tolerance for a specific test. I'm looking into adding that.
It'd be ideal if we could find some small change to the implementation to make the output match the reference output more closely so we don't have to relax the tolerance, so if you have any ideas there LMK.

[smk2007]

There's currently no way to specify a tolerance for a specific test. I'm looking into adding that. It'd be ideal if we could find some small change to the implementation to make the output match the reference output more closely so we don't have to relax the tolerance, so if you have any ideas there LMK.

Not sure here, taking a look.

[garymm]

I put up a PR to support per-test tolerances: #11775

Does anyone know if the C++ onnx_test_runner runs in CI? If so then I'll need to update the PR to cover the C++ as well. So far I haven't found it actually being run in CI.

[wschin]

Sorry. My original test script has a bug. This PR, onnx/onnx#4263, should unblock layer normalization tests.