Update BatchNorm to specify population variance #3402
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: neginraoof <neginmr@utexas.edu>
Signed-off-by: neginraoof <neginmr@utexas.edu>
askhade
approved these changes
Apr 7, 2021
jcwchen
approved these changes
Apr 7, 2021
jcwchen
reviewed
Apr 7, 2021
|
|
||
| Notice that ReduceVar refers to the population variance, and it equals to | ||
| sum(sqrd(x_i - x_avg)) / N | ||
| where N is the population size (this formula does not use sample size N - 1). |
There was a problem hiding this comment.
Don't forget to consider the comment here: #3406 (comment)
gramalingam
approved these changes
Apr 7, 2021
|
During mixed precision training, input_mean, input_var and running_mean and running_var are usually kept in fp32, while X and Y can be in fp16. see https://docs.nvidia.com/deeplearning/performance/mixed-precision-training/index.html#lossscaling We need to introduce a different type for input_mean, input_var and running_mean and running_var. |
SherlockNoMad
suggested changes
Apr 9, 2021
|
The corresponding cherry pick change has been already checked in release branch so we should not make any more changes to this PR... any changes should be made in a new PR |
neginraoof
added a commit
to neginraoof/onnx
that referenced
this pull request
Apr 22, 2021
* Specify population variance Signed-off-by: neginraoof <neginmr@utexas.edu> * Test file Signed-off-by: neginraoof <neginmr@utexas.edu> * Doc update Signed-off-by: neginraoof <neginmr@utexas.edu> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Signed-off-by: neginraoof <neginmr@utexas.edu>
gramalingam
added a commit
that referenced
this pull request
May 28, 2021
* Correct broken test url in ONNX Release doc (#3387) * fix broken url Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * change with valid url Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> Signed-off-by: neginraoof <neginmr@utexas.edu> * Implement NonZero shape inference (#3364) * Implement NonZero shape inference Signed-off-by: impactaky <impactaky@users.noreply.github.com> * Add shape inference code to NonZero-9 Signed-off-by: impactaky <impactaky@users.noreply.github.com> * Add type annotation Signed-off-by: impactaky <impactaky@users.noreply.github.com> Co-authored-by: impactaky <impactaky@users.noreply.github.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Michał Karzyński <4430709+postrational@users.noreply.github.com> Signed-off-by: neginraoof <neginmr@utexas.edu> * Adding optional type Signed-off-by: neginraoof <neginmr@utexas.edu> * Fix broken link to logo image. (#3405) Signed-off-by: Ewa21 <ewa.tusien@intel.com> Signed-off-by: neginraoof <neginmr@utexas.edu> * Adding helper and helper tests Signed-off-by: neginraoof <neginmr@utexas.edu> * Adding numpy helper and tests Signed-off-by: neginraoof <neginmr@utexas.edu> * Update BatchNorm to specify population variance (#3402) * Specify population variance Signed-off-by: neginraoof <neginmr@utexas.edu> * Test file Signed-off-by: neginraoof <neginmr@utexas.edu> * Doc update Signed-off-by: neginraoof <neginmr@utexas.edu> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Signed-off-by: neginraoof <neginmr@utexas.edu> * Add new type constrains for variance and mean in BatchNorm (#3415) * update def.cc first Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * update docs Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * drop but Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> Signed-off-by: neginraoof <neginmr@utexas.edu> * Fix Batchnorm type inference for mean and variance and add a test (#3432) * fix batchnorm type inference and add a test Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * sync X, scale and b as same type Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> Signed-off-by: neginraoof <neginmr@utexas.edu> * Fix url to logo image. (#3421) Signed-off-by: Ewa21 <ewa.tusien@intel.com> Co-authored-by: Chun-Wei Chen <jacky82226@gmail.com> Signed-off-by: neginraoof <neginmr@utexas.edu> * Clarify CONTRIBUTING.md documentation generation steps (#3443) * Clarify document generation steps Signed-off-by: Dwayne Robinson <dwayner@microsoft.com> * Add note about stat_coverage.py, which is undocumented but also essential for the checkin to succeed Signed-off-by: Dwayne Robinson <dwayner@microsoft.com> Co-authored-by: Chun-Wei Chen <jacky82226@gmail.com> Signed-off-by: neginraoof <neginmr@utexas.edu> * Add more automated verification for release packages (#3401) * windows first: test ORT, tensorflow-onnx, numpy, protobuf Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * git clone tensorflow-onnx Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * do not verify tensorflow on x86 Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * remove tensorflow-onnx verification; apply it on Linux and Mac Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * trigger Linux and Mac Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * not cd onnx Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * correct onnx path Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * correct command path Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * update right wheel name Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * remove main trigger before merge Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Signed-off-by: neginraoof <neginmr@utexas.edu> * Fix ort failure in CI and add manually trigger (#3444) * fix ort failure and add manually trigger Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> Signed-off-by: neginraoof <neginmr@utexas.edu> * Merging release 1.9.0 branch into master (#3445) * Merge release 1.9.0 branch into master Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> Signed-off-by: neginraoof <neginmr@utexas.edu> * [Dup] Update spec for Convtranspose to make it sync (#3440) * update documents Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> Signed-off-by: neginraoof <neginmr@utexas.edu> * Fix for comments part 1 Signed-off-by: neginraoof <neginmr@utexas.edu> * Fix for reviews round 2 Signed-off-by: neginraoof <neginmr@utexas.edu> * Fix for comments part 3 Signed-off-by: neginraoof <neginmr@utexas.edu> * Signed-off-by: neginraoof <neginmr@utexas.edu> * Signed-off-by: neginraoof <neginmr@utexas.edu> * Fix merge Signed-off-by: neginraoof <neginmr@utexas.edu> * Typo Signed-off-by: neginraoof <neginmr@utexas.edu> * Fix for tests part 1 Signed-off-by: neginraoof <neginmr@utexas.edu> * Fix tests part 2 Signed-off-by: neginraoof <neginmr@utexas.edu> * Fix tests part 3 Signed-off-by: neginraoof <neginmr@utexas.edu> * Signed-off-by: neginraoof <neginmr@utexas.edu> * Signed-off-by: neginraoof <neginmr@utexas.edu> * Serialization changes Signed-off-by: neginraoof <neginmr@utexas.edu> * Add Additional bfloat16 Support to Pow (#3412) * Add bfloat16 to Pow 2nd argument and bump to v15 Signed-off-by: Ian Bearman <ianb@microsoft.com> * update docs Signed-off-by: Ian Bearman <ianb@microsoft.com> Co-authored-by: G. Ramalingam <grama@microsoft.com> Signed-off-by: neginraoof <neginmr@utexas.edu> * Typo Signed-off-by: neginraoof <neginmr@utexas.edu> * Signed-off-by: neginraoof <neginmr@utexas.edu> * Fix checker Signed-off-by: neginraoof <neginmr@utexas.edu> * Signed-off-by: neginraoof <neginmr@utexas.edu> * Signed-off-by: neginraoof <neginmr@utexas.edu> * Signed-off-by: neginraoof <neginmr@utexas.edu> * make_optional_value_info Signed-off-by: neginraoof <neginmr@utexas.edu> * fix comments Signed-off-by: neginraoof <neginmr@utexas.edu> * fix 'types' in optional proto Signed-off-by: neginraoof <neginmr@utexas.edu> * fix for helper Signed-off-by: neginraoof <neginmr@utexas.edu> * flake8 Signed-off-by: neginraoof <neginmr@utexas.edu> * Removed dtype list Signed-off-by: neginraoof <neginmr@utexas.edu> * Adding test for make_optional sequence Signed-off-by: neginraoof <neginmr@utexas.edu> * Signed-off-by: neginraoof <neginmr@utexas.edu> * Added comments Signed-off-by: neginraoof <neginmr@utexas.edu> * Fix for comments Signed-off-by: neginraoof <neginmr@utexas.edu> Fix GitHub Action release CI failure: remove sudo ldconfig (#3485) * no sudo Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * remove in azurepipeline as well Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * only keep sudo make install in azurepipelines Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * add ldconfig back Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * remove ldconfig Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> Fix helper Signed-off-by: neginraoof <neginmr@utexas.edu> Fix for helpers Signed-off-by: neginraoof <neginmr@utexas.edu> Signed-off-by: neginraoof <neginmr@utexas.edu> Fix tests Signed-off-by: neginraoof <neginmr@utexas.edu> Test types Signed-off-by: neginraoof <neginmr@utexas.edu> Fix for feedback Signed-off-by: neginraoof <neginmr@utexas.edu> add PR template (#3492) Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Add a PR label to manually trigger release CIs (#3470) * trigger either by rel- branch or tag Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * use another jobs Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * apply to all CIs Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * PR on Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * consider other events to trigger CI Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * use base_ref to get branch name Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Make the version converter recurse into subgraphs (#3474) * Factor graph conversion in a separate method Signed-off-by: Matteo Salvarezza <matteo.salvarezza@gmail.com> * Recursively convert subgraph attributes Signed-off-by: Matteo Salvarezza <matteo.salvarezza@gmail.com> * Added test Signed-off-by: Matteo Salvarezza <matteo.salvarezza@gmail.com> * Lint Fixes Signed-off-by: Matteo Salvarezza <matteo.salvarezza@gmail.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Fix misleading global pooling test-case code (#3472) * Fix test-case globalaveragepool.py While the current code happens to be correct for 2d inputs, because `spatial_shape` happens to equal 2, it is incorrect for inputs that are 3d or greater. Signed-off-by: Calvin McCarter <77687912+calvinmccarter-at-lightmatter@users.noreply.github.com> * Fix test-case for globalmaxpool.py While the current code happens to be correct for 2d inputs, because `spatial_shape` happens to equal 2, it is incorrect for inputs that are 3d or greater. Signed-off-by: Calvin McCarter <77687912+calvinmccarter-at-lightmatter@users.noreply.github.com> * keepdims=True to eliminate spatial_shape and avoid expand_dims Signed-off-by: Calvin McCarter <77687912+calvinmccarter-at-lightmatter@users.noreply.github.com> * update Operators.md Signed-off-by: Calvin McCarter <77687912+calvinmccarter-at-lightmatter@users.noreply.github.com> * update TestCoverage.md Signed-off-by: Calvin McCarter <77687912+calvinmccarter-at-lightmatter@users.noreply.github.com> * update Operators.md Signed-off-by: Calvin McCarter <77687912+calvinmccarter-at-lightmatter@users.noreply.github.com> * update TestCoverage.md Signed-off-by: Calvin McCarter <77687912+calvinmccarter-at-lightmatter@users.noreply.github.com> Co-authored-by: Chun-Wei Chen <jacky82226@gmail.com> Add missing sparse tensor helpers (#3498) * Python API Overview: minor improvements for clarity (#3446) Also use repo-relative links so that they'll work as expected locally or in a branch. Signed-off-by: Gary Miguel <garymiguel@microsoft.com> * Make sure sparse attribute cases for get are covered. Add tests. Signed-off-by: Dmitri Smirnov <dmitrism@microsoft.com> * Add make_sparse_tensor_value_info along with the test. Signed-off-by: Dmitri Smirnov <dmitrism@microsoft.com> * Fix a typo. Signed-off-by: Dmitri Smirnov <dmitrism@microsoft.com> Co-authored-by: Gary Miguel <garymiguel@microsoft.com> Merge sparse tensor Signed-off-by: neginraoof <neginmr@utexas.edu> Types Signed-off-by: neginraoof <neginmr@utexas.edu> Rename make_sequence_value_info Signed-off-by: neginraoof <neginmr@utexas.edu> Signed-off-by: neginraoof <neginmr@utexas.edu> * Signed-off-by: neginraoof <neginmr@utexas.edu> * Fix for numpy_helper comments Signed-off-by: neginraoof <neginmr@utexas.edu> * Fixing some helper comments Signed-off-by: neginraoof <neginmr@utexas.edu> * fix for comments Signed-off-by: neginraoof <neginmr@utexas.edu> * fix for comments Signed-off-by: neginraoof <neginmr@utexas.edu> * fix for comments Signed-off-by: neginraoof <neginmr@utexas.edu> Co-authored-by: Chun-Wei Chen <jacky82226@gmail.com> Co-authored-by: impactaky <37619203+impactaky@users.noreply.github.com> Co-authored-by: impactaky <impactaky@users.noreply.github.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Michał Karzyński <4430709+postrational@users.noreply.github.com> Co-authored-by: Ewa Tusień <ewa.tusien@intel.com> Co-authored-by: Dwayne Robinson <dwayner@microsoft.com> Co-authored-by: Ian Bearman <ian.bearman@live.com> Co-authored-by: G. Ramalingam <grama@microsoft.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Update BatchNorm to specify population variance.
Signed-off-by: neginraoof neginmr@utexas.edu