Shapeinf for functions #3722

askhade · 2021-09-15T22:29:45Z

Description

Add shape inference for model local functions. Also includes some name formatting.
For schema less model local functions where a type and shape inference function is not available, fetch the function proto from model local functions container and traverse the function body to infer the shapes for the function node.

Motivation and Context

Why is this change required? What problem does it solve?
If it fixes an open issue, please link to the issue here.

Signed-off-by: Ashwini Khade <askhade@microsoft.com>

gramalingam · 2021-09-20T20:52:28Z

onnx/shape_inference/implementation.h

-      const int ir_version_in = IR_VERSION)
+      const int ir_version_in = IR_VERSION,
+      const ModelLocalFunctionsMap& model_local_functions_in = {},
+      GetFunctionMapIdFn get_func_id = nullptr)


Do we really need this (function to combine the domain and opname into one string) as a parameter? I understand having a single place where we can change it easily, if we want to, but unclear if we need to expose it as a parameter (that others start taking a dependence on).

+1 We should carefully review it since it's an important change that involves many functions and people will start to rely on it. It looks useful to me though. It will be more convincing if there are some real use cases for it.

gramalingam · 2021-09-20T20:58:58Z

onnx/test/cpp/function_verify_test.cc

  check_model(model);
+
+  ShapeInferenceOptions options{true, 1, true};
+  ONNX_NAMESPACE::shape_inference::InferShapes(model, OpSchemaRegistry::Instance(), options);


Just noting for future extension (not in this PR): Ideally, we should be saving the results of shape-inference applied to functions, and right now we don't have a place to save this. (That is, the analogue of "value_info" field in a graph.)

gramalingam · 2021-12-07T19:51:44Z

Wondering about the status of this PR. I guess not much change is required to merge this PR, is there? Thanks!

onnx/shape_inference/implementation.h

jcwchen · 2021-12-08T02:38:20Z

onnx/shape_inference/implementation.h

-      const int ir_version_in = IR_VERSION)
+      const int ir_version_in = IR_VERSION,
+      const ModelLocalFunctionsMap& model_local_functions_in = {},
+      GetFunctionMapIdFn get_func_id = nullptr)


+1 We should carefully review it since it's an important change that involves many functions and people will start to rely on it. It looks useful to me though. It will be more convincing if there are some real use cases for it.

askhade · 2021-12-08T18:01:18Z

Wondering about the status of this PR. I guess not much change is required to merge this PR, is there? Thanks!

No there isn't much work remaining. I will pick it up early next week and have it ready. Thanks!

Signed-off-by: Ashwini Khade <askhade@microsoft.com>

askhade · 2022-01-04T18:47:39Z

Linux CI failures are due to an unrelated issue. #3916 should fix it.

askhade · 2022-01-05T16:48:20Z

"LinuxRelease_i686 / build (3.8, x64) (pull_request) " pipeline is failing due to an unrelated reason. #3918 should fix it. Merging this PR now.

* add shape inference for model local functions Signed-off-by: Ashwini Khade <askhade@microsoft.com> * Plus tests and more changes Signed-off-by: Ashwini Khade <askhade@microsoft.com> * fix typo Signed-off-by: Ashwini Khade <askhade@microsoft.com> * Plus updates Signed-off-by: Ashwini Khade <askhade@microsoft.com> * Plus updates per review Signed-off-by: Ashwini Khade <askhade@microsoft.com> * plus updates Signed-off-by: Ashwini Khade <askhade@microsoft.com> Signed-off-by: Liqun Fu <liqfu@microsoft.com>

#3902) * Fix typos (#3894) Signed-off-by: Lewis Tunstall <lewis.c.tunstall@gmail.com> Signed-off-by: Liqun Fu <liqfu@microsoft.com> * Honor existing dim_param in shape inference (#3896) * Honor existing dim_param Before this change, existing dim_param was overwritten by generated symbols like `unk__`. `NonZero` is used to test the behavior. Signed-off-by: Shinichiro Hamaji <shinichiro.hamaji@gmail.com> * Fix test expectation of symbolic_shape_test.py Signed-off-by: Shinichiro Hamaji <shinichiro.hamaji@gmail.com> * Keep the original check as suggested in review Signed-off-by: Shinichiro Hamaji <shinichiro.hamaji@gmail.com> * Simplify the logic of mergeShapesAndTypes Signed-off-by: Shinichiro Hamaji <shinichiro.hamaji@gmail.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Signed-off-by: Liqun Fu <liqfu@microsoft.com> * for issue 3849 to confirm that type check is performed during checker.check_model call Signed-off-by: Liqun Fu <liqfu@microsoft.com> * fix path to InferenceError Signed-off-by: Liqun Fu <liqfu@microsoft.com> * replace whitelist by safelist (#3900) Signed-off-by: Alexandre Eichenberger <alexe@us.ibm.com> Signed-off-by: Liqun Fu <liqfu@microsoft.com> * add checker test case for input type mismatch conflict Signed-off-by: Liqun Fu <liqfu@microsoft.com> * remove old numpy in ipynb (#3916) Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> Signed-off-by: Liqun Fu <liqfu@microsoft.com> * Shapeinf for functions (#3722) * add shape inference for model local functions Signed-off-by: Ashwini Khade <askhade@microsoft.com> * Plus tests and more changes Signed-off-by: Ashwini Khade <askhade@microsoft.com> * fix typo Signed-off-by: Ashwini Khade <askhade@microsoft.com> * Plus updates Signed-off-by: Ashwini Khade <askhade@microsoft.com> * Plus updates per review Signed-off-by: Ashwini Khade <askhade@microsoft.com> * plus updates Signed-off-by: Ashwini Khade <askhade@microsoft.com> Signed-off-by: Liqun Fu <liqfu@microsoft.com> * Fix old ConvTranspose shape inference and softmax upgrader (#3893) * Fixed ConvTranspose-1 shape inference Brings change from #3188 (for ConvTranspose-11) into old shape inference Signed-off-by: Matteo Salvarezza <matteo.salvarezza@gmail.com> * Fix softmax adapter Signed-off-by: Matteo Salvarezza <matteo.salvarezza@gmail.com> Co-authored-by: G. Ramalingam <grama@microsoft.com> Signed-off-by: Liqun Fu <liqfu@microsoft.com> * Fix Linux i686 Release CI failure due to the latest NumPy (#3918) * 3.8 and 3.9 use numpy 1.21.5 Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * 1.21.4 Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * 6be8011a073feeca08d256ac64335a19fc8eee4c6312668b6781ace71db0de20 Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * 2021-12-19-a4d7f5a Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * 1.21.5 Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * 1.16.6 Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * 1.16.6 Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * do not check generated Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * remove unrelated numpy Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * if Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * remove generation Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * name Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * add back Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * for testing Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * no 1.16.6 Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> Signed-off-by: Liqun Fu <liqfu@microsoft.com> * Remind release manager to remove old onnx-weekly packages after release (#3923) * remind release manager to remove old onnx-weekly packages Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * add steps Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> Signed-off-by: Liqun Fu <liqfu@microsoft.com> * Simplify function definition of context-dependent functions (#3882) * Simplify function definition of context-dependent functions Signed-off-by: Ganesan Ramalingam <grama@microsoft.com> * Add missing parenthesis Signed-off-by: Ganesan Ramalingam <grama@microsoft.com> * Fix errors in function defs Signed-off-by: Ganesan Ramalingam <grama@microsoft.com> * Eliminate unused variable Signed-off-by: Ganesan Ramalingam <grama@microsoft.com> * Add int64 type specifier to literal Signed-off-by: Ganesan Ramalingam <grama@microsoft.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Signed-off-by: Liqun Fu <liqfu@microsoft.com> * Migration to using main branch (#3925) * add warning announce Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * Rename to main branch globally Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> Signed-off-by: Liqun Fu <liqfu@microsoft.com> * Fix the bug of shape in docs (#3927) * fix the bug of shape Signed-off-by: namasikanam <namasikanam@gmail.com> * fix the bug of shape Signed-off-by: namasikanam <namasikanam@gmail.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Signed-off-by: Liqun Fu <liqfu@microsoft.com> * Append dim even both dim value and param are not set (#3828) * Append dim even both dim value and param are not set Signed-off-by: Joe <joe@preferred.jp> * Copy dim Signed-off-by: Joe <joe@preferred.jp> * Revert "Copy dim" This reverts commit 6623505. Signed-off-by: Joe <joe@preferred.jp> * Simplify code Signed-off-by: Joe <joe@preferred.jp> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Signed-off-by: Liqun Fu <liqfu@microsoft.com> * bump to 10.15 in azp (#3941) Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> Signed-off-by: Liqun Fu <liqfu@microsoft.com> * Add explanation to run gtest (#3933) Signed-off-by: Joe <joe@preferred.jp> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Signed-off-by: Liqun Fu <liqfu@microsoft.com> * Update TreeEnsembleClassifier and TreeEnsembleRegressor to support tensor as attributes (#3897) * update TreeEnsembleClassifier and TreeEnsembleRegressor Signed-off-by: xavier dupré <xavier.dupre@gmail.com> * change the type of another attribute Signed-off-by: xavier dupré <xavier.dupre@gmail.com> * add missing file Signed-off-by: xavier dupré <xavier.dupre@gmail.com> * update documentation Signed-off-by: xavier dupré <xavier.dupre@gmail.com> * eol Signed-off-by: xavier dupré <xavier.dupre@gmail.com> * add field with _as_tensor Signed-off-by: xavier dupré <xavier.dupre@gmail.com> * fix error messages Signed-off-by: xavier dupré <xavier.dupre@gmail.com> * fix error message Signed-off-by: xavier dupré <xavier.dupre@gmail.com> * fix missing change Signed-off-by: xavier dupré <xavier.dupre@gmail.com> Co-authored-by: xavier dupré <xavier.dupre@gmail.com> Co-authored-by: G. Ramalingam <grama@microsoft.com> Signed-off-by: Liqun Fu <liqfu@microsoft.com> Co-authored-by: lewtun <lewis.c.tunstall@gmail.com> Co-authored-by: Shinichiro Hamaji <shinichiro.hamaji@gmail.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Alexandre Eichenberger <alexe@us.ibm.com> Co-authored-by: Chun-Wei Chen <jacky82226@gmail.com> Co-authored-by: Matteo Salvarezza <matteo.salvarezza@gmail.com> Co-authored-by: G. Ramalingam <grama@microsoft.com> Co-authored-by: Xingyu Xie <namasikanam@gmail.com> Co-authored-by: Joe <joe@preferred.jp> Co-authored-by: Xavier Dupré <xadupre@users.noreply.github.com> Co-authored-by: xavier dupré <xavier.dupre@gmail.com>

onnx#3902) * Fix typos (onnx#3894) Signed-off-by: Lewis Tunstall <lewis.c.tunstall@gmail.com> Signed-off-by: Liqun Fu <liqfu@microsoft.com> * Honor existing dim_param in shape inference (onnx#3896) * Honor existing dim_param Before this change, existing dim_param was overwritten by generated symbols like `unk__`. `NonZero` is used to test the behavior. Signed-off-by: Shinichiro Hamaji <shinichiro.hamaji@gmail.com> * Fix test expectation of symbolic_shape_test.py Signed-off-by: Shinichiro Hamaji <shinichiro.hamaji@gmail.com> * Keep the original check as suggested in review Signed-off-by: Shinichiro Hamaji <shinichiro.hamaji@gmail.com> * Simplify the logic of mergeShapesAndTypes Signed-off-by: Shinichiro Hamaji <shinichiro.hamaji@gmail.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Signed-off-by: Liqun Fu <liqfu@microsoft.com> * for issue 3849 to confirm that type check is performed during checker.check_model call Signed-off-by: Liqun Fu <liqfu@microsoft.com> * fix path to InferenceError Signed-off-by: Liqun Fu <liqfu@microsoft.com> * replace whitelist by safelist (onnx#3900) Signed-off-by: Alexandre Eichenberger <alexe@us.ibm.com> Signed-off-by: Liqun Fu <liqfu@microsoft.com> * add checker test case for input type mismatch conflict Signed-off-by: Liqun Fu <liqfu@microsoft.com> * remove old numpy in ipynb (onnx#3916) Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> Signed-off-by: Liqun Fu <liqfu@microsoft.com> * Shapeinf for functions (onnx#3722) * add shape inference for model local functions Signed-off-by: Ashwini Khade <askhade@microsoft.com> * Plus tests and more changes Signed-off-by: Ashwini Khade <askhade@microsoft.com> * fix typo Signed-off-by: Ashwini Khade <askhade@microsoft.com> * Plus updates Signed-off-by: Ashwini Khade <askhade@microsoft.com> * Plus updates per review Signed-off-by: Ashwini Khade <askhade@microsoft.com> * plus updates Signed-off-by: Ashwini Khade <askhade@microsoft.com> Signed-off-by: Liqun Fu <liqfu@microsoft.com> * Fix old ConvTranspose shape inference and softmax upgrader (onnx#3893) * Fixed ConvTranspose-1 shape inference Brings change from onnx#3188 (for ConvTranspose-11) into old shape inference Signed-off-by: Matteo Salvarezza <matteo.salvarezza@gmail.com> * Fix softmax adapter Signed-off-by: Matteo Salvarezza <matteo.salvarezza@gmail.com> Co-authored-by: G. Ramalingam <grama@microsoft.com> Signed-off-by: Liqun Fu <liqfu@microsoft.com> * Fix Linux i686 Release CI failure due to the latest NumPy (onnx#3918) * 3.8 and 3.9 use numpy 1.21.5 Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * 1.21.4 Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * 6be8011a073feeca08d256ac64335a19fc8eee4c6312668b6781ace71db0de20 Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * 2021-12-19-a4d7f5a Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * 1.21.5 Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * 1.16.6 Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * 1.16.6 Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * do not check generated Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * remove unrelated numpy Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * if Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * remove generation Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * name Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * add back Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * for testing Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * no 1.16.6 Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> Signed-off-by: Liqun Fu <liqfu@microsoft.com> * Remind release manager to remove old onnx-weekly packages after release (onnx#3923) * remind release manager to remove old onnx-weekly packages Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * add steps Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> Signed-off-by: Liqun Fu <liqfu@microsoft.com> * Simplify function definition of context-dependent functions (onnx#3882) * Simplify function definition of context-dependent functions Signed-off-by: Ganesan Ramalingam <grama@microsoft.com> * Add missing parenthesis Signed-off-by: Ganesan Ramalingam <grama@microsoft.com> * Fix errors in function defs Signed-off-by: Ganesan Ramalingam <grama@microsoft.com> * Eliminate unused variable Signed-off-by: Ganesan Ramalingam <grama@microsoft.com> * Add int64 type specifier to literal Signed-off-by: Ganesan Ramalingam <grama@microsoft.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Signed-off-by: Liqun Fu <liqfu@microsoft.com> * Migration to using main branch (onnx#3925) * add warning announce Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * Rename to main branch globally Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> Signed-off-by: Liqun Fu <liqfu@microsoft.com> * Fix the bug of shape in docs (onnx#3927) * fix the bug of shape Signed-off-by: namasikanam <namasikanam@gmail.com> * fix the bug of shape Signed-off-by: namasikanam <namasikanam@gmail.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Signed-off-by: Liqun Fu <liqfu@microsoft.com> * Append dim even both dim value and param are not set (onnx#3828) * Append dim even both dim value and param are not set Signed-off-by: Joe <joe@preferred.jp> * Copy dim Signed-off-by: Joe <joe@preferred.jp> * Revert "Copy dim" This reverts commit 6623505. Signed-off-by: Joe <joe@preferred.jp> * Simplify code Signed-off-by: Joe <joe@preferred.jp> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Signed-off-by: Liqun Fu <liqfu@microsoft.com> * bump to 10.15 in azp (onnx#3941) Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> Signed-off-by: Liqun Fu <liqfu@microsoft.com> * Add explanation to run gtest (onnx#3933) Signed-off-by: Joe <joe@preferred.jp> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Signed-off-by: Liqun Fu <liqfu@microsoft.com> * Update TreeEnsembleClassifier and TreeEnsembleRegressor to support tensor as attributes (onnx#3897) * update TreeEnsembleClassifier and TreeEnsembleRegressor Signed-off-by: xavier dupré <xavier.dupre@gmail.com> * change the type of another attribute Signed-off-by: xavier dupré <xavier.dupre@gmail.com> * add missing file Signed-off-by: xavier dupré <xavier.dupre@gmail.com> * update documentation Signed-off-by: xavier dupré <xavier.dupre@gmail.com> * eol Signed-off-by: xavier dupré <xavier.dupre@gmail.com> * add field with _as_tensor Signed-off-by: xavier dupré <xavier.dupre@gmail.com> * fix error messages Signed-off-by: xavier dupré <xavier.dupre@gmail.com> * fix error message Signed-off-by: xavier dupré <xavier.dupre@gmail.com> * fix missing change Signed-off-by: xavier dupré <xavier.dupre@gmail.com> Co-authored-by: xavier dupré <xavier.dupre@gmail.com> Co-authored-by: G. Ramalingam <grama@microsoft.com> Signed-off-by: Liqun Fu <liqfu@microsoft.com> Co-authored-by: lewtun <lewis.c.tunstall@gmail.com> Co-authored-by: Shinichiro Hamaji <shinichiro.hamaji@gmail.com> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Alexandre Eichenberger <alexe@us.ibm.com> Co-authored-by: Chun-Wei Chen <jacky82226@gmail.com> Co-authored-by: Matteo Salvarezza <matteo.salvarezza@gmail.com> Co-authored-by: G. Ramalingam <grama@microsoft.com> Co-authored-by: Xingyu Xie <namasikanam@gmail.com> Co-authored-by: Joe <joe@preferred.jp> Co-authored-by: Xavier Dupré <xadupre@users.noreply.github.com> Co-authored-by: xavier dupré <xavier.dupre@gmail.com> Signed-off-by: Liqun Fu <liqfu@microsoft.com>

askhade added 4 commits September 14, 2021 13:30

add shape inference for model local functions

43bd56c

Signed-off-by: Ashwini Khade <askhade@microsoft.com>

Plus tests and more changes

015c50c

Signed-off-by: Ashwini Khade <askhade@microsoft.com>

merge master

6a376fc

Signed-off-by: Ashwini Khade <askhade@microsoft.com>

fix typo

f46d5ee

Signed-off-by: Ashwini Khade <askhade@microsoft.com>

askhade requested a review from a team as a code owner September 15, 2021 22:29

askhade mentioned this pull request Sep 15, 2021

add shape inference for model local functions #3718

Closed

askhade added run release CIs Use this label to trigger release tests in CI shape inference Issues related to shape inference labels Sep 15, 2021

askhade requested review from gramalingam and jcwchen September 15, 2021 22:31

Plus updates

00cc502

Signed-off-by: Ashwini Khade <askhade@microsoft.com>

gramalingam reviewed Sep 20, 2021

View reviewed changes

jcwchen reviewed Dec 8, 2021

View reviewed changes

askhade added 2 commits December 17, 2021 13:38

merge master

b32cba2

Plus updates per review

d03836f

Signed-off-by: Ashwini Khade <askhade@microsoft.com>

askhade force-pushed the shapeinf_for_functions branch from a12af40 to d03836f Compare January 4, 2022 18:20

plus updates

7ab93e1

Signed-off-by: Ashwini Khade <askhade@microsoft.com>

gramalingam approved these changes Jan 4, 2022

View reviewed changes

Merge branch 'master' into shapeinf_for_functions

7e83540

askhade merged commit b3f5854 into onnx:master Jan 5, 2022

askhade added this to the 1.11 milestone Jan 5, 2022

jcwchen mentioned this pull request Feb 16, 2022

tf2onnx test test_hashtable_lookup failed with onnx 1.11 rc1 and rc2 version. #4020

Closed

jcwchen mentioned this pull request Jun 28, 2022

How to remove warning from shape_inference #3696

Closed

max-509 mentioned this pull request May 17, 2023

Shape inference should handle MapProto #5231

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Shapeinf for functions #3722

Shapeinf for functions #3722

askhade commented Sep 15, 2021

gramalingam Sep 20, 2021

jcwchen Dec 8, 2021

gramalingam Sep 20, 2021

gramalingam commented Dec 7, 2021

jcwchen Dec 8, 2021

askhade commented Dec 8, 2021

askhade commented Jan 4, 2022

askhade commented Jan 5, 2022

Shapeinf for functions #3722

Shapeinf for functions #3722

Conversation

askhade commented Sep 15, 2021

gramalingam Sep 20, 2021

Choose a reason for hiding this comment

jcwchen Dec 8, 2021

Choose a reason for hiding this comment

gramalingam Sep 20, 2021

Choose a reason for hiding this comment

gramalingam commented Dec 7, 2021

jcwchen Dec 8, 2021

Choose a reason for hiding this comment

askhade commented Dec 8, 2021

askhade commented Jan 4, 2022

askhade commented Jan 5, 2022