Update softmax family ops behavior to align with other frameworks (fix #2289) #2879

daquexian · 2020-07-06T14:08:24Z

This PR updates softmax and logsoftmax so that they are functions and align with other frameworks (tf, pytorch, etc), and hardmax is also updated as a primitive operator

…e_softmax

linkerzhang · 2020-07-08T07:42:52Z

onnx/defs/math/defs.cc

-        "from the back. Accepted range is [-r, r-1] where r = rank(input).",
-        AttributeProto::INT,
-        static_cast<int64_t>(1));
+        "axis", axis_attr, AttributeProto::INT, static_cast<int64_t>(-1));


The version should also be bumped, given the default value of "axis" changed.

In my opinion, if it's only default value change, I'd suggest to not change it. The benefit is not that big with fair change (version bump).

I think the current softmax op version has already been bumped since last release (current version is already 13). Is it necessary to bump it once more?

no in that way. Only one version bump in one release.

Yeah, changing any of these {attribute names, attribute default values, tensors meanings} is a version breaking change.

linkerzhang · 2020-07-08T07:54:33Z

onnx/defs/math/defs.cc

+        .FillUsing(
+            SoftmaxFamilyDocGenerator("softmax", "normalized exponential"))
+        .SetContextDependentFunctionBodyBuilder(
+            [](const FunctionBodyBuildContext& ctx,


did you verify the correctness of the subgraph? (the "expanded" and non "expanded" model generated in this PR, feeding them same inputs will get same outputs)

Here's the old decomposition I wrote down before (ignore the flattening part now, I guess):

https://fdwr.github.io/LostOnnxDocs/OperatorFormulas.html

function SoftMax(Input; Output; axis): FlattenedInput = Flatten(Input, axis) // Flatten to 2D NormalizedInput = SoftMax2D(FlattenedInput; ; axis) Output = Reshape(NormalizedInput, Shape(X)) endfunction function SoftMax2D(Input; Output; axis): MaxInput = ReduceMax(Input, axes=[1], keepdims=1) ExpInput = Exp(Input - MaxInput) ReducedExpInput = ReduceSum(ExpInput, axes=[1], keepdims=1) Output = ExpInput / ReducedExpInput endfunction

@linkerzhang @fdwr I have verified the correctness of the subgraph by inferencing the subgraph in onnxruntime and comparing the result with the reference numpy implementation

docs/Changelog.md

daquexian · 2020-08-13T03:29:54Z

@linkerzhang @wschin @fdwr I have updated the PR according to the comments. Please review it again. Thanks!

docs/Changelog.md

docs/Operators.md

daquexian · 2020-09-02T14:01:35Z

@wschin @linkerzhang I have updated this PR. Is it ok to merge it? Thanks!

wschin · 2020-09-18T23:38:12Z

@wschin @linkerzhang I have updated this PR. Is it ok to merge it? Thanks!

Could you please sync this branch with master again? Sorry for being late and we want this PR in for this release.

daquexian · 2020-09-19T02:36:18Z

Could you please sync this branch with master again? Sorry for being late and we want this PR in for this release.

Done. Do I need to sign off according to the DCO check? I tried signing off the commits according to its instructions but I found it makes git history confusing.

wschin · 2020-09-19T05:39:44Z

Could you please sync this branch with master again? Sorry for being late and we want this PR in for this release.

Done. Do I need to sign off according to the DCO check? I tried signing off the commits according to its instructions but I found it makes git history confusing.

@prasanthpul, Is DCO required now?

@daquexian, maybe you can do

git checkout master
git checkout -b new_branch
git merge --squash daquexian:update_softmax // Get everything from old branch as a single commit on "new_branch"
git branch -d daquexian:update_softmax
git checkout -b daquexian:update_softmax // Based on "new_branch" to recreate "daquexian:update_softmax"
git push -f

[Update] As discussed with @prasanthpul offline, DOC is not required now so I just merged it. Thank you for all the hard works!

…onnx#2289) (onnx#2879) * Update softmax family ops behavior to align with other frameworks * Update logsoftmax, hardmax tests, regenerate docs and test data * fix wrong input name in function * regenerate test data * fix flake8 error * regenerate docs * regenerate docs * add missing type annotation for hardmax * add the math for softmax family operators * remove the 'description' field in docs as it is covered by the math * fix wrong format in axis attr * replace name with description * restore the name field for axis attr * regenerate docs * regenerate docs * add the missing name * regenerate docs * update reducesum to align with master * regenerate tests Co-authored-by: Wei-Sheng Chin <wschin@outlook.com>

…onnx#2289) (onnx#2879) * Update softmax family ops behavior to align with other frameworks * Update logsoftmax, hardmax tests, regenerate docs and test data * fix wrong input name in function * regenerate test data * fix flake8 error * regenerate docs * regenerate docs * add missing type annotation for hardmax * add the math for softmax family operators * remove the 'description' field in docs as it is covered by the math * fix wrong format in axis attr * replace name with description * restore the name field for axis attr * regenerate docs * regenerate docs * add the missing name * regenerate docs * update reducesum to align with master * regenerate tests Co-authored-by: Wei-Sheng Chin <wschin@outlook.com> Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com>

* remove wrong description for pow Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * Add size check to make_tensor (#2987) Co-authored-by: Ke Zhang <linkerzhang@yeah.net> Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * Fix float16 data convert issue in numpy_helper.to_array (#3002) * handle f16 case for to_array * fix flake8 * nit: comment Co-authored-by: Ashwini Khade <askhade@microsoft.com> Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * Deprecate Travis CI (#2773) * adding gtests for c++ api test * deprecate travis * adding the build badges for the new pipelines in Azure, deprecating travis build badge, renaming circleCI badge * updating badge label * removing - in badge names * c++ api changes for linux * update environment variables * update env variables, setup tools call * Update Linux-CI.yml for Azure Pipelines * revert changes to Linux and Mac CIs * delete last travis file Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * Fix shape inference of scalar ConstantOfShape (#3005) When input shape is (0), we do not add any dim to inferred shape but we should initialize tensor_type.shape by calling mutable_shape(). Co-authored-by: Ashwini Khade <askhade@microsoft.com> Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * fix shape inference for loop (#3014) Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * Fix ConvTranspose: enhance attribute check (#3000) * add check for using auto_pad and pads simultaneously * fix description for auto_pads == SAME_UPPER * update docs for operator * fix the old one as well * add a test * Revert "fix description for auto_pads == SAME_UPPER" This reverts commit e75e287. * Revert "update docs for operator" This reverts commit 70952c0. * Revert "fix the old one as well" This reverts commit 8a0482d. Co-authored-by: Ashwini Khade <askhade@microsoft.com> Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * Schema change to support dynamic shapes in ORT (#2955) * Changes to schema and python tests * Modify test * Remove attribute that is input also * Changes to optimizers, adapters and tests * Run flake8 * undo unrequired comit files, fix formatting, review changes * Fix ci test, cleanup * Fix narrowing conversion error * add missed test model Co-authored-by: G. Ramalingam <grama@microsoft.com> Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * fix loop shape inference for ver 11 (#3023) Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * Update softmax family ops behavior to align with other frameworks (fix #2289) (#2879) * Update softmax family ops behavior to align with other frameworks * Update logsoftmax, hardmax tests, regenerate docs and test data * fix wrong input name in function * regenerate test data * fix flake8 error * regenerate docs * regenerate docs * add missing type annotation for hardmax * add the math for softmax family operators * remove the 'description' field in docs as it is covered by the math * fix wrong format in axis attr * replace name with description * restore the name field for axis attr * regenerate docs * regenerate docs * add the missing name * regenerate docs * update reducesum to align with master * regenerate tests Co-authored-by: Wei-Sheng Chin <wschin@outlook.com> Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> Co-authored-by: G. Ramalingam <grama@microsoft.com> Co-authored-by: Ke Zhang <linkerzhang@yeah.net> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Vinitra Swamy <vinitras@gmail.com> Co-authored-by: Shinichiro Hamaji <shinichiro.hamaji@gmail.com> Co-authored-by: ashbhandare <ash.bhandare@gmail.com> Co-authored-by: daquexian <daquexian566@gmail.com> Co-authored-by: Wei-Sheng Chin <wschin@outlook.com>

…onnx#2289) (onnx#2879) * Update softmax family ops behavior to align with other frameworks * Update logsoftmax, hardmax tests, regenerate docs and test data * fix wrong input name in function * regenerate test data * fix flake8 error * regenerate docs * regenerate docs * add missing type annotation for hardmax * add the math for softmax family operators * remove the 'description' field in docs as it is covered by the math * fix wrong format in axis attr * replace name with description * restore the name field for axis attr * regenerate docs * regenerate docs * add the missing name * regenerate docs * update reducesum to align with master * regenerate tests Co-authored-by: Wei-Sheng Chin <wschin@outlook.com>

…x#2999) * remove wrong description for pow Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * Add size check to make_tensor (onnx#2987) Co-authored-by: Ke Zhang <linkerzhang@yeah.net> Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * Fix float16 data convert issue in numpy_helper.to_array (onnx#3002) * handle f16 case for to_array * fix flake8 * nit: comment Co-authored-by: Ashwini Khade <askhade@microsoft.com> Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * Deprecate Travis CI (onnx#2773) * adding gtests for c++ api test * deprecate travis * adding the build badges for the new pipelines in Azure, deprecating travis build badge, renaming circleCI badge * updating badge label * removing - in badge names * c++ api changes for linux * update environment variables * update env variables, setup tools call * Update Linux-CI.yml for Azure Pipelines * revert changes to Linux and Mac CIs * delete last travis file Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * Fix shape inference of scalar ConstantOfShape (onnx#3005) When input shape is (0), we do not add any dim to inferred shape but we should initialize tensor_type.shape by calling mutable_shape(). Co-authored-by: Ashwini Khade <askhade@microsoft.com> Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * fix shape inference for loop (onnx#3014) Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * Fix ConvTranspose: enhance attribute check (onnx#3000) * add check for using auto_pad and pads simultaneously * fix description for auto_pads == SAME_UPPER * update docs for operator * fix the old one as well * add a test * Revert "fix description for auto_pads == SAME_UPPER" This reverts commit e75e287. * Revert "update docs for operator" This reverts commit 70952c0. * Revert "fix the old one as well" This reverts commit 8a0482d. Co-authored-by: Ashwini Khade <askhade@microsoft.com> Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * Schema change to support dynamic shapes in ORT (onnx#2955) * Changes to schema and python tests * Modify test * Remove attribute that is input also * Changes to optimizers, adapters and tests * Run flake8 * undo unrequired comit files, fix formatting, review changes * Fix ci test, cleanup * Fix narrowing conversion error * add missed test model Co-authored-by: G. Ramalingam <grama@microsoft.com> Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * fix loop shape inference for ver 11 (onnx#3023) Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> * Update softmax family ops behavior to align with other frameworks (fix onnx#2289) (onnx#2879) * Update softmax family ops behavior to align with other frameworks * Update logsoftmax, hardmax tests, regenerate docs and test data * fix wrong input name in function * regenerate test data * fix flake8 error * regenerate docs * regenerate docs * add missing type annotation for hardmax * add the math for softmax family operators * remove the 'description' field in docs as it is covered by the math * fix wrong format in axis attr * replace name with description * restore the name field for axis attr * regenerate docs * regenerate docs * add the missing name * regenerate docs * update reducesum to align with master * regenerate tests Co-authored-by: Wei-Sheng Chin <wschin@outlook.com> Signed-off-by: Chun-Wei Chen <jacky82226@gmail.com> Co-authored-by: G. Ramalingam <grama@microsoft.com> Co-authored-by: Ke Zhang <linkerzhang@yeah.net> Co-authored-by: Ashwini Khade <askhade@microsoft.com> Co-authored-by: Vinitra Swamy <vinitras@gmail.com> Co-authored-by: Shinichiro Hamaji <shinichiro.hamaji@gmail.com> Co-authored-by: ashbhandare <ash.bhandare@gmail.com> Co-authored-by: daquexian <daquexian566@gmail.com> Co-authored-by: Wei-Sheng Chin <wschin@outlook.com>

daquexian added 2 commits July 6, 2020 12:12

Update softmax family ops behavior to align with other frameworks

d89c36d

Update logsoftmax, hardmax tests, regenerate docs and test data

b92b98b

daquexian requested a review from a team as a code owner July 6, 2020 14:08

daquexian added 8 commits July 6, 2020 22:15

Merge branch 'master' into update_softmax

239c348

fix wrong input name in function

140ae44

regenerate test data

3b4e529

Merge branch 'update_softmax' of github.com:daquexian/onnx into updat…

e3288c1

…e_softmax

fix flake8 error

ea5581e

regenerate docs

53c34da

regenerate docs

62337a5

add missing type annotation for hardmax

3d52bcc

linkerzhang reviewed Jul 8, 2020

View reviewed changes

wschin reviewed Jul 24, 2020

View reviewed changes

docs/Changelog.md Show resolved Hide resolved

daquexian added 8 commits August 11, 2020 21:38

Merge remote-tracking branch 'origin/master' into update_softmax

1d633df

add the math for softmax family operators

d31adff

Merge remote-tracking branch 'origin/master' into update_softmax

29c0787

remove the 'description' field in docs as it is covered by the math

b34fa5f

fix wrong format in axis attr

9d6e91c

replace name with description

19062ab

restore the name field for axis attr

6bc0e6b

regenerate docs

c2181f2

daquexian added 2 commits August 15, 2020 09:39

Merge branch 'master' into update_softmax

4cc993e

Merge branch 'master' into update_softmax

6588b81

daquexian added the operator Issues related to ONNX operators label Aug 24, 2020

daquexian requested review from wschin and linkerzhang August 24, 2020 11:19

Merge branch 'master' into update_softmax

6b520c6

daquexian mentioned this pull request Aug 26, 2020

[Tracking] ONNX 1.8 Release #2942

Closed

wschin reviewed Aug 31, 2020

View reviewed changes

docs/Changelog.md Outdated Show resolved Hide resolved

wschin reviewed Aug 31, 2020

View reviewed changes

docs/Operators.md Outdated Show resolved Hide resolved

wschin approved these changes Aug 31, 2020

View reviewed changes

daquexian added 4 commits September 2, 2020 20:33

Merge remote-tracking branch 'origin/master' into update_softmax

5c6bac3

regenerate docs

72b4096

add the missing name

c624581

regenerate docs

739d02a

daquexian and others added 5 commits September 5, 2020 11:53

Merge branch 'master' into update_softmax

730072f

Merge branch 'master' into update_softmax

aa2eabb

Merge branch 'master' into update_softmax

f664029

Merge branch 'master' into update_softmax

f733e70

Merge branch 'master' into update_softmax

be65b08

daquexian added 2 commits September 19, 2020 10:08

update reducesum to align with master

d1e2822

regenerate tests

712d393

daquexian force-pushed the update_softmax branch from 712d393 to 56bf5bc Compare September 19, 2020 02:22

daquexian requested a review from a team as a code owner September 19, 2020 02:22

daquexian force-pushed the update_softmax branch from 56bf5bc to 712d393 Compare September 19, 2020 02:23

wschin merged commit 689c4e3 into onnx:master Sep 19, 2020

daquexian deleted the update_softmax branch September 20, 2020 03:05

jcwchen mentioned this pull request Oct 20, 2022

Question on the normalization dimension of ONNX Softmax op #4605

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update softmax family ops behavior to align with other frameworks (fix #2289) #2879

Update softmax family ops behavior to align with other frameworks (fix #2289) #2879

daquexian commented Jul 6, 2020 •

edited

Loading

linkerzhang Jul 8, 2020

linkerzhang Jul 8, 2020

daquexian Jul 8, 2020

linkerzhang Jul 9, 2020

fdwr Jul 31, 2020

linkerzhang Jul 8, 2020

fdwr Jul 31, 2020 •

edited

Loading

daquexian Aug 13, 2020

daquexian commented Aug 13, 2020 •

edited

Loading

daquexian commented Sep 2, 2020 •

edited

Loading

wschin commented Sep 18, 2020 •

edited

Loading

daquexian commented Sep 19, 2020 •

edited

Loading

wschin commented Sep 19, 2020 •

edited

Loading

Update softmax family ops behavior to align with other frameworks (fix #2289) #2879

Update softmax family ops behavior to align with other frameworks (fix #2289) #2879

Conversation

daquexian commented Jul 6, 2020 • edited Loading

linkerzhang Jul 8, 2020

Choose a reason for hiding this comment

linkerzhang Jul 8, 2020

Choose a reason for hiding this comment

daquexian Jul 8, 2020

Choose a reason for hiding this comment

linkerzhang Jul 9, 2020

Choose a reason for hiding this comment

fdwr Jul 31, 2020

Choose a reason for hiding this comment

linkerzhang Jul 8, 2020

Choose a reason for hiding this comment

fdwr Jul 31, 2020 • edited Loading

Choose a reason for hiding this comment

daquexian Aug 13, 2020

Choose a reason for hiding this comment

daquexian commented Aug 13, 2020 • edited Loading

daquexian commented Sep 2, 2020 • edited Loading

wschin commented Sep 18, 2020 • edited Loading

daquexian commented Sep 19, 2020 • edited Loading

wschin commented Sep 19, 2020 • edited Loading

daquexian commented Jul 6, 2020 •

edited

Loading

fdwr Jul 31, 2020 •

edited

Loading

daquexian commented Aug 13, 2020 •

edited

Loading

daquexian commented Sep 2, 2020 •

edited

Loading

wschin commented Sep 18, 2020 •

edited

Loading

daquexian commented Sep 19, 2020 •

edited

Loading

wschin commented Sep 19, 2020 •

edited

Loading