Skip to content
This repository has been archived by the owner on Nov 17, 2023. It is now read-only.

Fix SoftReLU fused operator numerical stability #17849

Merged
merged 2 commits into from
Mar 18, 2020
Merged

Fix SoftReLU fused operator numerical stability #17849

merged 2 commits into from
Mar 18, 2020

Conversation

RuRo
Copy link
Contributor

@RuRo RuRo commented Mar 16, 2020

Description

Fixes #17844

Checklist

Essentials

Please feel free to remove inapplicable items for your PR.

  • The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant JIRA issue created (except PRs with tiny changes)
  • Changes are complete (i.e. I finished coding on this PR)
  • All changes have test coverage:
  • Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
  • Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
  • Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
  • Code is well-documented:
  • For user-facing API changes, API doc string has been updated.
  • For new C++ functions in header files, their functionalities and arguments are documented.
  • For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
  • Check the API doc at https://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
  • To the best of my knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Changes

  • This PR copies the mshadow thresholding approach to ensure, that SoftReLU is numerically stable for large inputs values.

@leezu leezu requested a review from ptrendx March 16, 2020 18:35
ptrendx
ptrendx approved these changes Mar 16, 2020
View reviewed changes
Copy link
Member

@ptrendx ptrendx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch! Thank you for the fix

@leezu leezu merged commit 5996544 into apache:master Mar 18, 2020
@RuRo RuRo deleted the fix_softrelu_fused_op branch March 18, 2020 05:42
anirudh2290 added a commit to anirudh2290/mxnet that referenced this pull request Mar 27, 2020
@anirudh2290
Merge branch 'master' of https://github.com/apache/incubator-mxnet
ae654ba 
* 'master' of https://github.com/apache/incubator-mxnet: (192 commits)
  * impl - FFI for np einsum (apache#17869)
  [Numpy] FFI for diag/diagonal/diag_indices_from (apache#17789)
  [Numpy] Kron operator (apache#17323)
  cmake: Set DMLC_LOG_FATAL_THROW only for building mxnet and not for tvm (apache#17878)
  Add simplified HybridBlock.forward without F (apache#17530)
  Use FP32 copy of weights for norm (multitensor LAMB optimizer) (apache#17700)
  Use multi-tensor sumSQ in clip_global_norm (apache#17652)
  [Numpy] Add op fmax, fmin, fmod (apache#17567)
  Adding sparse support to MXTensor for custom operators (apache#17569)
  Update 3rdparty/mkldnn to v1.2.2 (apache#17313)
  Dynamic subgraph compile support (apache#17623)
  Refactor cpp-package CMakeLists.txt & add missing inference/imagenet_inference (apache#17835)
  staticbuild: Fix potential user-assisted execution of arbitrary code  (apache#17860)
  * FFI for np.argmax and np.argmin (apache#17843)
  ffi for roll/rot90 (apache#17861)
  Skip test_multi_worker_dataloader_release_pool on OS X (apache#17797)
  add ffi for full_like, binary (apache#17811)
  HybridBlock.export() to return created filenames (apache#17758)
  Fix SoftReLU fused operator numerical stability (apache#17849)
  CI: Test clang10 cpu & gpu builds with -WError (apache#17830)
  ...
MoisesHer pushed a commit to MoisesHer/incubator-mxnet that referenced this pull request Apr 10, 2020
@RuRo @MoisesHer
Fix SoftReLU fused operator numerical stability (apache#17849)
0f4233a 
* fix numerically unstable fused softrelu op

* implement test for softrelu numerical stability
anirudh2290 pushed a commit to anirudh2290/mxnet that referenced this pull request May 29, 2020
@RuRo @anirudh2290
Fix SoftReLU fused operator numerical stability (apache#17849)
171f9b2 
* fix numerically unstable fused softrelu op

* implement test for softrelu numerical stability
mseth10 pushed a commit to mseth10/incubator-mxnet that referenced this pull request Oct 17, 2020
@RuRo
Fix SoftReLU fused operator numerical stability (apache#17849)
04b51f2 
* fix numerically unstable fused softrelu op

* implement test for softrelu numerical stability
mseth10 pushed a commit to mseth10/incubator-mxnet that referenced this pull request Oct 20, 2020
@RuRo @mseth10
Fix SoftReLU fused operator numerical stability (apache#17849)
e3d25ae 
* fix numerically unstable fused softrelu op

* implement test for softrelu numerical stability
mseth10 pushed a commit to mseth10/incubator-mxnet that referenced this pull request Oct 20, 2020
@RuRo @mseth10
Fix SoftReLU fused operator numerical stability (apache#17849)
cf3f896 
* fix numerically unstable fused softrelu op

* implement test for softrelu numerical stability
mseth10 pushed a commit to mseth10/incubator-mxnet that referenced this pull request Oct 22, 2020
@RuRo @mseth10
Fix SoftReLU fused operator numerical stability (apache#17849)
9a8d454 
* fix numerically unstable fused softrelu op

* implement test for softrelu numerical stability
mseth10 pushed a commit to mseth10/incubator-mxnet that referenced this pull request Oct 22, 2020
@RuRo @mseth10
Fix SoftReLU fused operator numerical stability (apache#17849)
65afc61 
* fix numerically unstable fused softrelu op

* implement test for softrelu numerical stability
mseth10 pushed a commit to mseth10/incubator-mxnet that referenced this pull request Oct 22, 2020
@RuRo @mseth10
Fix SoftReLU fused operator numerical stability (apache#17849)
dc4b22e 
* fix numerically unstable fused softrelu op

* implement test for softrelu numerical stability
samskalicky pushed a commit that referenced this pull request Oct 23, 2020
@mseth10 @RuRo
Fix SoftReLU fused operator numerical stability (#17849) (#19391)
468e21d 
* fix numerically unstable fused softrelu op

* implement test for softrelu numerical stability

Co-authored-by: RuRo <ruro.ruro@ya.ru>
samskalicky pushed a commit that referenced this pull request Oct 23, 2020
@mseth10 @RuRo
Fix SoftReLU fused operator numerical stability (#17849) (#19390)
ddab468 
* fix numerically unstable fused softrelu op

* implement test for softrelu numerical stability

Co-authored-by: RuRo <ruro.ruro@ya.ru>
samskalicky pushed a commit that referenced this pull request Oct 23, 2020
@mseth10 @RuRo
Fix SoftReLU fused operator numerical stability (#17849) (#19370)
df7ef6c 
* fix numerically unstable fused softrelu op

* implement test for softrelu numerical stability

Co-authored-by: RuRo <ruro.ruro@ya.ru>
josephevans pushed a commit to josephevans/mxnet that referenced this pull request Dec 8, 2020
@mseth10 @RuRo
Fix SoftReLU fused operator numerical stability (apache#17849) (apach…
d03fb1d 
…e#19391)

* fix numerically unstable fused softrelu op

* implement test for softrelu numerical stability

Co-authored-by: RuRo <ruro.ruro@ya.ru>
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@szha szha szha left review comments

@ptrendx ptrendx ptrendx approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Hybridized SoftReLU on the GPU is not numerically stable
4 participants
@RuRo @szha @ptrendx @leezu