Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use nn.softmax in keras.activations.softmax #47572

Merged
merged 1 commit into from Mar 12, 2021
Merged

Use nn.softmax in keras.activations.softmax #47572

merged 1 commit into from Mar 12, 2021

Conversation

szutenberg
Copy link
Contributor

There are two issues with tf.keras.activations.softmax :

First:
it implements softmax for 3D+ tensors in not optimal way due to old workarounds:

  • Keras: Support for 3D+ tensors was added on Nov 29, 2015 keras-team/keras@8f2d6d2
  • TF: In that time tensorflow did not support multiple dimensions: issue was reported on Jun 29, 2016 (Softmax for multiple dimensions) Softmax for multiple dimensions #3101
  • TF: Support for multiple dimensions was added on Aug 29, 2016 aeac274
  • TF: Improved kernel accepting shapes with higher ranks was added on Aug 6, 2018 f3aa5d4

For 3D tensor, the softmax gradient is getting really complicated which seriously hurts performance. example:
image

Second:
It returns wrong results for 2D tensors, axis=0. I also prepared unit test for this scenario.

nn.softmax is used for tensors with all dimensions
added test_softmax_2d_axis0 covering fixed bug
@google-ml-butler google-ml-butler bot added the size:S CL Change Size: Small label Mar 4, 2021
@google-cla
Copy link

google-cla bot commented Mar 4, 2021

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.


What to do if you already signed the CLA

Individual signers
Corporate signers

ℹ️ Googlers: Go here for more info.

@google-cla google-cla bot added the cla: no label Mar 4, 2021
@szutenberg
Copy link
Contributor Author

@googlebot I signed it!

@google-cla google-cla bot added cla: yes and removed cla: no labels Mar 4, 2021
@gbaned gbaned self-assigned this Mar 5, 2021
@gbaned gbaned added the comp:keras Keras related issues label Mar 5, 2021
@gbaned gbaned added this to Assigned Reviewer in PR Queue via automation Mar 5, 2021
@gbaned gbaned self-requested a review March 5, 2021 03:48
Copy link
Member

@fchollet fchollet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks

PR Queue automation moved this from Assigned Reviewer to Approved by Reviewer Mar 10, 2021
@google-ml-butler google-ml-butler bot added kokoro:force-run Tests on submitted change ready to pull PR ready for merge process labels Mar 10, 2021
@kokoro-team kokoro-team removed the kokoro:force-run Tests on submitted change label Mar 10, 2021
@gbaned gbaned removed the ready to pull PR ready for merge process label Mar 12, 2021
@google-ml-butler google-ml-butler bot added kokoro:force-run Tests on submitted change ready to pull PR ready for merge process labels Mar 12, 2021
@kokoro-team kokoro-team removed the kokoro:force-run Tests on submitted change label Mar 12, 2021
@copybara-service copybara-service bot merged commit efe5732 into tensorflow:master Mar 12, 2021
PR Queue automation moved this from Approved by Reviewer to Merged Mar 12, 2021
@szutenberg szutenberg deleted the softmax-patch branch June 13, 2021 12:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cla: yes comp:keras Keras related issues ready to pull PR ready for merge process size:S CL Change Size: Small
Projects
PR Queue
  
Merged
Development

Successfully merging this pull request may close these issues.

None yet

4 participants