Add algorithmic optimizer to convert Log(Softmax(x)) to LogSoftmax(x) #25455
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
lgeiger
force-pushed
the
optimize_logsoftmax
branch
from
265b477
to
d8797a5
Compare
rthadur
added
awaiting review
rthadur
added
awaiting review
|
@rmlarsen Could you PTAL and approve or suggest changes(if required). |
lgeiger
force-pushed
the
optimize_logsoftmax
branch
from
d8797a5
to
8fb50c8
Compare
This PR adds an algorithmic optimizer which converts `Log(Softmax(x))` to `LogSoftmax(x)`. [`LogSoftmax`](https://www.tensorflow.org/api_docs/cc/class/tensorflow/ops/log-softmax) is numerically more stable and may be a bit faster in some cases. This could be expanded in the future to also optimize `log(softmax(x) * y) = logsoftmax(x) + log(y)` and `log(softmax(x) / y) = logsoftmax(x) - log(y)`.
lgeiger
force-pushed
the
optimize_logsoftmax
branch
from
8fb50c8
to
597f02c
Compare
|
Rebased since #25300 introduced some merge conflicts. |
|
@hgadig Any updates on this? |
Waiting for the review. Also, this is assigned to rthadur and he will be taking care of this PR(fyi). Thanks ! |
ezhulenev
approved these changes
Mar 6, 2019
|
Thanks! Sorry for long review cycle. |
rthadur
added
kokoro:force-run
joshrose
pushed a commit
to joshrose/tensorflow
that referenced
this pull request
Mar 7, 2019
PiperOrigin-RevId: 237148047
Thanks @ezhulenev, you've been really helpful with getting my fixes merged. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds an algorithmic optimizer which converts
Log(Softmax(x))toLogSoftmax(x).LogSoftmaxis numerically more stable and may be a bit faster in some cases.This could be expanded in the future to also optimize
log(softmax(x) * y) = logsoftmax(x) + log(y)andlog(softmax(x) / y) = logsoftmax(x) - log(y).