MKL DNN: fix the TF1.6 speed issue by fixing MKL DNN LRN taking the optimum path #17605
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tatianashp
added
awaiting testing (then merge)
kokoro:force-run
|
Thank you for the fix!. The speed issue is the one described in #17383, correct? |
|
@tatianashp Yes |
frankchn
approved these changes
Mar 13, 2018
tatianashp
pushed a commit
to tatianashp/tensorflow
that referenced
this pull request
Mar 16, 2018
…ptimum path (tensorflow#17605) * MKL DNN: fix the TF1.6 speed issue by fixing MKL DNN LRN * fixed typos in the doc for LrnRewrite
StanislawAntol
pushed a commit
to StanislawAntol/tensorflow
that referenced
this pull request
Mar 23, 2018
…ptimum path (tensorflow#17605) * MKL DNN: fix the TF1.6 speed issue by fixing MKL DNN LRN * fixed typos in the doc for LrnRewrite
StanislawAntol
pushed a commit
to StanislawAntol/tensorflow
that referenced
this pull request
Mar 23, 2018
…ptimum path (tensorflow#17605) (tensorflow#17751) * MKL DNN: fix the TF1.6 speed issue by fixing MKL DNN LRN * fixed typos in the doc for LrnRewrite
copybara-service bot
pushed a commit
that referenced
this pull request
Mar 14, 2023
Imported from GitHub PR keras-team/keras#17605 Add Lion optimizer to keras, using the experimental optimizer API. Authors [implementation](http://github.com/google/automl/tree/master/lion) is based on legacy optimizer API. Request for contribution was made [here](https://github.com/keras-team/keras/issues/17570) Copybara import of the project: -- 89ba3e45094943931d577c9c21194aaec1764f97 by Malo <malo@milvue.com>: Add Lion optimizer -- 6074929ccdfd2d4889a7c039e8b7163236c3533a by Malo <malo@milvue.com>: Add missing docstring, remove checks -- 51041a5915c772a67314781c186639184993884a by Malo <malo@milvue.com>: improve update step -- 0e0d77fd5f1544545980e9388bb0264751e01997 by Malo <malo@milvue.com>: remove not needed stuff -- 2ede9c711ecdc731d644bc913944fe24d1b45c8c by Malo <malo@milvue.com>: add lr and wd value recommendation -- 2a8e49939b37791f39b8f537bbe54e1f7187a376 by Malo <malo@milvue.com>: allow beta_1 = 0 -- 61224483219906ce4889cc9e94408ffa12d29b19 by Malo <malo@milvue.com>: revert beta_1 = 0 and add correctness test -- b368b93f88f4c9cedf0f0fe0c4a480d795f231ca by Malo <malo@milvue.com>: add missing newline -- 4901592dd143f682c5ca43861dc5917977ee41dc by Malo <malo@milvue.com>: revert back to register_keras + print value -- fee5345c724ff53c056055a151c2f39b1445797e by Malo <malo@milvue.com>: improve error message Merging this change closes #17605 PiperOrigin-RevId: 516388327
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There is a performance regression for TF 1.6 comparing to TF 1.5 for cifar 10. The root cause it cifar 10 uses depth radius = 4, for which MKL DNN takes unoptimized path. Thus we fix this issue by using following strategy:
If the depth_radius of LRN is not 2, then MKL DNN takes unoptimized path. The unoptimized path is slow. Thus we dont rewrite the node and use default Eigen. But for depth_radius=2, MKL DNN optimized
path is taken, i.e., eigen LRN node is rewritten by MKl DNN LRN node.