Add mkldnn_softmax #4331
Merged
Add mkldnn_softmax #4331
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tensor-tang
force-pushed
the
mkldnn_softmax
branch
from
89cd07a
to
799f80a
Compare
luotao1
reviewed
Sep 25, 2017
| return Error(); | ||
| /** | ||
| * @brief Base class of MKLDNN softmax Activation, | ||
| * only have mkldnn forward, use cpu implement for backward. |
There was a problem hiding this comment.
因为mkldnn目前还没有实现softmax的backward，所以先保留与CPU的实现一样。
| "mkldnn_" #ACT_TYPE); \ | ||
| }); | ||
|
|
||
| /** | ||
| * @def DEFINE_MKLDNN_ELTWISE_ACTIVATION | ||
| */ | ||
| #define DEFINE_MKLDNN_ELTWISE_ACTIVATION(ACT_TYPE, ALPHA, BWD_ALPHA) \ |
There was a problem hiding this comment.
class MKLDNN_ACTIVATION_CLASS_NAME类能继承MKLDNN_ACTIVATION_CLASS_NAME么？
·这样以下内容就不用重复写了：
55行，60行，64-65行，68-71行。
paddle/math/Matrix.cpp
Outdated
| @@ -3637,7 +3637,7 @@ void CpuMatrix::oneHotCrossEntropy(Matrix& output, IVector& label) { | |||
| for (size_t i = 0; i < numSamples; ++i, out += dim) { | |||
| CHECK_GE(lbl[i], 0); | |||
| CHECK_LT((size_t)lbl[i], dim); | |||
| cost[i] = -std::log(out[lbl[i]]); | |||
| cost[i] = -std::log(std::max(out[lbl[i]], real(FLT_MIN))); | |||
There was a problem hiding this comment.
FLT_MIN这个数是在哪儿设置的？paddle中没有这个变量。
There was a problem hiding this comment.
定义在#include <float.h>里面
paddle/math/Matrix.cpp
Outdated
| @@ -3652,7 +3652,7 @@ void CpuMatrix::oneHotCrossEntropyBp(Matrix& output, IVector& label) { | |||
| real* grad = getData(); | |||
| int* lbl = label.getData(); | |||
| for (size_t i = 0; i < numSamples; ++i, out += dim, grad += dim) { | |||
| grad[lbl[i]] -= 1 / out[lbl[i]]; | |||
| grad[lbl[i]] -= 1 / std::max(out[lbl[i]], real(FLT_MIN)); | |||
There was a problem hiding this comment.
定义在#include <float.h>里面
| @@ -93,42 +128,21 @@ class MKLDNNEltwiseActivation : public MKLDNNActivation { | |||
| return (mkldnn::algorithm)0; | |||
There was a problem hiding this comment.
119-124行，可以用正则表达式，把mkldnn换成eltwise
tensor-tang
force-pushed
the
mkldnn_softmax
branch
2 times, most recently
from
c5216eb
to
d5f5828
Compare
tensor-tang
force-pushed
the
mkldnn_softmax
branch
from
d5f5828
to
2c6ac62
Compare
luotao1
reviewed
Sep 26, 2017
| if (outputG->useGpu()) { | ||
| outputG->softmaxBackward(*outputV); | ||
| } else { | ||
| SetDevice device(act.deviceId); |
There was a problem hiding this comment.
这一段是直接复制的SoftmaxActivation::backward吧，多复制了193-196行。这里可以使用MatrixPtr么？
There was a problem hiding this comment.
确实可以删掉，thx。 done
tensor-tang
force-pushed
the
mkldnn_softmax
branch
2 times, most recently
from
007a64a
to
043aa8a
Compare
tensor-tang
force-pushed
the
mkldnn_softmax
branch
from
043aa8a
to
672c968
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
solve #4330