Fix Softplus precision issue#10576
Conversation
| ConstEigenVectorArrayMap<T> xm(this->input + first, len); | ||
| EigenVectorArrayMap<T> ym(output_ptr, len); | ||
| ym = (xm > 0).select(xm + ((-xm).exp() + 1.0f).log(), ((xm).exp() + 1.0f).log()); | ||
| ym = (xm > 0).select(xm + ((-xm).exp()).log1p(), ((xm).exp()).log1p()); |
There was a problem hiding this comment.
There is a bit concern here, since Eigen log use SSE2 and AVX, while log1p use std::log1p, this may cause the perf loss on certain platform.
See, https://eigen.tuxfamily.org/dox/group__CoeffwiseMathFunctions.html
There was a problem hiding this comment.
@yufenglee is there anything in mlas that could be used here?
There was a problem hiding this comment.
MLAS has MlasComputeExp and MlasComputeLogistic, but doesn't have one for Softplus. If we compute Softplus with MlasComputeExp and MlasComputeLogistic, it will go though memory multiple times.
Looks like this activation is not used commonly. If it is, we can add optimization for it.
There was a problem hiding this comment.
AFAIK Softmax is typically used as a post-processing step and often done outside of the model or as a final step inside the model. As such perf may not be critical.
There was a problem hiding this comment.
This is for Softplus, not Softmax
There was a problem hiding this comment.
Ahh sorry - I have never seen a model that uses Softplus.
Correctness has to come first. If there's a perf issue that affects a production model we can look at adding something to MLAS in the future. That said, from some very brief research it seems like Relu is generally preferred over Softplus, so any perf issue could potentially be addressed by switching to that.
3 participants
Description: Fix Softplus precision issue
Motivation and Context
log(1+x)if the numberxis small. Eigen provides thelog1pfor log 1 plus with higher precision