[MXNET-346] Hard Sigmoid Operator #10633
Conversation
| DMLC_REGISTER_PARAMETER(HardSigmoidParam); | ||
| MXNET_OPERATOR_REGISTER_UNARY(hard_sigmoid) | ||
| .describe(R"code(Computes hard sigmoid of x element-wise. | ||
|
|
There was a problem hiding this comment.
You could add an example here.
There was a problem hiding this comment.
It's unnecessary for this kind of function, please refer to "sigmoid" function above.
|
@rahul003 @reminisce @anirudh2290 @eric-haibin-lin Could you please give a review for this? |
|
It’s widely used in LSTM in Keras. Could add this op into LSTM or GRU, etc.. |
|
@chinakook Thanks for your advice! I'll definitely talk to related persons about this. |
haojin2
force-pushed
the
hard_sigmoid
branch
2 times, most recently
from
218fda1
to
5f875ec
Compare
|
|
||
|
|
||
| template<typename xpu> | ||
| void HardSigmoidForward(const nnvm::NodeAttrs& attrs, |
There was a problem hiding this comment.
why do you need to write forward and backward? You can use unary compute like https://github.com/apache/incubator-mxnet/blob/master/src/operator/tensor/elemwise_unary_op_basic.cc#L87
There was a problem hiding this comment.
Okay, I'll move the kernels to mshadow_op.h, does that sound good?
There was a problem hiding this comment.
Actually there's a problem with that, as hard sigmoid takes 2 parameters, which is different from other unary operators.
There was a problem hiding this comment.
So it's okay to leave this in this way?
| }; | ||
|
|
||
| template<int req> | ||
| struct hard_sigmoid_forward { |
There was a problem hiding this comment.
please inherit tunable for fwd and bwd and add a entry here https://github.com/apache/incubator-mxnet/blob/daf821c09b3df581399a564a1b40b0a2e68750c9/src/operator/operator_tune.cc#L347
see https://github.com/apache/incubator-mxnet/blob/master/src/operator/mxnet_op.h#L596
There was a problem hiding this comment.
Hi Eric, there's a major difference between the given op and hard sigmoid here. set_to_int have a version of Map that takes 0 arguments so it could make use of the existing IMPLEMENT_BLANK_WORKLOAD_FWD macro, but hard sigmoid is a tertiary function, so I may need to add IMPLEMENT_TERTIARY_WORKLOAD_FWD and IMPLEMENT_TERTIARY_WORKLOAD_BWD in operator_tune.cc together with a new TertiaryOpTune class in operator_tune-inl.h. I'm not sure if there's any more functions that could make use of this, if so then I guess it's necessary to implement the things mentioned above.
There was a problem hiding this comment.
@cjolivier01 would know more about this. He is drafting a doc for autotune. I'll merge this for now. Please revisit after you go through that doc.
There was a problem hiding this comment.
Sure, I actually talked with him yesterday after seeing your comments and those were what he told me, I think if we see more similar functions with this one then we can definitely get that done.
Description
Implementation of Hard Sigmoid function
Checklist
Essentials
Changes
Comments
For bridging the operator gaps between MXNet and ONNX