Add SOFTPLUS_FWD and SWISH_FWD pointwise ops#355
Conversation
Emit torch.aten.softplus with configurable beta and threshold attributes (defaulting to 1.0 and 20.0) and torch.aten.silu for swish. Adds sample and lit tests for both ops. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: Rob Suderman <rob.suderman@gmail.com>
sjain-stanford
left a comment
sjain-stanford
left a comment
There was a problem hiding this comment.
This breaks compat with cudnn / hipdnn's swish for beta != 1. The swish function is not a SILU when beta !=1.
Swish: https://en.wikipedia.org/wiki/Swish_function
https://docs.pytorch.org/docs/stable/generated/torch.nn.SiLU.html
Requesting changes (in a follow-on since this landed before I could review):
Include setSwishBeta and getSwishBeta methods on the attribute (with default 1.0f). Then either 1) reject non-default beta in pre-validate, or 2) emit aten.sigmoid + aten.mul for beta != 1, and aten.silu otherwise.
In general, please wait for a comment from either me or one of the other contributors who is familiar with adding new ops (like @IanWood1).
| double xD = static_cast<double>(x); | ||
| // SOFTPLUS(x) = log(1 + exp(x)); for x > threshold, result ~= x. | ||
| // Matches torch.aten.softplus with beta=1.0 and threshold=20.0. | ||
| constexpr double threshold = 20.0; |
There was a problem hiding this comment.
nit: use k-prefix convention for constants.. kSoftplusThreshold
There was a problem hiding this comment.
The k prefix only applies to global constants. We could change the clang-tidy rules to apply to local ones, too. There are some other cases without the prefix. e.g.
fusilli/include/fusilli/node/conv_node.h
Line 341 in aeed46f
4 participants
Emit torch.aten.softplus with configurable beta and threshold attributes (defaulting to 1.0 and 20.0) and torch.aten.silu for swish. Adds sample and lit tests for both ops.