Flattened Threshold Swish Activation function - PyTorch implementation
PyTorch implementation of Flattened, Threshold Swish like activation function for deep learning. The theory was developed in this paper: https://arxiv.org/abs/1812.06247
Added ability for mean shift, adjustable threshold and max value clamping.
FTSwish is:
X>0 = Relu(x) * Sigmoid(x) +T
x<0 = T (default = -.20)
For positive value it mimics swish activation, minus the threshold. For negative values it allows a fixed threshold for < 0 values.