activation_layers.rst

Activation Layers

Layer	Description
Elu	Exponential linear unit
Identity	Output the input tensor
LeakyRelu	Leaky relu
LogSoftmax	Logarithm of softmax function
Relu	Rectified linear unit
Softmax	Softmax

Elu

The Elu layer is similar to Relu but with negative values that cause the mean of the Elu activation function to shift toward 0.

$$\begin{aligned} \text{ELU}(x; \alpha) = \begin{cases} x & x > 0 \\\ \alpha (e^x - 1) & x \leq 0 \end{cases} \end{aligned}$$

α should be non-negative. See:

Djork-Arne Clevert, Thomas Unterthiner, and Sepp Hochreiter. "Fast and accurate deep network learning by exponential linear units (ELUs)." arXiv preprint arXiv:1511.07289 (2015).

Arguments:

alpha

(double, optional) Default: 1. Should be >=0

Back to Top<activation-layers>

Identity

The Identity layer outputs the input tensor.

This layer is very cheap since it just involves setting up tensor views.

Arguments: None

Back to Top<activation-layers>

LeakyRelu

LeakyRelu modifies the Relu function to allow for

a small, non-zero gradient when the unit is saturated and not active.

$$\begin{aligned} \text{LeakyReLU}(x; \alpha) = \begin{cases} x & x > 0 \\\ \alpha x & x \leq 0 \end{cases} \end{aligned}$$

See:

Andrew L. Maas, Awni Y. Hannun, and Andrew Y. Ng. "Rectifier nonlinearities improve neural network acoustic models." In Proc. ICML, vol. 30, no. 1, p. 3. 2013.

Arguments:

negative_slope

(double, optional) Default: 0.01

Back to Top<activation-layers>

LogSoftmax

LogSoftmax is the logarithm of the softmax function.

log softmax(x)_i = x_i − log ∑_je^x_j

Arguments: None

Back to Top<activation-layers>

Relu

The Relu layer outputs input directly if positive, otherwise outputs zero.

ReLU(x) = max(x, 0)

Arguments: None

Back to Top<activation-layers>

Softmax

The Softmax layer turns a vector of K real values into a vector of K real values that sum to 1.

$$\text{softmax}(x)_i = \frac{e^{x_i}}{\sum_j e^{x_j}}$$

Arguments:

softmax_mode

(string, optional) Options: instance (default), channel

Back to Top<activation-layers>