Exponential Linear Units

[mohomran]

Implementation of the Exponential Linear Units proposed in:

Clevert, D.-A., Unterthiner, T., & Hochreiter, S. (2015). Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs). http://arxiv.org/abs/1511.07289

I made one minor modification to the formula from the paper: f(x) = x, if x > 0 rather than if x >=0, with the corresponding change to the gradient. I did this for two reasons:

1. This way, when alpha = 0, an ELU reduces exactly to a ReLU as implemented in Caffe - basically: f'(0) = 0 instead of 1 as specified in the paper.
2. Also given the original formula, when alpha = 0 the loss would diverge during MNIST training. I would be happy to receive additional verification and to revise this change if necessary.

[beniz]

Great job! Was actually coming to check on ELU, found this :) Will report on performances when I can.

[f0k]

I made one minor modification to the formula from the paper: f(x) = x, if x > 0 rather than if x >=0, with the corresponding change to the gradient.

It seems this is actually what they did for the paper as well:
untom/binet@2c8a6bd
@untom, you might want to change the formula in the paper accordingly!

[untom]

Thanks for the head's up :)

Note that mathematically that as long as alpha == 1, this doesn't make a difference since exp(0) == 1, so both transfer function and gradient output the same thing regardless of > vs >= . Also due to the way ELUs look, it's pretty hard for an activation to hit 0 precisely, anyhow. But you're right, we used > 0 during our own experiments, both in the binet code as well as in our own caffe fork. If we make another paper revision we will definitely include that change.

[shelhamer]

Thanks for this @mohomran! That was quick.

I'm sorry that this was caught by the switch to layer headers in #3315 but could you update this to reflect the new arrangement? See the new ReLU header for an example.

[mohomran]

@beniz: Thanks. :) So far, I've only tested it on MNIST and CIFAR-10 ("quick"), but neither network is deep enough to result in significant gains according to the paper. The updated CIFAR-10 network seemed to converge a bit faster though.

@f0k, @untom: Thanks, good to know! As said, I encountered problems when alpha was set to 0, which prompted the change.

@shelhamer: Rebased and ready to go. :)

[shelhamer]

Drop commented code.

[shelhamer]

@jeffdonahue when Leaky ReLU was added it was incorporated into ReLU in #740. Do you have an opinion on a separate ELU layer?

[jeffdonahue]

I'd be fine with incorporating it into ReLU if there's a near 0 performance impact, but this feels more to me like it should be a separate layer than leaky ReLU (which felt like a more natural generalization to me, still being piecewise linear).

[beniz]

@mohomran so I've tested on GoogleNet, an even with BN activated, just for the sake of it. It appears to work fine, though the memory requirement appears to grow significantly, which translates into smaller batches. The typical memory error (or so I guess) happens on the CUDA_POST_KERNEL_CHECK in elu_layer.cu. FTR, I had cuDNN activated though of course ELU is not using it. I have some GPU time to kill over the next few days if some more experiments or reports can help.
EDIT: the memory bump is likely due to not using CuDNN vs using it for ReLU.

[vchuravy]

@untom It does make a difference for the gradient, for any a != 0

[untom]

Is there anything I can do to help move this PR forward?

[shelhamer]

Thanks for the non-linearity @mohomran and thanks for checking in regarding the paper details @untom!