Swish-1 and GELU activation for transformers

[mjdenkowski]

This adds support for two activation types to transformers:

• Swish-1 (Ramachandran et al. 2017: Searching for Activation Functions)
• GELU (Hendrycks and Gimpel 2016: Bridging Nonlinearities and Stochastic Regularizers with Gaussian Error Linear Units)

The first paper reports that Swish-1 beats ReLU on transformer models. The second reports that GELU beats Swish-1, but not explicitly tested on transformers.

Edit: If we want to avoid a bunch of sequential major version bumps, this can wait until #226 is ready to go and we can have a single bump for all transformer updates.

Pull Request Checklist

• Changes are complete (if posting work-in-progress code, prefix your pull request title with '[WIP]'
  until you can check this box.
• Unit tests pass (pytest)
• System tests pass (pytest test/system)
• Passed code style checking (./pre-commit.sh or manual run of pylint & mypy)
• You have considered writing a test
• Updated major/minor version in sockeye/__init__.py. Major version bump if this is a backwards incompatible change.
• Updated CHANGELOG.md

[fhieber]

+1 on syncing with #226 and #222 to not end up with Sockeye 1.inf :)

[fhieber]

Cool change, thanks!

[fhieber]

Have you observed increased memory or speed decreases with the GELU?
You could think about implementing these as custom operators to avoid creating multiple nodes in the computation graph if these turn out to work better than the Relu.
At least for swish, the backward implementation is easy, but the savings will be negligible there.

[mjdenkowski]

Compared to ReLU, decoding with Swish-1 takes 100.5% time and GELU takes 103.5% time. Swish-1 uses the same amount of memory and GELU uses 110% memory. Swish-1 is basically free while GELU is more expensive, but so far shows the best perplexity. If they work consistently better than ReLU, we should definitely look at performance optimization. I think this base implementation is fine for experiments.

[fhieber]

sounds good, we can revisit this if and when it becomes a bottleneck.

[mjdenkowski]

This should be up to date and ready to merge.