-
Notifications
You must be signed in to change notification settings - Fork 324
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Swish-1 and GELU activation for transformers #229
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Cool change, thanks!
:param act_type: Type of activation. | ||
:return: output Symbol with same shape as input. | ||
""" | ||
# TODO: Contribute these to MXNet? For now it appears that registered activation types must be implemented in C++. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Have you observed increased memory or speed decreases with the GELU?
You could think about implementing these as custom operators to avoid creating multiple nodes in the computation graph if these turn out to work better than the Relu.
At least for swish, the backward implementation is easy, but the savings will be negligible there.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Compared to ReLU, decoding with Swish-1 takes 100.5% time and GELU takes 103.5% time. Swish-1 uses the same amount of memory and GELU uses 110% memory. Swish-1 is basically free while GELU is more expensive, but so far shows the best perplexity. If they work consistently better than ReLU, we should definitely look at performance optimization. I think this base implementation is fine for experiments.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
sounds good, we can revisit this if and when it becomes a bottleneck.
This should be up to date and ready to merge. |
This adds support for two activation types to transformers:
The first paper reports that Swish-1 beats ReLU on transformer models. The second reports that GELU beats Swish-1, but not explicitly tested on transformers.
Edit: If we want to avoid a bunch of sequential major version bumps, this can wait until #226 is ready to go and we can have a single bump for all transformer updates.
Pull Request Checklist
until you can check this box.
pytest
)pytest test/system
)./pre-commit.sh
or manual run of pylint & mypy)sockeye/__init__.py
. Major version bump if this is a backwards incompatible change.