Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove Fused LayerNorm deprecation #1931

Merged
merged 5 commits into from
Feb 1, 2023

Conversation

nik-mosaic
Copy link
Contributor

A deprecation warning for Fused LayerNorm was added in PR #1789, informing users that FLN would be removed in Composer 0.13. The thought behind this was that Low Precision LayerNorm was strictly better, since it does not depend on Apex, and has equivalent performance.

However, our NLP testing plan found that Low Precision LayerNorm is not scriptable (i.e. we cannot use torchscript to export a model with LowPrecision LayerNorm for inference), making it inferior to Fused LayerNorm for users who want to export their models.

We should hold off deprecating Fused LayerNorm until this issue has been resolved, since it is an important component of many NLP models. Both algorithms can live side-by-side for the time being---the documentation for LPLN clearly states it is an alternative to FLN.

@nik-mosaic nik-mosaic marked this pull request as ready for review January 31, 2023 22:17
@nik-mosaic nik-mosaic merged commit b4aea03 into mosaicml:dev Feb 1, 2023
@nik-mosaic nik-mosaic deleted the remove-deprecation branch February 1, 2023 01:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants