[feat] Add optimizer state sharding (ZeRO) #636
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
facebook-github-bot
added
CLA Signed
|
This pull request was exported from Phabricator. Differential Revision: D24317858 |
vedanuj
added a commit
to vedanuj/mmf
that referenced
this pull request
Oct 19, 2020
Summary: Pull Request resolved: facebookresearch#636 Adding Zero optimizer state sharding to MMF from [fairscale](https://github.com/facebookresearch/fairscale) library. Added to tutorials It can used with this option : `optimizer.enable_state_sharding=True` Differential Revision: D24317858 fbshipit-source-id: bc6cd687f0e580716b71b7a4daf143dc5097cc3e
vedanuj
force-pushed
the
export-D24317858
branch
from
2466ea9
to
bc3d78a
Compare
|
This pull request was exported from Phabricator. Differential Revision: D24317858 |
vedanuj
added a commit
to vedanuj/mmf
that referenced
this pull request
Oct 20, 2020
Summary: Pull Request resolved: facebookresearch#636 Adding Zero optimizer state sharding to MMF from [fairscale](https://github.com/facebookresearch/fairscale) library. Added to tutorials It can used with this option : `optimizer.enable_state_sharding=True` Differential Revision: D24317858 fbshipit-source-id: 471ce803f716b81dc4928d8cba281cceb75ab8d4
vedanuj
force-pushed
the
export-D24317858
branch
from
bc3d78a
to
72a7537
Compare
|
This pull request was exported from Phabricator. Differential Revision: D24317858 |
vedanuj
added a commit
to vedanuj/mmf
that referenced
this pull request
Oct 23, 2020
Summary: Pull Request resolved: facebookresearch#636 Adding Zero optimizer state sharding to MMF from [fairscale](https://github.com/facebookresearch/fairscale) library. Added to tutorials It can used with this option : `optimizer.enable_state_sharding=True` Differential Revision: D24317858 fbshipit-source-id: 4d3e09697df4f4f3b7c84e337d3287595e1b421f
vedanuj
force-pushed
the
export-D24317858
branch
from
72a7537
to
90c717d
Compare
|
This pull request was exported from Phabricator. Differential Revision: D24317858 |
1 similar comment
|
This pull request was exported from Phabricator. Differential Revision: D24317858 |
vedanuj
added a commit
to vedanuj/mmf
that referenced
this pull request
Oct 23, 2020
Summary: Pull Request resolved: facebookresearch#636 Adding Zero optimizer state sharding to MMF from [fairscale](https://github.com/facebookresearch/fairscale) library. Added to tutorials It can used with this option : `optimizer.enable_state_sharding=True` Differential Revision: D24317858 fbshipit-source-id: e1c25d81c231f7ab18fbe61e374a539d988e4556
vedanuj
force-pushed
the
export-D24317858
branch
from
90c717d
to
1ffa8b9
Compare
vedanuj
added a commit
to vedanuj/mmf
that referenced
this pull request
Oct 23, 2020
Summary: Pull Request resolved: facebookresearch#636 Adding Zero optimizer state sharding to MMF from [fairscale](https://github.com/facebookresearch/fairscale) library. Added to tutorials It can used with this option : `optimizer.enable_state_sharding=True` Differential Revision: D24317858 fbshipit-source-id: aa2b5868a9cddddaf09cb045ea2c5be6bec86b14
vedanuj
force-pushed
the
export-D24317858
branch
from
1ffa8b9
to
5b51998
Compare
|
This pull request was exported from Phabricator. Differential Revision: D24317858 |
apsdehal
reviewed
Oct 24, 2020
.circleci/config.yml
Outdated
| @@ -22,6 +22,7 @@ install_dep: &install_dep | |||
| command: | | |||
| source activate mmf | |||
| pip install --upgrade setuptools | |||
| pip install --progress-bar off -r requirements.txt | |||
There was a problem hiding this comment.
I am somewhat not a fan of this as it creates confusion. setup.py should in itself be enough to install any dependencies.
mmf/utils/build.py
Outdated
| optimizer = optimizer_class(parameters, **params) | ||
|
|
||
| if optimizer_config.get("enable_state_sharding", False): | ||
| from fairscale.optim.oss import OSS |
There was a problem hiding this comment.
We can also do a try, except ImportError and ask user to install the fairscale library separately if we don't want explicit dependency on it. This would also solve some things on fbcode side, user who want it should include fairscale in the targets. Adding more dependency always causes more pain.
vedanuj
added a commit
to vedanuj/mmf
that referenced
this pull request
Oct 24, 2020
Summary: Pull Request resolved: facebookresearch#636 Adding Zero optimizer state sharding to MMF from [fairscale](https://github.com/facebookresearch/fairscale) library. Added to tutorials It can used with this option : `optimizer.enable_state_sharding=True` Differential Revision: D24317858 fbshipit-source-id: 90161c7b3314b631e85e59e50ca1701ed8daec79
vedanuj
force-pushed
the
export-D24317858
branch
from
5b51998
to
a9f8f65
Compare
|
This pull request was exported from Phabricator. Differential Revision: D24317858 |
Summary: Pull Request resolved: facebookresearch#636 Adding Zero optimizer state sharding to MMF from [fairscale](https://github.com/facebookresearch/fairscale) library. Added to tutorials It can used with this option : `optimizer.enable_state_sharding=True` Differential Revision: D24317858 fbshipit-source-id: 6c16e0614bb568c9ac35a70760af560d4a8c4d60
vedanuj
force-pushed
the
export-D24317858
branch
from
a9f8f65
to
25fa55e
Compare
|
This pull request was exported from Phabricator. Differential Revision: D24317858 |
apsdehal
reviewed
Oct 29, 2020
| @@ -226,7 +226,24 @@ def build_optimizer(model, config): | |||
| ) | |||
|
|
|||
| parameters = get_optimizer_parameters(model, config) | |||
| optimizer = optimizer_class(parameters, **params) | |||
|
|
|||
| if optimizer_config.get("enable_state_sharding", False): | |||
There was a problem hiding this comment.
It would make sense to write a simple test to test this. We have had issue with build_optimizer in past.
apsdehal
approved these changes
Oct 29, 2020
|
This pull request has been merged in 3ae71ec. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
Adding Zero optimizer state sharding to MMF from fairscale library.
Added to tutorials
It can used with this option :
optimizer.enable_state_sharding=TrueDifferential Revision: D24317858