Skip to content

ZeroRedundancyOptimizer: an implementation of a standalone sharded optimizer wrapper #46750

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 52 commits into from
Closed

ZeroRedundancyOptimizer: an implementation of a standalone sharded optimizer wrapper #46750

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
52 commits
Select commit Hold shift + click to select a range
36587bf
First commit straight from Fairscale
blefaudeux Oct 15, 2020
19e43ee
pulling in unit test
blefaudeux Oct 15, 2020
342918c
unit test up and running, could be clearner
blefaudeux Oct 15, 2020
4742740
Merge remote-tracking branch 'origin/master' into sharded_optimizer
blefaudeux Oct 22, 2020
ce11a2a
zeroptimizer, rename and interface
blefaudeux Oct 22, 2020
cdddf35
adjusting the unit test
blefaudeux Oct 22, 2020
24185f6
removing default optim, running unit test
blefaudeux Oct 22, 2020
c72fd7f
cleeaner bucketing
blefaudeux Oct 23, 2020
b94a577
reverting changes which slipped in
blefaudeux Oct 23, 2020
84169a2
Merge remote-tracking branch 'origin/master' into sharded_optimizer
blefaudeux Oct 23, 2020
873e253
Fix broken copy pasta
blefaudeux Oct 23, 2020
c6941d5
trailing bucket case, fixed
blefaudeux Oct 23, 2020
bd74249
brainfart, sorry about that
blefaudeux Oct 23, 2020
bb770c0
partial code review, not everything is addressed.
blefaudeux Oct 26, 2020
78cfdc5
Merge branch 'master' into sharded_optimizer
blefaudeux Nov 12, 2020
5c5c65e
adding clip_grad_norms and the matching test
blefaudeux Nov 12, 2020
fd9e490
linting
blefaudeux Nov 12, 2020
5a8f475
some more linting
blefaudeux Nov 12, 2020
ccab3ae
Merge remote-tracking branch 'origin/master' into sharded_optimizer
blefaudeux Nov 12, 2020
a465be0
Renaming and code update
blefaudeux Dec 4, 2020
54f678c
addressing some comments, trying to get the torch dev setup back to life
blefaudeux Jan 7, 2021
922feec
wip
blefaudeux Jan 8, 2021
dd034e7
rough update to the current fairscale state, needs testing and many l…
blefaudeux Jan 8, 2021
6ad4bf9
more ironing out, not qute there
blefaudeux Jan 12, 2021
8ad1853
Merge branch 'sharded_optimizer' of github.com:blefaudeux/pytorch; br…
blefaudeux Jan 12, 2021
3b0d209
Merge branch 'master' into sharded_optimizer
blefaudeux Jan 12, 2021
c32e927
unit tests now locally passing except for world-size related once, ne…
blefaudeux Jan 12, 2021
a492151
enabling/fixing the two last tests which are world size dependent
blefaudeux Jan 12, 2021
766ba48
more fixes, still trying to figure out the documentation
blefaudeux Jan 12, 2021
5ddb6c1
mypy fixes
blefaudeux Jan 15, 2021
a274bb4
flake8 fixes, better docstring
blefaudeux Jan 15, 2021
5f10f7d
code review, some cleaning up
blefaudeux Jan 15, 2021
5b76958
Merge remote-tracking branch 'origin/master' into sharded_optimizer
blefaudeux Jan 15, 2021
c6c8148
unit test cleaning up, still a rough time building from source to tes…
blefaudeux Jan 15, 2021
1154696
flake8..
blefaudeux Jan 15, 2021
e21ef38
unit test fix, now need to check cuda ones
blefaudeux Jan 15, 2021
36f2033
unit test -> spawn/fork as suggested by the asan test, slightly bette…
blefaudeux Jan 17, 2021
a382724
formatting, really in need of a proper setup
blefaudeux Jan 17, 2021
ecbd05a
yet another run of typos + windows blocklist for the test, torch.dist…
blefaudeux Jan 18, 2021
3b9bebb
adding some more notes
blefaudeux Jan 18, 2021
0d26098
unit test minor improvements, hard to debug distributed and remote
blefaudeux Jan 18, 2021
67f0925
enabling all tests but DDP parity to run on all platforms
blefaudeux Jan 18, 2021
3fb0992
typos while trying to get conda up..
blefaudeux Jan 19, 2021
0658dd1
Merge remote-tracking branch 'origin/master' into sharded_optimizer
blefaudeux Jan 19, 2021
a7939b8
fix unit tests, at least through local testing
blefaudeux Jan 19, 2021
d283d75
default to cpu tensors if not NCCL
blefaudeux Jan 19, 2021
8f5d539
fixing some unit tests tied to a fixed world size, preemptive fix on …
blefaudeux Jan 20, 2021
3442f71
adding the doc, still need to fix a unit test for some platform
blefaudeux Jan 20, 2021
e121e57
skip two tests with rocm, add some documentation
blefaudeux Jan 20, 2021
8cc99fa
removing the doc changes, will go in another PR
blefaudeux Jan 20, 2021
cbb02e2
Merge remote-tracking branch 'origin/master' into sharded_optimizer
blefaudeux Jan 20, 2021
4a75669
Merge branch 'master' into sharded_optimizer
blefaudeux Jan 20, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Loading