Enhance Distributed Fused Adam #1832
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Wil Kong <alpha0422@gmail.com>
Signed-off-by: Wil Kong <alpha0422@gmail.com>
Signed-off-by: Wil Kong <alpha0422@gmail.com>
Signed-off-by: Wil Kong <alpha0422@gmail.com>
Signed-off-by: Wil Kong <alpha0422@gmail.com>
Signed-off-by: Wil Kong <alpha0422@gmail.com>
Signed-off-by: Wil Kong <alpha0422@gmail.com>
Signed-off-by: Wil Kong <alpha0422@gmail.com>
… copy after all-gather. Signed-off-by: Wil Kong <alpha0422@gmail.com>
Signed-off-by: Wil Kong <alpha0422@gmail.com>
Call unscale_grads within step if grad scaler is provided. Revert grad clipping logic. Signed-off-by: Tim Moon <tmoon@nvidia.com>
alpha0422
force-pushed
the
wkong/dist-adam
branch
from
f9f2061 to
d9c0765
Compare
crcrpar
reviewed
Aug 22, 2024
apex/contrib/csrc/optimizers/multi_tensor_distopt_adam_kernel.cu
Outdated
Show resolved
Hide resolved
apex/contrib/csrc/optimizers/multi_tensor_distopt_adam_kernel.cu
Outdated
Show resolved
Hide resolved
timmoon10
approved these changes
Aug 23, 2024
Contributor
timmoon10
left a comment
timmoon10
left a comment
There was a problem hiding this comment.
Overall looks good to me aside from some stylistic suggestions.
apex/contrib/csrc/optimizers/multi_tensor_distopt_adam_kernel.cu
Outdated
Show resolved
Hide resolved
Co-authored-by: Tim Moon <4406448+timmoon10@users.noreply.github.com>
Co-authored-by: Tim Moon <4406448+timmoon10@users.noreply.github.com>
This reverts commit 857e8f4.
crcrpar
reviewed
Aug 29, 2024
Collaborator
crcrpar
left a comment
crcrpar
left a comment
There was a problem hiding this comment.
could you add a test case for capturable?
crcrpar
approved these changes
Aug 30, 2024
Collaborator
crcrpar
left a comment
crcrpar
left a comment
There was a problem hiding this comment.
agreed on @alpha0422 will add a test in a follow-up.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Continue of #1794.
This PR enhances distributed fused adam by:
@timmoon10 @crcrpar Please help review, thanks.