Add process set support for MXNet #3043
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
21 tasks
Signed-off-by: Max H. Gerlach <git@maxgerlach.de>
Signed-off-by: Max H. Gerlach <git@maxgerlach.de>
Signed-off-by: Max H. Gerlach <git@maxgerlach.de>
Signed-off-by: Max H. Gerlach <git@maxgerlach.de>
maxhgerlach
force-pushed
the
subset-mxnet-pr
branch
from
2f3f79d
to
9823377
Compare
tgaddair
approved these changes
Jul 23, 2021
Unit Test Results 784 files ±0 784 suites ±0 6h 37m 3s ⏱️ ±0s For more details on these failures, see this check. Results for commit 0a42194. ± Comparison against base commit 0a42194. ♻️ This comment has been updated with latest results. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Checklist before submitting
Description
This is a straightforward extension of #2839, adding the process set feature to the MXNet API.
Included are the ops
allgather,allreduce,alltoall,broadcast, andgrouped_allreduce, as well as theDistributedOptimizer.There's also the
DistributedTrainerthat I didn't look into. I don't have much experience working with MXNet, but it seems that a Gluon Trainer would be a higher level concept meant to operate on an entire neural net. In that case adding an overallprocess_setargument might not be the right call as users will typically want to non-globally aggregate the gradients for just a subset of their parameters, not for the entire model.