reproduce UCC error at exit #28
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Summary: fix runtime error when reporting pair collective - fix typo when recording latency Differential Revision: D31291699 fbshipit-source-id: d51f1c7cc981efba06d374b0e3bc6b1237273cfa
Summary: [This is for debug purpose, **DO NOT MERGE**] Error happens when there are more than 2 PGs created and not destroying them explicitly/properly at exit sample reproducer: `mpirun -np 16 <any MPI variables> python comms.py --backend ucc --device cuda --collective all_to_allv --e 1M --collective-pair all_to_allv --collective-pair-size 8K --overlap-pair-pgs 1 --pair 1` Differential Revision: D31291835 fbshipit-source-id: 9f298d44d9e1e596f244d42e37ea31f3cdf6ef42
kingchc
requested review from
amirstar,
naghshineh and
srinivas212
as code owners
|
This pull request was exported from Phabricator. Differential Revision: D31291835 |
facebook-github-bot
added
the
CLA Signed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
[This is for debug purpose, DO NOT MERGE]
Error happens when there are more than 2 PGs created and not destroying them explicitly/properly at exit
sample reproducer:
mpirun -np 16 <any MPI variables> python comms.py --backend ucc --device cuda --collective all_to_allv --e 1M --collective-pair all_to_allv --collective-pair-size 8K --overlap-pair-pgs 1 --pair 1Differential Revision: D31291835