[distributed] add test to ensure that dist autograd contexts are cleaned up in the case of nested rpcs #28485
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ned up in the case of nested rpcs This diff adds a test to ensure that when we have multiple nested RPCs inside a dist autograd context, the context that is created as a result of a nested rpc is cleaned up after the node creating the context exits the context manager. For example, worker 0 might send an rpc to worker 1 that results in an rpc to worker 2, so worker 2 will have 0's context, even though worker 0 never directly talked to 2. This test ensures that the context on 2 would also be cleaned up. Differential Revision: [D18079212](https://our.internmc.facebook.com/intern/diff/D18079212/) [ghstack-poisoned]
rohan-varma
added a commit
that referenced
this pull request
Oct 23, 2019
…ned up in the case of nested rpcs This diff adds a test to ensure that when we have multiple nested RPCs inside a dist autograd context, the context that is created as a result of a nested rpc is cleaned up after the node creating the context exits the context manager. For example, worker 0 might send an rpc to worker 1 that results in an rpc to worker 2, so worker 2 will have 0's context, even though worker 0 never directly talked to 2. This test ensures that the context on 2 would also be cleaned up. Differential Revision: [D18079212](https://our.internmc.facebook.com/intern/diff/D18079212/) ghstack-source-id: 92433816 Pull Request resolved: #28485
…ts are cleaned up in the case of nested rpcs" This diff adds a test to ensure that when we have multiple nested RPCs inside a dist autograd context, the context that is created as a result of a nested rpc is cleaned up after the node creating the context exits the context manager. For example, worker 0 might send an rpc to worker 1 that results in an rpc to worker 2, so worker 2 will have 0's context, even though worker 0 never directly talked to 2. This test ensures that the context on 2 would also be cleaned up. Differential Revision: [D18079212](https://our.internmc.facebook.com/intern/diff/D18079212/) [ghstack-poisoned]
…ts are cleaned up in the case of nested rpcs" the case of nested rpcs** This diff adds a test to ensure that when we have multiple nested RPCs inside a dist autograd context, the context that is created as a result of a nested rpc is cleaned up after the node creating the context exits the context manager. For example, worker 0 might send an rpc to worker 1 that results in an rpc to worker 2, so worker 2 will have 0's context, even though worker 0 never directly talked to 2. This test ensures that the context on 2 would also be cleaned up. Differential Revision: [D18079212](https://our.internmc.facebook.com/intern/diff/D18079212/) [ghstack-poisoned]
…ts are cleaned up in the case of nested rpcs" the case of nested rpcs** Closes #28124. This diff adds a test to ensure that when we have multiple nested RPCs inside a dist autograd context, the context that is created as a result of a nested rpc is cleaned up after the node creating the context exits the context manager. For example, worker 0 might send an rpc to worker 1 that results in an rpc to worker 2, so worker 2 will have 0's context, even though worker 0 never directly talked to 2. This test ensures that the context on 2 would also be cleaned up. Differential Revision: [D18079212](https://our.internmc.facebook.com/intern/diff/D18079212/) [ghstack-poisoned]
…ts are cleaned up in the case of nested rpcs" the case of nested rpcs** the case of nested rpcs** Closes #28124. This diff adds a test to ensure that when we have multiple nested RPCs inside a dist autograd context, the context that is created as a result of a nested rpc is cleaned up after the node creating the context exits the context manager. For example, worker 0 might send an rpc to worker 1 that results in an rpc to worker 2, so worker 2 will have 0's context, even though worker 0 never directly talked to 2. This test ensures that the context on 2 would also be cleaned up. Differential Revision: [D18079212](https://our.internmc.facebook.com/intern/diff/D18079212/) [ghstack-poisoned]
rohan-varma
added a commit
that referenced
this pull request
Oct 23, 2019
…ned up incase of nested rpcs Pull Request resolved: #28485 This diff adds a test to ensure that when we have multiple nested RPCs inside a dist autograd context, the context that is created as a result of a nested rpc is cleaned up after the node creating the context exits the context manager. For example, worker 0 might send an rpc to worker 1 that results in an rpc to worker 2, so worker 2 will have 0's context, even though worker 0 never directly talked to 2. This test ensures that the context on 2 would also be cleaned up. ghstack-source-id: 92440587 Differential Revision: [D18079212](https://our.internmc.facebook.com/intern/diff/D18079212/)
…ts are cleaned up in the case of nested rpcs" Closes #28124. This diff adds a test to ensure that when we have multiple nested RPCs inside a dist autograd context, the context that is created as a result of a nested rpc is cleaned up after the node creating the context exits the context manager. For example, worker 0 might send an rpc to worker 1 that results in an rpc to worker 2, so worker 2 will have 0's context, even though worker 0 never directly talked to 2. This test ensures that the context on 2 would also be cleaned up. Differential Revision: [D18079212](https://our.internmc.facebook.com/intern/diff/D18079212/) [ghstack-poisoned]
rohan-varma
added a commit
that referenced
this pull request
Oct 24, 2019
…ned up incase of nested rpcs Pull Request resolved: #28485 This diff adds a test to ensure that when we have multiple nested RPCs inside a dist autograd context, the context that is created as a result of a nested rpc is cleaned up after the node creating the context exits the context manager. For example, worker 0 might send an rpc to worker 1 that results in an rpc to worker 2, so worker 2 will have 0's context, even though worker 0 never directly talked to 2. This test ensures that the context on 2 would also be cleaned up. ghstack-source-id: 92527553 Differential Revision: [D18079212](https://our.internmc.facebook.com/intern/diff/D18079212/)
zhaojuanmao
approved these changes
Oct 24, 2019
…ts are cleaned up in the case of nested rpcs" the case of nested rpcs** Closes #28124. This diff adds a test to ensure that when we have multiple nested RPCs inside a dist autograd context, the context that is created as a result of a nested rpc is cleaned up after the node creating the context exits the context manager. For example, worker 0 might send an rpc to worker 1 that results in an rpc to worker 2, so worker 2 will have 0's context, even though worker 0 never directly talked to 2. This test ensures that the context on 2 would also be cleaned up. Differential Revision: [D18079212](https://our.internmc.facebook.com/intern/diff/D18079212/) [ghstack-poisoned]
…ts are cleaned up in the case of nested rpcs" the case of nested rpcs** the case of nested rpcs** Closes #28124. This diff adds a test to ensure that when we have multiple nested RPCs inside a dist autograd context, the context that is created as a result of a nested rpc is cleaned up after the node creating the context exits the context manager. For example, worker 0 might send an rpc to worker 1 that results in an rpc to worker 2, so worker 2 will have 0's context, even though worker 0 never directly talked to 2. This test ensures that the context on 2 would also be cleaned up. Differential Revision: [D18079212](https://our.internmc.facebook.com/intern/diff/D18079212/) [ghstack-poisoned]
rohan-varma
added a commit
that referenced
this pull request
Oct 24, 2019
…ned up incase of nested rpcs Pull Request resolved: #28485 This diff adds a test to ensure that when we have multiple nested RPCs inside a dist autograd context, the context that is created as a result of a nested rpc is cleaned up after the node creating the context exits the context manager. For example, worker 0 might send an rpc to worker 1 that results in an rpc to worker 2, so worker 2 will have 0's context, even though worker 0 never directly talked to 2. This test ensures that the context on 2 would also be cleaned up. ghstack-source-id: 92589157 Differential Revision: [D18079212](https://our.internmc.facebook.com/intern/diff/D18079212/)
|
@zhaojuanmao updated the PR to remove |
…ts are cleaned up in the case of nested rpcs" Closes #28124. This diff adds a test to ensure that when we have multiple nested RPCs inside a dist autograd context, the context that is created as a result of a nested rpc is cleaned up after the node creating the context exits the context manager. For example, worker 0 might send an rpc to worker 1 that results in an rpc to worker 2, so worker 2 will have 0's context, even though worker 0 never directly talked to 2. This test ensures that the context on 2 would also be cleaned up. Differential Revision: [D18079212](https://our.internmc.facebook.com/intern/diff/D18079212/) [ghstack-poisoned]
…ts are cleaned up in the case of nested rpcs" the case of nested rpcs** Closes #28124. This diff adds a test to ensure that when we have multiple nested RPCs inside a dist autograd context, the context that is created as a result of a nested rpc is cleaned up after the node creating the context exits the context manager. For example, worker 0 might send an rpc to worker 1 that results in an rpc to worker 2, so worker 2 will have 0's context, even though worker 0 never directly talked to 2. This test ensures that the context on 2 would also be cleaned up. Differential Revision: [D18079212](https://our.internmc.facebook.com/intern/diff/D18079212/) [ghstack-poisoned]
rohan-varma
added a commit
that referenced
this pull request
Oct 25, 2019
…ned up incase of nested rpcs Pull Request resolved: #28485 This diff adds a test to ensure that when we have multiple nested RPCs inside a dist autograd context, the context that is created as a result of a nested rpc is cleaned up after the node creating the context exits the context manager. For example, worker 0 might send an rpc to worker 1 that results in an rpc to worker 2, so worker 2 will have 0's context, even though worker 0 never directly talked to 2. This test ensures that the context on 2 would also be cleaned up. ghstack-source-id: 92611018 Differential Revision: [D18079212](https://our.internmc.facebook.com/intern/diff/D18079212/)
|
sounds good! |
|
This pull request has been merged in 4242385. |
thiagocrepaldi
pushed a commit
to thiagocrepaldi/pytorch
that referenced
this pull request
Feb 4, 2020
…of nested rpcs (pytorch#28485) Summary: Pull Request resolved: pytorch#28485 This diff adds a test to ensure that when we have multiple nested RPCs inside a dist autograd context, the context that is created as a result of a nested rpc is cleaned up after the node creating the context exits the context manager. For example, worker 0 might send an rpc to worker 1 that results in an rpc to worker 2, so worker 2 will have 0's context, even though worker 0 never directly talked to 2. This test ensures that the context on 2 would also be cleaned up. ghstack-source-id: 92611018 Test Plan: Ran the unit test. Differential Revision: D18079212 fbshipit-source-id: d49f0cda0bf2908747546e5c8a967256c848c685
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Stack from ghstack:
the case of nested rpcs
Closes #28124. This diff adds a test to ensure that when we have multiple nested RPCs
inside a dist autograd context, the context that is created as a result of a
nested rpc is cleaned up after the node creating the context exits the context
manager. For example, worker 0 might send an rpc to worker 1 that results in an
rpc to worker 2, so worker 2 will have 0's context, even though worker 0 never
directly talked to 2. This test ensures that the context on 2 would also be
cleaned up.
Differential Revision: D18079212