-
Notifications
You must be signed in to change notification settings - Fork 21.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix use after free in tensorpipe agent #87627
Conversation
[ghstack-poisoned]
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/87627
Note: Links to docs will display an error until the docs builds have been completed. ✅ No Failures, 1 PendingAs of commit 16ec408: This comment was automatically generated by Dr. CI and updates every 15 minutes. |
ghstack-source-id: ace0cb559dcc62fc68e0fffe0db0152c0d6633df Pull Request resolved: #87627
cc @Flamefire |
@pytorchbot merge |
Merge startedYour change will be merged once all checks pass (ETA 0-4 Hours). Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
if (reverseDeviceMaps.find(it->first) == reverseDeviceMaps.end()) { | ||
it = reverseDeviceMaps_.erase(it); | ||
} else { | ||
it++; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Note that it++
creates a temporary copy while ++it
does not
Fixes pytorch#87359, which identifies use after free for reverse device maps. This is only in the dynamic RPC feature and not effecting stable RPC code path. Unfortunately the test `TensorPipeRpcTest.test_dynamic_rpc_existing_rank_can_communicate_with_new_rank_cuda` that is failing is also running into separate issue. I've temporarily disabled some of the test code to investigate the error in asychronously. Testing plan: - tested all the dynamic RPC tests Pull Request resolved: pytorch#87627 Approved by: https://github.com/rohan-varma
Fixes pytorch#87359, which identifies use after free for reverse device maps. This is only in the dynamic RPC feature and not effecting stable RPC code path. Unfortunately the test `TensorPipeRpcTest.test_dynamic_rpc_existing_rank_can_communicate_with_new_rank_cuda` that is failing is also running into separate issue. I've temporarily disabled some of the test code to investigate the error in asychronously. Testing plan: - tested all the dynamic RPC tests Pull Request resolved: pytorch#87627 Approved by: https://github.com/rohan-varma
Fixes pytorch#87359, which identifies use after free for reverse device maps. This is only in the dynamic RPC feature and not effecting stable RPC code path. Unfortunately the test `TensorPipeRpcTest.test_dynamic_rpc_existing_rank_can_communicate_with_new_rank_cuda` that is failing is also running into separate issue. I've temporarily disabled some of the test code to investigate the error in asychronously. Testing plan: - tested all the dynamic RPC tests Pull Request resolved: pytorch#87627 Approved by: https://github.com/rohan-varma
Stack from ghstack:
Fixes #87359, which identifies use after free for reverse device maps. This is only in the dynamic RPC feature and not effecting stable RPC code path.
Unfortunately the test
TensorPipeRpcTest.test_dynamic_rpc_existing_rank_can_communicate_with_new_rank_cuda
that is failing is also running into separate issue. I've temporarily disabled some of the test code to investigate the error in asychronously.Testing plan: