[dynamo] avoid popen.communicate() #87335
Conversation
It seems like when popen.communicate() is used it waits for all the desendents of popen to close the stdin/stderr. However, if we have have worker processes running in the child, and the child segfaults, those processes will stay alive until someone waitpid's the child. Since those children have open handles to the stdin/stderr pipe, communicate never returns. This change just writes the output to temp files and directly calls wait() on the child, which returns as soon as it dies. [ghstack-poisoned]
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/87335
Note: Links to docs will display an error until the docs builds have been completed. ❌ 1 Failures, 7 PendingAs of commit 89fda0d: The following jobs have failed:
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
It seems like when popen.communicate() is used it waits for all the desendents of popen to close the stdin/stderr. However, if we have have worker processes running in the child, and the child segfaults, those processes will stay alive until someone waitpid's the child. Since those children have open handles to the stdin/stderr pipe, communicate never returns. This change just writes the output to temp files and directly calls wait() on the child, which returns as soon as it dies. ghstack-source-id: 9877330 Pull Request resolved: #87335
|
@pytorchbot merge |
Merge startedYour change will be merged once all checks pass (ETA 0-4 Hours). Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
Merge failedReason: 2 additional jobs have failed, first few of them are: trunk ,trunk / linux-bionic-cuda11.7-py3.10-gcc7 / test (slow, 1, 2, linux.4xlarge.nvidia.gpu) Details for Dev Infra teamRaised by workflow job |
|
@pytorchbot merge -f "unrelated failure in nvfuser" |
Merge startedYour change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
|
Hey @zdevito. |
4 participants
Stack from ghstack (oldest at bottom):
It seems like when popen.communicate() is used it waits for all the
desendents of popen to close the stdin/stderr. However, if we have
have worker processes running in the child, and the child segfaults,
those processes will stay alive until someone waitpid's the child.
Since those children have open handles to the stdin/stderr pipe,
communicate never returns.
This change just writes the output to temp files and directly calls
wait() on the child, which returns as soon as it dies.
cc @jansel @lezcano @fdrocha