fix(run_command): fix subprocess timeout recovery #179
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Normally, when a transfer times out, it's because it's stuck deep down in I/O wait, meaning SIGKILL is not going to be handled in a timely manner, and, so, trying to `.communicate()` with it again is just futile. With the current code, after a timeout, the worker just gets stuck in the same wait it tried to get out of with the kill, as if there were no timeout given. So, let's just send it a kill and then abandon it. Related to #175 (abandonning subprocesses may lead to zombies), but this doesn't really address the items brought up in that issue, which are higher- level ideas.
rikvl
approved these changes
May 15, 2024
|
It was the semicolons! (I think I've been spending too much time in perl recently). |
ljgray
approved these changes
May 15, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Normally, when a transfer times out, it's because it's stuck deep down in I/O wait, meaning SIGKILL
is not going to be handled in a timely manner, and, so, trying to
.communicate()with it again is just futile.With the current code, after a timeout, the worker just gets stuck in the same wait it tried to get out of with the kill, as if there were no timeout given.
So, let's just send it a kill and then abandon it.
Related to #175 (abandonning subprocesses may lead to zombies), but this doesn't really address the items brought up in that issue, which are higher-level ideas.
I have deployed this to cedar.