DM-34076: Improve timeout handling by MPGraphExecutor #174
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
timj
approved these changes
Mar 16, 2022
|
|
||
| @property | ||
| def state(self): | ||
| """Job processing state (JobState)""" | ||
| return self._state | ||
|
|
||
| @property | ||
| def terminated(self): | ||
| """Return True if job was killed by stop() method (`bool`)""" |
There was a problem hiding this comment.
but also if the exit code was negative?
There was a problem hiding this comment.
This is sort of implied, if stop was called but did not kill the process then it was not killed by stop. I'll expand the docstring.
| jobs.setJobState(job, JobState.FAILED) | ||
| if job.terminated: | ||
| # Was killed due to timeout. | ||
| if self.report.status == ExecutionStatus.SUCCESS: |
There was a problem hiding this comment.
If it is killed due to time out and reports success why can't that be treated as success? How can you be timed out and successful?
There was a problem hiding this comment.
It is treated as success, terminated is False for that case, and check for exitcode == 0 is in the if branch above. This branch is only for the case when it was actually killed by a signal (had non-zero exitcode)
Detect the case when a task manages to finish successfully before it gets killed due to timeout. This guarantees more consistent status in a quantum report.
Running time for timing-out task increased to 100 seconds, this should reduce the chance for it to finish successfully before timeout is detected by a parent.
andy-slac
force-pushed
the
tickets/DM-34076
branch
from
63d7cb8
to
19f7295
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Detect the case when a task manages to finish successfully before it
gets killed due to timeout. This guarantees more consistent status in
a quantum report.
Increase running time for timed out task in a unit test.
Running time for timing-out task increased to 100 seconds, this should
reduce the chance for it to finish successfully before timeout is
detected by a parent.
Checklist
doc/changes