Pause JobProcess when transport task falls through exponential backoff
#1903
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
sphuber
force-pushed
the
fix_1835_pause_job_process_excepted_transport_task
branch
from
39d6703
to
03c56fb
Compare
muhrin
approved these changes
Aug 23, 2018
All transport tasks for the `JobProcess` are wrapped in the exponential backoff retry coroutine utility, which when an exception occurs during the transport task, will reschedule the task with an exponential backoff. However, the backoff has a maximum number of retries, which when hit would bubble up the exception and causing the process to except. With the new pausing functionality in place, instead we can catch the `TransportTaskException` and pause the process. The user then has the chance to investigate the logs to determine the problem. If the problem was just of a temporary nature, the user can then resume the process. If instead the failure was of unrecoverable nature, the user can always decide to kill the process.
sphuber
force-pushed
the
fix_1835_pause_job_process_excepted_transport_task
branch
from
03c56fb
to
10ffdd6
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #1835
All transport tasks for the
JobProcessare wrapped in the exponentialbackoff retry coroutine utility, which when an exception occurs during
the transport task, will reschedule the task with an exponential backoff.
However, the backoff has a maximum number of retries, which when hit would
bubble up the exception and causing the process to except. With the new
pausing functionality in place, instead we can catch the
TransportTaskExceptionand pause the process. The user then has the chanceto investigate the logs to determine the problem. If the problem was just
of a temporary nature, the user can then resume the process. If instead
the failure was of unrecoverable nature, the user can always decide to kill
the process.