Skip to content

Commit

Permalink
Make the Redis broker clean up aborted jobs
Browse files Browse the repository at this point in the history
The Redis broker uses a non-zero value for `max_retries` to determine
if it should remove a job from the "running" state.  So when an
`AbortException` is thrown, max out the `job.retries` counter,
rather than zeroing out the `job.max_retries` counter.  This will
have the same effect, but play more nicely with what the Redis
broker is expecting.

The MemoryBroker is unaffected by this.

Fixes Issue #20
  • Loading branch information
nisimond committed Apr 30, 2022
1 parent 35d3928 commit ace8026
Show file tree
Hide file tree
Showing 2 changed files with 5 additions and 2 deletions.
2 changes: 1 addition & 1 deletion spinach/job.py
Original file line number Diff line number Diff line change
Expand Up @@ -189,7 +189,7 @@ def advance_job_status(namespace: str, job: Job, duration: float,
return

if isinstance(err, AbortException):
job.max_retries = 0
job.retries = job.max_retries
logger.error(
'Fatal error during execution of %s after %s, canceling retries',
job, duration, exc_info=err
Expand Down
5 changes: 4 additions & 1 deletion tests/test_job.py
Original file line number Diff line number Diff line change
Expand Up @@ -118,10 +118,13 @@ def test_advance_job_status(job):
RetryException('Must retry', at=now))
assert job.status is JobStatus.FAILED

# The job should have retried twice due to previous tests, ensure that
# an AbortException tops off the retry counter
job.status = JobStatus.RUNNING
job.max_retries = 10
assert job.retries == 2
advance_job_status('namespace', job, 1.0, AbortException('kaboom'))
assert job.max_retries == 0
assert job.retries == 10
assert job.status is JobStatus.FAILED


Expand Down

0 comments on commit ace8026

Please sign in to comment.