-
Notifications
You must be signed in to change notification settings - Fork 4
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Signal job failures when recovering dead jobs
The `enqueue_jobs_from_dead_broker.lua` script doesn't re-enqueue jobs if the max_retries was exceeded. This is very surprising behaviour for most people, and because the failure handler is not called it means jobs can be left in an inconsistent state. This change will make sure that the failure handler is called and the job moved to the FAILED state. Drive-by: Add functional tests for concurrency that I forgot to `git add` previously. Drive-by: Add tags file to .gitignore Drive-by: add flake8 to tox venv dependencies so that vscode works better in that venv Fixes: #14
- Loading branch information
Showing
10 changed files
with
123 additions
and
16 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -9,3 +9,5 @@ doc/_build | |
|
||
.coverage | ||
.pytest_cache | ||
|
||
tags |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Empty file.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,31 @@ | ||
import pytest | ||
import time | ||
|
||
from spinach.brokers.memory import MemoryBroker | ||
from spinach.brokers.redis import RedisBroker | ||
from spinach.engine import Engine | ||
|
||
|
||
@pytest.fixture(params=[MemoryBroker, RedisBroker]) | ||
def spin(request): | ||
broker = request.param | ||
spin = Engine(broker(), namespace='tests') | ||
yield spin | ||
|
||
|
||
def test_concurrency_limit(spin): | ||
count = 0 | ||
|
||
@spin.task(name='do_something', max_retries=10, max_concurrency=1) | ||
def do_something(index): | ||
nonlocal count | ||
assert index == count | ||
count += 1 | ||
|
||
for i in range(0, 5): | ||
spin.schedule(do_something, i) | ||
|
||
# Start two workers; test that only one job runs at once as per the | ||
# Task definition. | ||
spin.start_workers(number=2, block=True, stop_when_queue_empty=True) | ||
assert count == 5 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -14,6 +14,7 @@ deps = | |
pytest-cov | ||
pytest-threadleak | ||
pycodestyle | ||
flake8 | ||
flask | ||
django | ||
|
||
|