Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Handling task failures+retries during testing #57
As someone new to dramatiq, the retry middleware may cause task failures during testing to appear as if the test worker is hanging indefinitely or is otherwise failing to execute tasks. What actually happened:
Given the lack of output and apparent hang, my initial reaction was confusion and to assume that I had somehow misconfigured the test worker. The reality was that everything was working correctly, and pytest was just eating the output.
The unit testing guide should probably mention that apparent hangs may be the result of the retry middleware, and that they should try temporarily setting
I've run into this problem myself and been confused by it so it's definitely something that ought to get fixed. I think adding a timeout to
Excuseme againg @Bogdanp :)
I'm using PyTest (and I followed your example with a broker and a worker as fixtures) in a Django project and I not understand how to set the timeout properly (if it's the fix for this issue).
Setting an timeout in the
Following that @rpkilby says, I tried to disable
Here is my code:
import dramatiq import pytest @pytest.fixture def broker(): broker = dramatiq.get_broker() broker.flush_all() return broker @pytest.fixture def worker(broker): worker = dramatiq.Worker(broker, worker_timeout=100) worker.start() yield worker worker.stop()
@pytest.mark.django_db def test_optout(broker, worker): # A bunch of code (?) create_my_model_instance_task.send("Pepe") broker.join("default") worker.join() assert MyModel.objects.get(name="Pepe")
Based on the error, I assume you're using sqlite, which doesn't support concurrent writes (see: "High Concurrency"). What's happening is that your test case opens a database transaction, creating a lock and preventing other connections from writing. While the test case holds the transaction open, the task is sent and processed in the worker thread, attempting to issue a write with a different connection. Since the database is already locked by a separate connection, sqlite complains with an
@Bogdanp - please correct me if I got anything wrong here. My knowledge of this area is not that deep.
The real exception that my
Maybe I must test the task isolated (without
I hope @Bogdanp has a good idea :) (Sorry if I sound rude or pedantic, English is't my native language).
Apologies @nachopro, I could swear I made a PR to the project you linked a couple of weeks ago fixing the problem, but I don't see it anymore so I must've done something wrong.
If you change this line to
(what @rpkilby was suggesting), everything will work fine.