EagerBroker

[xelhark]

Hi!

Have you ever considered adding an EagerBroker?

My problem is that during test case execution, the join method will hang indefinitely if there is any exception in my code (and since I'm testing, most of the time there actually is one 😄 )

Also, the exceptions aren't bubbled up so I don't see them during the execution of the tests.

I wrote this very simple EagerBroker, but I'm wondering if you think this is good enough, or if there's any better way to handle this kind of situation.

Just in case, I'll leave this here as a reference if anyone googles this:

class EagerBroker(StubBroker):
    def __init__(self):
        super(EagerBroker, self).__init__()

    def process_message(self, message):
        actor = self.get_actor(message.actor_name)
        actor(*message.args, **message.kwargs)

    def enqueue(self, message, *, delay=None):
        self.process_message(message)

[Bogdanp]

It might be worth adding this to the documentation at the very least. I understand the simplicity of having an eager broker but personally think it's a bit of an anti-pattern for integration tests because it changes the operating model of the (presumably controller) code that you're testing.

What I tend to do is join with a timeout on the queue and then investigate if that fails when doing integration tests and unit test my tasks directly to catch things like exceptions. This better preserves the operational model of the code being tested.

[xelhark]

I perfectly agree, maybe the best way would be to have a special Worker that bubbles exceptions so that they are raised when calling worker.join('my_queue'), but then again, this would be a special worker class and it might behave differently in production.

[ryanhiebert]

I'm anticipating writing a custom worker for testing a project I'm working on, that would have controls to run the tasks that are currently on the queue and stop after the previously known tasks in the queue are complete, even if more tasks have been enqueued in the process. I'm not sure how relevant that thought is to what you're looking for here, but perhaps there's some overlap.

[Bogdanp]

@xelhark what do you think about https://github.com/Bogdanp/dramatiq/pull/203/files ? This seems like a decent compromise between the two options. The only thing to watch out for is the fact that exceptions are only raised once a task has exhausted its retries.

[dnmellen]

If anyone likes the "eager" approach for testing I improved a little @xelhark's EagerBroker and I added support for pipelines and groups and GroupCallbacks: https://gist.github.com/dnmellen/17c89389339fdf72e240b2aeb5af4c5a