Possible prefetch issue affecting performances #183
What OS are you using?
Docker Python instance (
What version of Dramatiq are you using?
What did you do?
I launched 15 tasks with 15 workers
What did you expect would happen?
Each worker should execute exactly one task
Some worker executed multiple tasks, other workers executed none, slowing down the performances of the final results.
I noticed some performance issues in my app and managed to reproduce them with the following code:
Basically, I have an actor that will spawn multiple tasks, each task will take some time to complete then print something.
I'd expect the tasks to be processed concurrently, given enough workers, but apparently it isn't so. Some workers execute multiple tasks while others execute none, which makes me think that there might be some preload issue:
Some workers might take multiple tasks and execute them one by one.
I created an example app that I used to reproduce the issue:
I started 15 workers with the following script:
Some log files contain no executions:
while others contain multiple executions:
As you can see, the executions are happening one after the other after a full second (the time it takes for the first task to complete).
This is incredibly detrimental for the performances of my application, having all of the tasks executed at the same time would be great.
I'm using a custom broker that allows me to send multiple tasks with a redis pipeline, here's the code of the broker, but I don't think it has any effect on the preload issue:
I used the following configuration:
The text was updated successfully, but these errors were encountered:
This is expected since we prefetch 2 messages for every worker thread (see this code). As noted in the comment, starvation (what you're seeing here) is possible. If you want to work around this either change that line in your own distribution of Dramatiq or make a PR to make that value configurable via an environment variable (like this) and I'll merge it in.
I also added a system to allow the user to specify his own Worker Class, but I'll send you a different PR for that if you prefer.
I would much rather install a packaged version rather than my own fork honestly, if you give me a couple of guidelines I can add docs for these features too or adjust the PR according to your preferences, just let me know.
This'll get merged this week.
I'll take a look if you open another PR, but I don't think that's something I would be willing to merge into Dramatiq. Being configurable at that level is not one of the project's goals.