[Python] Any way to better distribute jobs?

[RolandasRazma]

I have multi node setup with concurrency=100 (just for example):

|----------|  |----------|
|  Worker  |  |  Worker  |
|----------|  |----------| ...

It works. When job generation rate is high, all nodes utilized.
Slight downside is that if job queue under 100 every job gets assigned to same worker. My preference would be to spread jobs among workers.

[manast]

Thanks for this question, currently it is not possible to improve this particular edge case even if it is quite common, but we are going to see if there is something we can do to mitigate it in newer versions.

[roggervalf]

hi @RolandasRazma, just in case. Which version are you using?

[RolandasRazma]

I'm using v3.12.8, but this is not python version related question :)

[roggervalf]

that's a pretty old version, but since v5.27.0 there was an improvement for our marker strategy

[RolandasRazma]

sorry, thought you asking about python version. Latest bullmq (python) version is 2.13.1