Multiple Queues (v.0.16)

[tsutsarin]

How I'm supposed to create multiple queues with arq v0.16?

Looking at old docs I see that you could specify queue name, but now you can't.
What was the purpose of deleting queues from API?

I tried to start workers with different functions, but if worker doesn't see registered function it just deletes job.

So, if I want to use multiple queues on the same redis instance I have to use multiple redis databases (that have only numerical names)?

[samuelcolvin]

What was the purpose of deleting queues from API?

arq was completely redesigned to use sorted sets rather than lists to allow deferred calling and proper retry on failure/cancellation. Removing separate queues was just a bi-product of that.

The short answer is that arq v0.16 doesn't do multiple queues because I don't find them useful.

If you have a job that's higher priority than others, call it with a negative _defer_by and it'll run before other jobs enqueued at the same time, for example:

• if you enqueued a job with _defer_by=timedelta(minutes=-1) it would run before other jobs enqueued in the last minite
• if you enqueued a job with _defer_by=timedelta(days=-1) it would run before all jobs enqueued in the last year, in other words immediately that there's a slot available to run the job.

By doing this you can get most of the functionality of different priority jobs.

Does that help?

[tsutsarin]

My use case is to run different queues, which are tasks for unrelated workers, not to sort them by priority.
(But that info was useful anyway)

[tsutsarin]

Also, redis cluster doesn't support multiple databases, so separating queues using them won't work.

Redis Cluster does not support multiple databases like the stand alone version of Redis. There is just database 0 and the SELECT command is not allowed.
https://redis.io/topics/cluster-spec

[samuelcolvin]

It shouldn't be very hard to add custom queue names, so you'd use something like

await redis.enqueue_job('download_content', url, _queue_name='other')

and then setup the worker to work on that queue. Worker's would still only process jobs from one queue, but it sounds like that would work in your use case?

If that works, PR welcome.

[tsutsarin]

What will be your guidance to implement this feature?

[samuelcolvin]

I'm not sure what you mean?

If you'd like the feature, I'd happily accept a pull request where you implement it.

[Minstel]

@samuelcolvin

if you enqueued a job with _defer_by=timedelta(days=-1) it would run before all jobs enqueued in the last year, in other words immediately that there's a slot available to run the job.

Do you mean enqueued in the last day instead of year?