Docs & Features

[ccrvlh]

Hi, we've been using Celery for a while on our projects, but it's been a pain with all the complexity and confusing/conflicting documentation (not to say the lack of).

There are a few features we are looking for on a new job queue lib, I was wondering what are your thoughts on it, and plans going forward.

• Progress / Meta Data: we heavily use meta data on Celery tasks (self.update_state(state='PENDING', meta={'current': i, 'total': NTOTAL}) on a task with bind=True. I couldn't find anything on more granular/custom state handler on any of the libs we've found (Arq, RQ, Saq, Dramatiq), wondering what are the plans/thoughts on that. I would imagine that I could do something similar using the context, maybe?
• Dynamic Schedule: we have used Redbeat to make Celery's Beat module dynamic, so we can be able to turn on/off a schedule programatically (that wasn't possible with Celery's Beat only). Given the examples I would think this should be possible, but couldn't find a way to do it.
• Retry Logic: The way we are currently using retry logic is a bit weird, raise self.retry(exc=Exception(str(error)), countdown=countdown, max_retries=5) but it's been working well... wondering how would I go by doing retries using Saq.
• Gevent: Out app runs with about 250 MB of memory, so leveraging N workers can be taxing on RAM (tasks are complex - many functions - and I/O heavy, taking up to 3-4 minutes). So in this scenario Celery gevent worker with concurrency has been great, not many workers, a ton of parallel tasks. I think Dramatiq has implemented greenlets, but RQ hasn't (there is code somewhere in the wild that implements a custom Gevent Worker if I'm not mistaken)
• Different Queues / Priority: while we don't use the priority concept directly, we do leverage different queues (and workers that execute on specific queues only), wondering how could we implement this in Saq.

Also, just to add to the discussion and share my experiences:

• RQ: For now we decided not to go that route because of the lack of concurrency, the API seems pretty nice though.
• Celery: a huge pain to setup, manage and debug. Bad documentation, confusing, and even worse error messages (think of Celery and Flask contexts)
• Dramatiq: seems nice, I tend to like RQ API's better, but the scheduler has to be a third party library. Still want to invest more time understanding the library.
• Arq: seems great also, but it does look like is less mature than the libraries above (documentation, material, help etc), also couldn't explicitly find the things I was looking for.

I know this is quite a list in terms of features, I was just wondering what are your thoughts and if you reckon it could make sense for the future. I'm probably not the most knowledgable person, but I would be more than happy to help!

Thanks!

[tobymao]

@lowercase00 saq already has progress metadata, you can just do job.update(progress=0.2) in terms of intermediate state, you can already do that with a custom result object, you can store anything you want in results, but happy to discuss other options

i'm not sure what you mean by dynamic schedule, right now you can schedule a job for later or do a cron

i don't understand what you're asking for retry logic?

we use asyncio, so if your code is async, you can run many concurrent workers with limited overhead

yes, this is easily done in saq, simply having a new settings hash with a different queue name.

[tobymao]

I'm happy to continue answering more questions here, but I'm going to close the issue for now.