Identify candidate task queue and message broker options for task management in Arches

[chiatt]

Celery as a task queue with RabbitMQ as the message broker seem to be likely candidates, but would it make sense to use alternative technologies (e.g. Redis instead of Rabbit)? We should produce a list of possible technologies for discussion.

[chiatt]

Worth looking into: http://policystat.github.io/jobtastic/
Using the ORM as the backend: http://docs.celeryproject.org/en/latest/django/first-steps-with-django.html#django-celery-results-using-the-django-orm-cache-as-a-result-backend

[apeters]

https://denibertovic.com/posts/celery-best-practices/
https://wiredcraft.com/blog/3-gotchas-for-celery

I think at this point, Celery seems like our best bet. It's very well documented and sets the standard as far as Python integration goes.

As far a what broker to use (Rabbit MQ, redis, etc...) Arches should allow an admin to make that choice. In fact Arches should, as much as possible, be agnostic of the task manager it uses to manage long running/memory intensive tasks.

Redis vs. RabbitMQ: https://www.linkedin.com/pulse/redis-vs-rabbitmq-message-broker-vishnu-kiran-k-v

[chiatt]

So far there haven't been any objections to this strategy so I'm closing this ticket.