Replace RQ with Celery as queuing backend

[jathanism]

Environment

• Python version: 3.6+
• Nautobot version: v1.1.0 maybe?

Proposed Functionality

• Add Celery as queue backend.
• Add Celery's built in Django integration.
• Add Celery worker
• Consider using Django database as a Celery results backend

Use Case

• Celery is more robust than RQ and fully implements AMQP for advanced message routing.
• Celery workers can instantiate multiple subprocesses at launch time and reuse them for multiple tasks. With RQ, you have to fork on every job. (Forking has been known to cause issues with leaking state of database connections).
  • Further, RQ can only run on systems that support fork
• Celery has much more advanced monitoring and telemetry out of the box
• Celery supports RabbitMQ for even more advanced enterprise support options such as message persistence, clustering
• Celery supports subtasks

Largely the justification revolves around evolving from just using task queueing for basic background tasks, at which RQ excels, to supporting advanced task queuing workflows, at which Celery excels.

Database Changes

None, unless Django as a Celery results backend is implemented.

External Dependencies

None.

Celery has full support for Redis as a broker so dependency upon Redis would not need to change.

[jedelman8]

Can we add actual use cases that are driving this? Otherwise, it seems subjective and just weighing technical merits of RQ and celery.

[jathanism]

Other important use cases:

• Built in support for storing task results as database objects w/ args/kwargs/inputs w/ django-celery-results
• Built in support for scheduling/recurring/crontab tasks w/ args/kwargs as database objects w/ django-celery-beat

[jedelman8]

Has been addressed from a few different PRs as shown above.