Add internal queue monitoring

[dcramer]

Specifically we want to capture the size of each queue.

We'll need a gauge type in tsdb to handle this, as well as some service running (either as its own process, or part of an existing) which captures the stats routinely. This service may also be able to replace Celerybeat.

• Display in UI when backlogged Detect if the worker issue is actually because of backlog or not #2867
• CLI tooling to fetch queue sizes Add command line util to list and purge queues #2844
• Update docs Add documentation around queue monitoring #2944
• Internally track and graph size over time

[mattrobenolt]

@dcramer Care to elaborate on how something like this could replace celery beat? I was hoping the first pass would be to give it a dedicated queue and just celery beat it every minute. Then we track the results. I think we can expand on this later after this is working. I'm unsure of the immediate value to try and replace that entirely unless I'm missing some other better reason.

[dcramer]

We can't rely on the queue to monitor the queue. A distinct queue doesn't guarantee workers won't block for an extended period of time before the tasks are run. Most of our celerybeat usage is fixed window jobs which fire other jobs which is why I said it could potentially replace that.

[mattrobenolt]

We can't rely on the queue to monitor the queue.

Oh, right. :)

[mattrobenolt]

@dcramer So Celery beat has this thing CELERYBEAT_SCHEDULE_FILENAME which is a file that persists the last runs of tasks. This way when you say "run every 5 minutes", it'll continue to do every 5 minutes through restarts.

I'd rather ditch this and work on clock schedules. I'd rather define jobs crontab style, and we always just leverage the clock time instead of needing to persist to disk.

Do you have an opinion about this?