Make sure Celery is retrying jobs

[ryana]

It just dawned on me I did not check to see how Celery retries. I'll take a look this week unless someone beats me to the punch and verifies this works as expected (retries job up to a reasonable number of times before failing for good).

[hucares]

Another thing that might be cogent is alerting on stale jobs. Delayed job queues generally do a pretty good job of retrying things but not flushing out erroring jobs.

[micahalcorn]

Assuming that nothing is broken here 😬