Django/celery/AWS Elasticache Redis only running tasks about half the time

[aas395]

I'm stumped. I'm running django with celery and I'm using Redis on AWS Elasticache and posting to an API endpoint to handle an action. The endpoint contains some data operations that don't involve celery and then at the end there's a call to .delay(). About half the time this works -- the task is run -- and the other half of the time the api endpoint returns successfully but the task is not run. Often with consecutive requests the first request will trigger the task successfully and the second one won't, although sometimes both will work. There doesn't seem to be a pattern to it.

The celery logs don't show any kind of error, in fact they don't even show that there has been any activity related to celery for the requests that don't trigger the task to actually run.

What could cause behavior like this? I'm fairly new to this and not really sure where to look if not the celery logs. Thank you for any help you can provide.

[aas395]

For anyone else who comes across something similar, the issue ended up being that I had two different versions of my application pointing to the same redis instance and sharing the same queue. There were two workers on different servers taking tasks from the queue and some events were being sent to a version of the application that was not set up to handle that particular task. I solved this by giving the newer version of the app a different queue name and having the celery worker representing the newer version of the app pull from the new queue.