Possible to catch celery WorkerLostError?

[bdon]

I am following the guide at http://raven.readthedocs.org/en/latest/integrations/celery.html, and can report exceptions within a Celery task fine, but it doesn't seem to report to Sentry when a worker exits prematurely, such as running out of memory. Is this expected?

[TallGirlVanessa]

I ran into this myself today. Here's what I found about the problem:

Raven integrates with Celery by adding a root logger handler as well as a signal listener for failures inside the workers. That results in the following cases:

• Exception in task
  • Worker: task_failed signal handler grabs stack trace and frame args and sends it to Sentry.
  • MainProcess: SentryHandler on the root logger receives log message from the failure. A filter (CeleryFilter) ignores it: it's designed to only allow "internal" messages through. That's good because otherwise we'd get two Sentry messages every time a task raises. Also, the message from the MainProcess isn't as good because it doesn't (can't?) put args in its stack trace.
  • MainProcess: Sends an email via Celery's error email sending feature.
• Internal error (not sure exactly what this would look like?)
  • MainProcess: SentryHandler receives this log message, and the filter allows it since it's got internal set. It then goes to Sentry.
• Something like logger.error in task
  • Worker: It's got a SentryHandler too, which handles this. Doesn't get to the main process since the worker doesn't raise.
• Worker dies (or some other problem outside the task proper)
  • MainProcess: SentryHandler receives this log message...and ignores it, due to the CeleryFilter. I'm not really sure what makes some log records internal and others not, but apparently WorkerLostError is not.
  • MainProcess: Sends an email via Celery's error email sending feature.

So: we have a choice between getting duplicate emails for our exceptions inside tasks, or not getting emails about e.g. workers dying. Not really sure what the right way forward is.

[dcramer]

@garaden thanks for putting this together, it's extremely helpful. I haven't had time to dig into it yet (I'm traveling this week), but I'm sure we can come up with something. That said, our last attempt to fix Celery reporting caused a bunch of additional issues.

[ThiefMaster]

With Celery 4 (where email sending has been removed) being able to log WorkerLostError in Sentry would be extremely useful.

[davidemerritt]

+1

[davidemerritt]

This isn't an ideal solution, but after messing around for a bit, I just monkey patched the celery.util.mail.ErrorMail.send method to log to sentry, not pretty but it works. But yes this will stop working once we upgrade to celery 4.