Error emails never sent

[cyounkins]

See mailing list thread: https://groups.google.com/forum/?fromgroups=#!topic/celery-users/Ayxd6ddvNs0%5B1-25%5D

With CELERY_SEND_TASK_ERROR_EMAILS = True, no error emails are ever sent. I have not gotten a single one to work since upgrading to 3.0.

I have a task that references an unknown variable (and raises NameError). It records the task and the error in the celery log, but it never sends an email. No SMTP connection is attempted, so it's not my credentials.

I'm not sure what else to do. Again, see the thread for more details.

Craig Younkins

[ask]

What exact version of Celery is this?

I just tried to enable it with an invalid server and got this error:

[2012-08-24 16:26:20,252: WARNING/MainProcess] /opt/devel/celery/celery/utils/mail.py:81: SendmailWarning: Mail could not be sent: timeout('timed out',) {'To': 'ask@celeryproject.org', 'Subject': u"[celery@halcyon.local] Error: Task tasks.raising (db4253ca-9788-42e5-80a9-d22873b75d27): KeyError('foo',)"}
[' File "/Users/asksol/.virtualenvs/def/bin/celery", line 8, in \n load_entry_point('celery==3.0.6', 'console_scripts', 'celery')()\n', ' File "/opt/devel/celery/celery/main.py", line 14, in main\n main()\n', ' File "/opt/devel/celery/celery/bin/celery.py", line 942, in main\n cmd.execute_from_commandline(argv)\n', ' File "/opt/devel/celery/celery/bin/celery.py", line 886, in execute_from_commandline\n super(CeleryCommand, self).execute_from_commandline(argv)))\n', ' File "/opt/devel/celery/celery/bin/base.py", line 175, in execute_from_commandline\n return self.handle_argv(prog_name, argv[1:])\n', ' File "/opt/devel/celery/celery/bin/celery.py", line 881, in handle_argv\n return self.execute(command, argv)\n', ' File "/opt/devel/celery/celery/bin/celery.py", line 856, in execute\n return cls(app=self.app).run_from_argv(self.prog_name, argv)\n', ' File "/opt/devel/celery/celery/bin/celery.py", line 142, in run_from_argv\n return self(_args, *_options)\n', ' File "/opt/devel/celery/celery/bin/celery.py", line 112, in call\n ret = self.run(_args, *_kwargs)\n', ' File "/opt/devel/celery/celery/bin/celery.py", line 214, in run\n return self.target.run(_args, _kwargs)\n', ' File "/opt/devel/celery/celery/bin/celeryd.py", line 153, in run\n return self.app.Worker(_kwargs).run()\n', ' File "/opt/devel/celery/celery/apps/worker.py", line 173, in run\n self.run_worker()\n', ' File "/opt/devel/celery/celery/apps/worker.py", line 258, in run_worker\n worker.start()\n', ' File "/opt/devel/celery/celery/worker/init.py", line 353, in start\n component.start()\n', ' File "/opt/devel/celery/celery/worker/consumer.py", line 369, in start\n self.consume_messages()\n', ' File "/opt/devel/celery/celery/worker/consumer.py", line 453, in consume_messages\n readers[fileno](fileno, event)\n', ' File "/Users/asksol/.virtualenvs/def/lib/python2.7/site-packages/billiard-2.7.3.12-py2.7-macosx-10.6-intel.egg/billiard/pool.py", line 672, in handle_event\n self._it.next()\n', ' File "/Users/asksol/.virtualenvs/def/lib/python2.7/site-packages/billiard-2.7.3.12-py2.7-macosx-10.6-intel.egg/billiard/pool.py", line 659, in _process_result\n on_state_change(task)\n', ' File "/Users/asksol/.virtualenvs/def/lib/python2.7/site-packages/billiard-2.7.3.12-py2.7-macosx-10.6-intel.egg/billiard/pool.py", line 639, in on_state_change\n state_handlersstate\n', ' File "/Users/asksol/.virtualenvs/def/lib/python2.7/site-packages/billiard-2.7.3.12-py2.7-macosx-10.6-intel.egg/billiard/pool.py", line 630, in on_ready\n item._set(i, obj)\n', ' File "/Users/asksol/.virtualenvs/def/lib/python2.7/site-packages/billiard-2.7.3.12-py2.7-macosx-10.6-intel.egg/billiard/pool.py", line 1402, in _set\n self._callback, self._value)\n', ' File "/Users/asksol/.virtualenvs/def/lib/python2.7/site-packages/billiard-2.7.3.12-py2.7-macosx-10.6-intel.egg/billiard/pool.py", line 144, in safe_apply_callback\n fun(_args)\n', ' File "/opt/devel/celery/celery/worker/job.py", line 310, in on_success\n return self.on_failure(ret_value)\n', ' File "/opt/devel/celery/celery/worker/job.py", line 362, in on_failure\n self._log_error(exc_info)\n', ' File "/opt/devel/celery/celery/worker/job.py", line 406, in _log_error\n self.task.send_error_email(context, einfo.exception)\n', ' File "/opt/devel/celery/celery/app/task.py", line 747, in send_error_email\n self.ErrorMail(self, **kwargs).send(context, exc)\n', ' File "/opt/devel/celery/celery/utils/mail.py", line 186, in send\n fail_silently=fail_silently)\n', ' File "/opt/devel/celery/celery/app/base.py", line 289, in mail_admins\n use_tls=self.conf.EMAIL_USE_TLS)\n', ' File "/opt/devel/celery/celery/loaders/base.py", line 205, in mail_admins\n mailer.send(message, fail_silently=fail_silently)\n', ' File "/opt/devel/celery/celery/utils/mail.py", line 81, in send\n traceback.format_stack())))\n']
traceback.format_stack())))

[ask]

Btw, I just tried it with a correct setup as well and it works well for me.

Here's my (censored) email settings:

CELERY_SEND_TASK_ERROR_EMAILS = True
SERVER_EMAIL = 'celery@example.org'
ADMINS = [
        ('Ask Solem', 'ask@example.org')
]
EMAIL_USE_TLS = True
EMAIL_PORT = 587
EMAIL_TIMEOUT = 10
EMAIL_HOST = 'mail.example.org'
EMAIL_HOST_USER = 'ask@example.org'
EMAIL_HOST_PASSWORD = 'x'

You can easily test your settings using the mail_admins method of the app instance:

>>> from celery import current_app
>>> current_app.mail_admins('Subject', 'body', fail_silently=False)

[cyounkins]

This is celery 3.0.5.

Manually calling mail_admins with the same configuration as normal does work. This is consistent with what I saw before - sending email is not even attempted when a task throws an exception.

I tried again with an exception-throwing task, but it did not work. Now I'm seeing this error, which seems to happen intermittently:

[2012-08-28 11:29:03,886: INFO/MainProcess] Got task from broker: test_exception[0719b190-630f-4123-ad74-23af74ff48e4]
[2012-08-28 11:29:03,960: ERROR/MainProcess] Task test_exception[0719b190-630f-4123-ad74-23af74ff48e4] raised exception: NameError("global name 'this_will_error' is not defined",)
Traceback (most recent call last):
File "/srv/autoref/virtual_env/lib/python2.6/site-packages/celery/task/trace.py", line 212, in trace_task
R = retval = fun(_args, *_kwargs)
File "/srv/autoref/dynamic/autoref/tasks/test_exception.py", line 10, in test_exception
this_will_error
NameError: global name 'this_will_error' is not defined
[2012-08-28 11:29:03,960: WARNING/MainProcess] Traceback (most recent call last):
[2012-08-28 11:29:03,960: WARNING/MainProcess] File "/srv/autoref/virtual_env/lib/python2.6/site-packages/gevent/greenlet.py", line 390, in run
[2012-08-28 11:29:03,961: WARNING/MainProcess] result = self._run(_self.args, *_self.kwargs)
[2012-08-28 11:29:03,961: WARNING/MainProcess] File "/srv/autoref/virtual_env/lib/python2.6/site-packages/celery/concurrency/base.py", line 27, in apply_target
[2012-08-28 11:29:03,971: WARNING/MainProcess] callback(target(_args, *_kwargs))
[2012-08-28 11:29:03,971: WARNING/MainProcess] File "/srv/autoref/virtual_env/lib/python2.6/site-packages/celery/worker/job.py", line 305, in on_success
[2012-08-28 11:29:03,972: WARNING/MainProcess] return self.on_failure(ret_value)
[2012-08-28 11:29:03,973: WARNING/MainProcess] File "/srv/autoref/virtual_env/lib/python2.6/site-packages/celery/worker/job.py", line 357, in on_failure
[2012-08-28 11:29:03,973: WARNING/MainProcess] self._log_error(exc_info)
[2012-08-28 11:29:03,973: WARNING/MainProcess] File "/srv/autoref/virtual_env/lib/python2.6/site-packages/celery/worker/job.py", line 400, in _log_error
[2012-08-28 11:29:03,973: WARNING/MainProcess] self.task.send_error_email(context, einfo.exception)
[2012-08-28 11:29:03,973: WARNING/MainProcess] File "/srv/autoref/virtual_env/lib/python2.6/site-packages/celery/app/task.py", line 747, in send_error_email
[2012-08-28 11:29:03,974: WARNING/MainProcess] self.ErrorMail(self, **kwargs).send(context, exc)
[2012-08-28 11:29:03,974: WARNING/MainProcess] File "/srv/autoref/virtual_env/lib/python2.6/site-packages/celery/utils/mail.py", line 168, in init
[2012-08-28 11:29:03,998: WARNING/MainProcess] self.error_whitelist = getattr(task, 'error_whitelist')
[2012-08-28 11:29:03,998: WARNING/MainProcess] AttributeError: 'test_exception' object has no attribute 'error_whitelist'
[2012-08-28 11:29:03,999: WARNING/MainProcess] <Greenlet at 0x3904370: apply_target(<function trace_task_ret at 0x2710aa0>, ('test_exception', '0719b190-630f-4123-ad74-23af74, {}, <bound method Request.on_success of <Request 0719b, <bound method Request.on_accepted of <Request 0719)> failed with AttributeError

Looks like this is the offending line: https://github.com/celery/celery/blob/v3.0.5/celery/utils/mail.py#L182 . It's still in 3.0.7. I'm confused as to why this works for some people. I'll do more investigation...

[ask]

Ah, so that's probably the problem.

The new task class does not define error_whitelist anymore, as it's deprecated, but the old compat task class does.

[cyounkins]

Hey! I finally got an error email! I changed the @task annotation to include error_whitelist

@celery.task(name="test_exception",error_whitelist=[])

I'm pretty sure that is deprecated so this should not be necessary.

[yeago]

I'm still having this problem with celery 3.0.11

[ask]

@yeago Have you tried upgrading to 3.0.21?

[yeago]

http://dpaste.com/1334469/

Any idea? That's after upgrading.

...ignore that, I hadn't appropriately updated kombu.

[yeago]

Hey there,

Yeah, I upgraded just fine and still no emails!

[ask]

Do you get error email when you set an error_whitelist (which is what this issue is about), if not then this is a new issue.

Are you able to send mails using the setting you have configured? Try:

 $ celery -A myproj shell
 >>> celery.mail_admins('Test subject', 'Test body', fail_silently=False)

[yeago]

from celery import Celery
app = Celery()
app.mail_admins("test", "test", fail_silently=False)

Nothing happens. No email, no error. I am probably initializing it incorrectly. When I tried your method I got

(mmdb)mmdb@vii-mmdb-app1:~/code/mmdb$ celery -A myproj
Traceback (most recent call last):
File "/home/mmdb/.virtualenvs/mmdb/bin/celery", line 5, in
from pkg_resources import load_entry_point
File "/home/mmdb/.virtualenvs/mmdb/local/lib/python2.7/site-packages/pkg_resources.py", line 3011, in
parse_requirements(requires), Environment()
File "/home/mmdb/.virtualenvs/mmdb/local/lib/python2.7/site-packages/pkg_resources.py", line 626, in resolve
raise DistributionNotFound(req)
pkg_resources.DistributionNotFound: kombu>=2.5.12,<3.0

[chrisspen]

Note, I had this problem up to version 3.1.1. I began receiving error emails again after I upgraded to 3.1.7.