Having issues automating mail send through celery - KeyError: 'lockfile'

[ChrisWhiten]

Hi there,

I'm trying to periodically send out queued up mail with celery. The way I'm going about this is by creating a task that I will periodically call, which basically just calls into the function that triggers mail send. This function looks like this:

from post_office import mail
from post_office.management.commands import send_queued_mail

@task
def send_emails():
    cmd = send_queued_mail.Command()
    cmd.execute(processes=1)

When this task gets called, I see the following stack trace within RabbitMQ:

[2014-10-12 22:15:43,877: ERROR/MainProcess] Task booker.tasks.send_emails[26d93b2e-20ed-448f-ae19-29615a2a7906] raised unexpected: KeyError('lockfile',)
Traceback (most recent call last):
  File "/Users/chris/Envs/myappvenv/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/Users/chris/Envs/myappvenv/lib/python2.7/site-packages/celery/app/trace.py", line 437, in __protected_call__
    return self.run(*args, **kwargs)
  File "/Users/chris/dev/bitbucket/myapp/myapp/booker/tasks.py", line 11, in send_emails
    cmd.execute(processes=num_processes)
  File "/Users/chris/Envs/myappvenv/lib/python2.7/site-packages/django/core/management/base.py", line 285, in execute
    output = self.handle(*args, **options)
  File "/Users/chris/Envs/myappvenv/lib/python2.7/site-packages/post_office/management/commands/send_queued_mail.py", line 29, in handle
    options['lockfile'])
KeyError: 'lockfile'
[2014-10-12 22:15:43,887: ERROR/MainProcess] Task booker.tasks.send_emails[b61c6620-cd4c-472b-9ebb-e8cf7ef782e7] raised unexpected: KeyError('lockfile',)
Traceback (most recent call last):
  File "/Users/chris/Envs/myappvenv/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/Users/chris/Envs/myappvenv/lib/python2.7/site-packages/celery/app/trace.py", line 437, in __protected_call__
    return self.run(*args, **kwargs)
  File "/Users/chris/dev/bitbucket/myapp/myapp/booker/tasks.py", line 11, in send_emails
    cmd.execute(processes=num_processes)
  File "/Users/chris/Envs/myappvenv/lib/python2.7/site-packages/django/core/management/base.py", line 285, in execute
    output = self.handle(*args, **options)
  File "/Users/chris/Envs/myappvenv/lib/python2.7/site-packages/post_office/management/commands/send_queued_mail.py", line 29, in handle
    options['lockfile'])
KeyError: 'lockfile'
[2014-10-12 22:15:43,888: ERROR/MainProcess] Task booker.tasks.send_emails[63c97beb-d56c-44ec-ac0a-07d1b0b77a69] raised unexpected: KeyError('lockfile',)
Traceback (most recent call last):
  File "/Users/chris/Envs/myappvenv/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/Users/chris/Envs/myappvenv/lib/python2.7/site-packages/celery/app/trace.py", line 437, in __protected_call__
    return self.run(*args, **kwargs)
  File "/Users/chris/dev/bitbucket/myapp/myapp/booker/tasks.py", line 11, in send_emails
    cmd.execute(processes=num_processes)
  File "/Users/chris/Envs/myappvenv/lib/python2.7/site-packages/django/core/management/base.py", line 285, in execute
    output = self.handle(*args, **options)
  File "/Users/chris/Envs/myappvenv/lib/python2.7/site-packages/post_office/management/commands/send_queued_mail.py", line 29, in handle
    options['lockfile'])
KeyError: 'lockfile'

I'm not seeing any clear documentation on the expected usage of this with Celery, so I may be misusing it entirely... Is this a misuse of the library entirely, or is there a deeper issue at play? How do I trigger periodically emptying the queue and sending out pending emails?

[selwin]

Hi there, you're right, this is not mentioned anywhere in the docs. There are two primary use cases for sending emails:

1. Transactional - emails like sign up verification email, confirmation. These emails should be sent immediately to customers and you should use a background worker like celery to run commands like this:

mail.send(
    ['recipient1@example.com'],
    'from@example.com',
    template='welcome_email',
    context={'foo': 'bar'},
    priority='now', # Send immediately
)

1. Sending bulk emails like newsletter etc. To send these, simply run send_queued_mail management command (this shouldn't be run through task queues because there can only be one process active at any given time (hence the need for a lock file). If you have lots of email, you can speed up the process by passing in the --processes argument to send_queued_mail management command and it will spawn multiple processes to send more emails faster.

Hope this answers your question. I'm closing this issue for now. Feel free to reopen if there's anything that's unclear.