Working with Celery and asynchronous tasks

Environment Setup

Follow appropriate steps to ensure celery/AMQP broker is setup and configured. See Environment Setup notes.

The basics

See Celery docs for the basics of what it is and how to use it: Introduction to Celery.

Broker Configuration

If using RabbitMQ, you could rely on its default configuration. The following should already be present in your dev.py file:

BROKER_URL = 'amqp://guest:guest@localhost:5672//'

Alternatively, you're free to create a user, virtualhost, grant permissions for the new user, and use that instead of a "guest" account.

$ rabbitmqctl add_user myuser mypassword
$ rabbitmqctl add_vhost myvhost
$ rabbitmqctl set_permissions -p myvhost myuser ".*" ".*" ".*"

If you go that route, you need to overwrite the default BROKER_URL setting within local_settings.py file.

Broker Start/Stop/Status

When everything is installed, ensure RabbitMQ is running by checking its status:

$ sudo rabbitmqctl status

To start the server:

$ sudo rabbitmq-server

To start the server in background:

$ sudo rabbitmq-server -detached

Never kill the server process. To stop the server use rabbitmqctl:

$ sudo rabbitmqctl stop

Run Celery worker

To launch celery worker(s), ensure RabbitMQ server is running and use this basic command:

$ python manage.py celery worker --loglevel=info

Learn more about how to configure celery workers - autoscaling, logging, etc:

$ python manage.py celery worker --help

See all of celery command line options:

$ python manage.py celery help

Testing tasks

In order to execute specific tasks, ensure celery worker is running, and invoke them from the shell:

$ python manage.py shell
from apps.analytics import tasks as t

# schedule the task
# see results of its execution wherever worker logging is outputted (worker CL, file...)
t.redo_analytics.delay()

# do some work on the code
# restart celery worker so that it picks up new task code

# reload python module
reload(t)

# schedule task again:
t.redo_analytics.delay()

Amazon SQS caveats

Deploy to our Celery workers on Amazon using one of the four commands.

• fab deploy
• fab deploy:test,dev
• fab deploy:test,master
• fab deploy:production,master

In production, we're currently using Amazon SQS. In development, use something like RabbitMQ. This discrepancy might result in unwanted behaviour if one's is unaware of the differences. However, these differences are well documented.

• If a task is not acknowledged within the visibility_timeout, the task will be redelivered to another worker and executed.

  This causes problems with ETA/countdown/retry tasks where the time to execute exceeds the visibility timeout; in fact if that happens it will be executed again, and again in a loop.

  So you have to increase the visibility timeout to match the time of the longest ETA you are planning to use.

  Note that Celery will redeliver messages at worker shutdown, so having a long visibility timeout will only delay the redelivery of ‘lost’ tasks in the event of a power failure or forcefully terminated workers.

  Periodic tasks will not be affected by the visibility timeout, as it is a concept separate from ETA/countdown.

  The maximum visibility timeout supported by AWS as of this writing is 12 hours (43200 seconds):

  BROKER_TRANSPORT_OPTIONS = {'visibility_timeout': 43200}

• SQS does not yet support worker remote control commands.

• SQS does not yet support events, and so cannot be used with celery events, celerymon or the Django Admin monitor.

Taken from Celery docs on Amazon SQS.

Analytics

Sample output From Google API