Skip to content
This repository

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

MOVED TO: celery/celery

This branch is 0 commits ahead and 0 commits behind master

Octocat-spinner-32 celery Forgot to add test_hub module June 06, 2012
Octocat-spinner-32 contrib Remove BROKER_HOST/USER/PASSWORD/PORT/VHOST from docs, and use URLs c… June 02, 2012
Octocat-spinner-32 docs Don't show body, just show moved message June 06, 2012
Octocat-spinner-32 examples Tests passing May 11, 2012
Octocat-spinner-32 funtests Remove BROKER_HOST/USER/PASSWORD/PORT/VHOST from docs, and use URLs c… June 02, 2012
Octocat-spinner-32 requirements Use MP_MAIN_MODULE when execv is used to properly rename __main__ tas… June 03, 2012
Octocat-spinner-32 .gitignore Ignoring eclipse / pydev settings files. August 04, 2011
Octocat-spinner-32 .travis.yml Tests Py2.5 on Travis May 31, 2012
Octocat-spinner-32 AUTHORS Adds Mark Thurman to AUTHORS June 01, 2012
Octocat-spinner-32 Changelog 2.6.0 Release Candidate 1 April 27, 2012
Octocat-spinner-32 FAQ correct tiny typo in FAQ around Windows celeryd invocation April 16, 2012
Octocat-spinner-32 INSTALL Documentation improvements October 28, 2010
Octocat-spinner-32 LICENSE Updates copyright years to include 2012 January 06, 2012
Octocat-spinner-32 MANIFEST.in Installation broke on Windows, because of this line in MANIFEST.in August 05, 2011
Octocat-spinner-32 README Bump version to 0.8.0 September 22, 2009
Octocat-spinner-32 README.rst Ad copy of millions of tasks per day not so impressive anymore :) June 05, 2012
Octocat-spinner-32 THANKS Fixes incorrect attribution in THANKS file February 22, 2012
Octocat-spinner-32 TODO Adds some TODO items November 08, 2011
Octocat-spinner-32 pavement.py Tests passing + flake8 April 12, 2012
Octocat-spinner-32 setup.cfg Use MP_MAIN_MODULE when execv is used to properly rename __main__ tas… June 03, 2012
Octocat-spinner-32 setup.py build: setup: only try removing the working dir from the path if its … June 05, 2012
Octocat-spinner-32 tox.ini Tests Py2.5 on Travis May 31, 2012
README.rst

celery - Distributed Task Queue

http://cloud.github.com/downloads/ask/celery/celery_128.png
Version: 2.6.0rc4
Web: http://celeryproject.org/
Download: http://pypi.python.org/pypi/celery/
Source: http://github.com/ask/celery/
Keywords: task queue, job queue, asynchronous, rabbitmq, amqp, redis, python, webhooks, queue, distributed

--

Synopsis

Celery is an open source asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operation, but supports scheduling as well.

The execution units, called tasks, are executed concurrently on one or more worker nodes using multiprocessing, Eventlet or gevent. Tasks can execute asynchronously (in the background) or synchronously (wait until ready).

Celery is used in production systems to process millions of tasks every hour.

Celery is written in Python, but the protocol can be implemented in any language. It can also operate with other languages using webhooks. There's also RCelery for the Ruby programming language, and a PHP client.

The recommended message broker is RabbitMQ, but support for Redis, MongoDB, Beanstalk, Amazon SQS, CouchDB and databases (using SQLAlchemy or the Django ORM) is also available.

Celery is easy to integrate with web frameworks, some of which even have integration packages:

Django django-celery
Pyramid pyramid_celery
Pylons celery-pylons
Flask flask-celery
web2py web2py-celery
Tornado tornado-celery

Overview

This is a high level overview of the architecture.

http://cloud.github.com/downloads/ask/celery/Celery-Overview-v4.jpg

The broker delivers tasks to the worker nodes. A worker node is a networked machine running celeryd. This can be one or more machines depending on the workload.

The result of the task can be stored for later retrieval (called its "tombstone").

Example

You probably want to see some code by now, so here's an example task adding two numbers:

from celery import task

@task
def add(x, y):
    return x + y

You can execute the task in the background, or wait for it to finish:

>>> result = add.delay(4, 4)
>>> result.wait() # wait for and return the result
8

Simple!

Features

Messaging Supported brokers include RabbitMQ, Redis, MongoDB, Beanstalk, SQL databases, Amazon SQS and more.
Fault-tolerant Excellent configurable error recovery when using RabbitMQ, ensures your tasks are never lost.
Distributed Runs on one or more machines. Supports broker clustering and HA when used in combination with RabbitMQ. You can set up new workers without central configuration (e.g. use your grandma's laptop to help if the queue is temporarily congested).
Concurrency Concurrency is achieved by using multiprocessing, Eventlet, gevent or a mix of these.
Scheduling Supports recurring tasks like cron, or specifying an exact date or countdown for when after the task should be executed.
Latency Low latency means you are able to execute tasks while the user is waiting.
Return Values Task return values can be saved to the selected result store backend. You can wait for the result, retrieve it later, or ignore it.
Result Stores Database, MongoDB, Redis, Memcached, Cassandra, or AMQP (message notification).
Webhooks Your tasks can also be HTTP callbacks, enabling cross-language communication.
Rate limiting Supports rate limiting by using the token bucket algorithm, which accounts for bursts of traffic. Rate limits can be set for each task type, or globally for all.
Routing Using AMQP's flexible routing model you can route tasks to different workers, or select different message topologies, by configuration or even at runtime.
Remote-control Worker nodes can be controlled from remote by using broadcast messaging. A range of built-in commands exist in addition to the ability to easily define your own. (AMQP/Redis only)
Monitoring You can capture everything happening with the workers in real-time by subscribing to events. A real-time web monitor is in development.
Serialization Supports Pickle, JSON, YAML, or easily defined custom schemes. One task invocation can have a different scheme than another.
Tracebacks Errors and tracebacks are stored and can be investigated after the fact.
UUID Every task has an UUID (Universally Unique Identifier), which is the task id used to query task status and return value.
Retries Tasks can be retried if they fail, with configurable maximum number of retries, and delays between each retry.
Task Sets A Task set is a task consisting of several sub-tasks. You can find out how many, or if all of the sub-tasks has been executed, and even retrieve the results in order. Progress bars, anyone?
Made for Web You can query status and results via URLs, enabling the ability to poll task status using Ajax.
Error Emails Can be configured to send emails to the administrators when tasks fails.

Documentation

The latest documentation with user guides, tutorials and API reference is hosted at Github.

Installation

You can install Celery either via the Python Package Index (PyPI) or from source.

To install using pip,:

$ pip install -U Celery

To install using easy_install,:

$ easy_install -U Celery

Bundles

Celery also defines a group of bundles that can be used to install Celery and the dependencies for a given feature.

The following bundles are available:

celery-with-redis:
  for using Redis as a broker.
celery-with-mongodb:
  for using MongoDB as a broker.
django-celery-with-redis:
  for Django, and using Redis as a broker.
django-celery-with-mongodb:
  for Django, and using MongoDB as a broker.

Downloading and installing from source

Download the latest version of Celery from http://pypi.python.org/pypi/celery/

You can install it by doing the following,:

$ tar xvfz celery-0.0.0.tar.gz
$ cd celery-0.0.0
$ python setup.py build
# python setup.py install # as root

Using the development version

You can clone the repository by doing the following:

$ git clone https://github.com/ask/celery
$ cd celery
$ python setup.py develop

The development version will usually also depend on the development version of kombu, the messaging framework Celery uses to send and receive messages, so you should also install that from git:

$ git clone https://github.com/ask/kombu
$ cd kombu
$ python setup.py develop

Getting Help

Mailing list

For discussions about the usage, development, and future of celery, please join the celery-users mailing list.

IRC

Come chat with us on IRC. The #celery channel is located at the Freenode network.

Bug tracker

If you have any suggestions, bug reports or annoyances please report them to our issue tracker at http://github.com/ask/celery/issues/

Wiki

http://wiki.github.com/ask/celery/

Contributing

Development of celery happens at Github: http://github.com/ask/celery

You are highly encouraged to participate in the development of celery. If you don't like Github (for some reason) you're welcome to send regular patches.

Be sure to also read the Contributing to Celery section in the documentation.

License

This software is licensed under the New BSD License. See the LICENSE file in the top distribution directory for the full license text.

Something went wrong with that request. Please try again.