Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
Newer
Older
100644 383 lines (291 sloc) 14.627 kb
4434bf2 Add information about using celery with Python and other languages to
Ask Solem authored
1 =================================
2 celery - Distributed Task Queue
3 =================================
912c78d README.rst is now the real textfile, while README is a symbolic link
Ask Solem authored
4
72aa282 @ask Use new logo
authored
5 .. image:: http://cloud.github.com/downloads/ask/celery/celery_128.png
a3639f3 Move README logo to the top.
Ask Solem authored
6
dbb9e97 @ask Renders README
authored
7 :Version: 2.6.0rc4
7d1ddbe Rendered README.rst
Ask Solem authored
8 :Web: http://celeryproject.org/
9 :Download: http://pypi.python.org/pypi/celery/
10 :Source: http://github.com/ask/celery/
e6fd05c Rendered README.rst
Ask Solem authored
11 :Keywords: task queue, job queue, asynchronous, rabbitmq, amqp, redis,
dc853ae Updated README
Ask Solem authored
12 python, webhooks, queue, distributed
912c78d README.rst is now the real textfile, while README is a symbolic link
Ask Solem authored
13
ad935fd Reorganized documentation
Ask Solem authored
14 --
d6dfee1 Updated README
Ask Solem authored
15
5e26137 @ask Mention Pyramid in readme
authored
16 .. contents::
17 :local:
18
f6199fb Loads of documentation improvements
Ask Solem authored
19 .. _celery-synopsis:
20
5e26137 @ask Mention Pyramid in readme
authored
21 Synopsis
22 ========
23
528f148 Bumped version to 2.0.1
Ask Solem authored
24 Celery is an open source asynchronous task queue/job queue based on
6a6ae66 README/introduction/homepage: Added link to Flask-Celery
Ask Solem authored
25 distributed message passing. It is focused on real-time operation,
528f148 Bumped version to 2.0.1
Ask Solem authored
26 but supports scheduling as well.
d6dfee1 Updated README
Ask Solem authored
27
f6199fb Loads of documentation improvements
Ask Solem authored
28 The execution units, called tasks, are executed concurrently on one or
88d92ed @ask Bumps version to 2.2.0rc2
authored
29 more worker nodes using multiprocessing, `Eventlet`_ or `gevent`_. Tasks can
30 execute asynchronously (in the background) or synchronously
31 (wait until ready).
d6dfee1 Updated README
Ask Solem authored
32
3219c2d @ask Ad copy of millions of tasks per day not so impressive anymore :)
authored
33 Celery is used in production systems to process millions of tasks every hour.
d2beda1 Fix broken link in README.rst
Ask Solem authored
34
84a22ae Updated README text mentioning Django
Ask Solem authored
35 Celery is written in Python, but the protocol can be implemented in any
6a6ae66 README/introduction/homepage: Added link to Flask-Celery
Ask Solem authored
36 language. It can also `operate with other languages using webhooks`_.
82842bd @ask Mention ruby celery, php-celery, web2py and pyramid in the readme
authored
37 There's also `RCelery` for the Ruby programming language, and a `PHP client`.
e0235b2 Updated documentation to reflect changes in announcement
Ask Solem authored
38
82842bd @ask Mention ruby celery, php-celery, web2py and pyramid in the readme
authored
39 The recommended message broker is `RabbitMQ`_, but support for
5e26137 @ask Mention Pyramid in readme
authored
40 `Redis`_, `MongoDB`_, `Beanstalk`_, `Amazon SQS`_, `CouchDB`_ and
88d92ed @ask Bumps version to 2.2.0rc2
authored
41 databases (using `SQLAlchemy`_ or the `Django ORM`_) is also available.
4434bf2 Add information about using celery with Python and other languages to
Ask Solem authored
42
5e26137 @ask Mention Pyramid in readme
authored
43 Celery is easy to integrate with web frameworks, some of which even have
44 integration packages:
45
f4e9d0f @mher Adds tornado-celery into the intagration list
mher authored
46 +--------------------+------------------------+
47 | `Django`_ | `django-celery`_ |
48 +--------------------+------------------------+
49 | `Pyramid`_ | `pyramid_celery`_ |
50 +--------------------+------------------------+
51 | `Pylons`_ | `celery-pylons`_ |
52 +--------------------+------------------------+
53 | `Flask`_ | `flask-celery`_ |
54 +--------------------+------------------------+
55 | `web2py`_ | `web2py-celery`_ |
56 +--------------------+------------------------+
df33002 @mher Adds Tornado url
mher authored
57 | `Tornado`_ | `tornado-celery`_ |
f4e9d0f @mher Adds tornado-celery into the intagration list
mher authored
58 +--------------------+------------------------+
84a22ae Updated README text mentioning Django
Ask Solem authored
59
82842bd @ask Mention ruby celery, php-celery, web2py and pyramid in the readme
authored
60 .. _`RCelery`: http://leapfrogdevelopment.github.com/rcelery/
61 .. _`PHP client`: https://github.com/gjedeer/celery-php
872620c examples/django moved to django-celery
Ask Solem authored
62 .. _`RabbitMQ`: http://www.rabbitmq.com/
63 .. _`Redis`: http://code.google.com/p/redis/
84a22ae Updated README text mentioning Django
Ask Solem authored
64 .. _`SQLAlchemy`: http://www.sqlalchemy.org/
88d92ed @ask Bumps version to 2.2.0rc2
authored
65 .. _`Django`: http://djangoproject.com/
66 .. _`Django ORM`: http://djangoproject.com/
547c071 @ask Installation of git version should mention Kombu dep
authored
67 .. _`Memcached`: http://memcached.org/
88d92ed @ask Bumps version to 2.2.0rc2
authored
68 .. _`Eventlet`: http://eventlet.net/
69 .. _`gevent`: http://gevent.org/
70 .. _`Beanstalk`: http://kr.github.com/beanstalkd/
71 .. _`MongoDB`: http://mongodb.org/
72 .. _`CouchDB`: http://couchdb.apache.org/
82842bd @ask Mention ruby celery, php-celery, web2py and pyramid in the readme
authored
73 .. _`Amazon SQS`: http://aws.amazon.com/sqs/
6a6ae66 README/introduction/homepage: Added link to Flask-Celery
Ask Solem authored
74 .. _`Pylons`: http://pylonshq.com/
75 .. _`Flask`: http://flask.pocoo.org/
82842bd @ask Mention ruby celery, php-celery, web2py and pyramid in the readme
authored
76 .. _`web2py`: http://web2py.com/
77 .. _`Bottle`: http://bottlepy.org/
78 .. _`Pyramid`: http://docs.pylonsproject.org/en/latest/docs/pyramid.html
5e26137 @ask Mention Pyramid in readme
authored
79 .. _`pyramid_celery`: http://pypi.python.org/pypi/pyramid_celery/
dc853ae Updated README
Ask Solem authored
80 .. _`django-celery`: http://pypi.python.org/pypi/django-celery
3ffd0f8 Now links to celery-pylons on PyPI
Ask Solem authored
81 .. _`celery-pylons`: http://pypi.python.org/pypi/celery-pylons
5e26137 @ask Mention Pyramid in readme
authored
82 .. _`flask-celery`: http://github.com/ask/flask-celery/
82842bd @ask Mention ruby celery, php-celery, web2py and pyramid in the readme
authored
83 .. _`web2py-celery`: http://code.google.com/p/web2py-celery/
df33002 @mher Adds Tornado url
mher authored
84 .. _`Tornado`: http://www.tornadoweb.org/
f4e9d0f @mher Adds tornado-celery into the intagration list
mher authored
85 .. _`tornado-celery`: http://github.com/mher/tornado-celery/
84a22ae Updated README text mentioning Django
Ask Solem authored
86 .. _`operate with other languages using webhooks`:
937f556 Fixed link to HTTP callback tasks
Ask Solem authored
87 http://ask.github.com/celery/userguide/remote-tasks.html
283928d @ask Updates Changelog for 2.3.1
authored
88 .. _`limited support`:
89 http://kombu.readthedocs.org/en/latest/introduction.html#transport-comparison
912c78d README.rst is now the real textfile, while README is a symbolic link
Ask Solem authored
90
f6199fb Loads of documentation improvements
Ask Solem authored
91 .. _celery-overview:
92
912c78d README.rst is now the real textfile, while README is a symbolic link
Ask Solem authored
93 Overview
94 ========
95
e96852f Illustrational look inside the worker server.
Ask Solem authored
96 This is a high level overview of the architecture.
97
afa85ff README: Link to new images
Ask Solem authored
98 .. image:: http://cloud.github.com/downloads/ask/celery/Celery-Overview-v4.jpg
912c78d README.rst is now the real textfile, while README is a symbolic link
Ask Solem authored
99
88d92ed @ask Bumps version to 2.2.0rc2
authored
100 The broker delivers tasks to the worker nodes.
101 A worker node is a networked machine running `celeryd`. This can be one or
4d496f2 Bumped version to v1.1.2
Ask Solem authored
102 more machines depending on the workload.
e96852f Illustrational look inside the worker server.
Ask Solem authored
103
104 The result of the task can be stored for later retrieval (called its
105 "tombstone").
106
f6199fb Loads of documentation improvements
Ask Solem authored
107 .. _celery-example:
108
ad935fd Reorganized documentation
Ask Solem authored
109 Example
110 =======
111
7d1ddbe Rendered README.rst
Ask Solem authored
112 You probably want to see some code by now, so here's an example task
ad935fd Reorganized documentation
Ask Solem authored
113 adding two numbers:
ac82259 sphinx-to-rst: Also process includes
Ask Solem authored
114 ::
ad935fd Reorganized documentation
Ask Solem authored
115
7e3e22f @ask Bumps version to 2.6.0a2
authored
116 from celery import task
ad935fd Reorganized documentation
Ask Solem authored
117
118 @task
119 def add(x, y):
120 return x + y
121
122 You can execute the task in the background, or wait for it to finish::
123
124 >>> result = add.delay(4, 4)
125 >>> result.wait() # wait for and return the result
126 8
127
128 Simple!
129
f6199fb Loads of documentation improvements
Ask Solem authored
130 .. _celery-features:
131
912c78d README.rst is now the real textfile, while README is a symbolic link
Ask Solem authored
132 Features
133 ========
134
e6fd05c Rendered README.rst
Ask Solem authored
135 +-----------------+----------------------------------------------------+
88d92ed @ask Bumps version to 2.2.0rc2
authored
136 | Messaging | Supported brokers include `RabbitMQ`_, `Redis`_, |
547c071 @ask Installation of git version should mention Kombu dep
authored
137 | | `MongoDB`_, `Beanstalk`_, SQL databases, |
138 | | Amazon SQS and more. |
e6fd05c Rendered README.rst
Ask Solem authored
139 +-----------------+----------------------------------------------------+
88d92ed @ask Bumps version to 2.2.0rc2
authored
140 | Fault-tolerant | Excellent configurable error recovery when using |
141 | | `RabbitMQ`, ensures your tasks are never lost. |
e6fd05c Rendered README.rst
Ask Solem authored
142 +-----------------+----------------------------------------------------+
143 | Distributed | Runs on one or more machines. Supports |
88d92ed @ask Bumps version to 2.2.0rc2
authored
144 | | broker `clustering`_ and `HA`_ when used in |
145 | | combination with `RabbitMQ`_. You can set up new |
146 | | workers without central configuration (e.g. use |
147 | | your grandma's laptop to help if the queue is |
148 | | temporarily congested). |
e6fd05c Rendered README.rst
Ask Solem authored
149 +-----------------+----------------------------------------------------+
88d92ed @ask Bumps version to 2.2.0rc2
authored
150 | Concurrency | Concurrency is achieved by using multiprocessing, |
151 | | `Eventlet`_, `gevent` or a mix of these. |
e6fd05c Rendered README.rst
Ask Solem authored
152 +-----------------+----------------------------------------------------+
153 | Scheduling | Supports recurring tasks like cron, or specifying |
154 | | an exact date or countdown for when after the task |
155 | | should be executed. |
156 +-----------------+----------------------------------------------------+
88d92ed @ask Bumps version to 2.2.0rc2
authored
157 | Latency | Low latency means you are able to execute tasks |
158 | | *while the user is waiting*. |
e6fd05c Rendered README.rst
Ask Solem authored
159 +-----------------+----------------------------------------------------+
160 | Return Values | Task return values can be saved to the selected |
161 | | result store backend. You can wait for the result, |
162 | | retrieve it later, or ignore it. |
163 +-----------------+----------------------------------------------------+
547c071 @ask Installation of git version should mention Kombu dep
authored
164 | Result Stores | Database, `MongoDB`_, `Redis`_, `Memcached`_, |
88d92ed @ask Bumps version to 2.2.0rc2
authored
165 | | `Cassandra`, or `AMQP`_ (message notification). |
e6fd05c Rendered README.rst
Ask Solem authored
166 +-----------------+----------------------------------------------------+
167 | Webhooks | Your tasks can also be HTTP callbacks, enabling |
168 | | cross-language communication. |
169 +-----------------+----------------------------------------------------+
170 | Rate limiting | Supports rate limiting by using the token bucket |
171 | | algorithm, which accounts for bursts of traffic. |
172 | | Rate limits can be set for each task type, or |
173 | | globally for all. |
174 +-----------------+----------------------------------------------------+
88d92ed @ask Bumps version to 2.2.0rc2
authored
175 | Routing | Using AMQP's flexible routing model you can route |
176 | | tasks to different workers, or select different |
177 | | message topologies, by configuration or even at |
178 | | runtime. |
e6fd05c Rendered README.rst
Ask Solem authored
179 +-----------------+----------------------------------------------------+
88d92ed @ask Bumps version to 2.2.0rc2
authored
180 | Remote-control | Worker nodes can be controlled from remote by |
181 | | using broadcast messaging. A range of built-in |
182 | | commands exist in addition to the ability to |
183 | | easily define your own. (AMQP/Redis only) |
e6fd05c Rendered README.rst
Ask Solem authored
184 +-----------------+----------------------------------------------------+
185 | Monitoring | You can capture everything happening with the |
186 | | workers in real-time by subscribing to events. |
187 | | A real-time web monitor is in development. |
188 +-----------------+----------------------------------------------------+
189 | Serialization | Supports Pickle, JSON, YAML, or easily defined |
190 | | custom schemes. One task invocation can have a |
191 | | different scheme than another. |
192 +-----------------+----------------------------------------------------+
193 | Tracebacks | Errors and tracebacks are stored and can be |
194 | | investigated after the fact. |
195 +-----------------+----------------------------------------------------+
196 | UUID | Every task has an UUID (Universally Unique |
197 | | Identifier), which is the task id used to query |
198 | | task status and return value. |
199 +-----------------+----------------------------------------------------+
200 | Retries | Tasks can be retried if they fail, with |
201 | | configurable maximum number of retries, and delays |
202 | | between each retry. |
203 +-----------------+----------------------------------------------------+
204 | Task Sets | A Task set is a task consisting of several |
205 | | sub-tasks. You can find out how many, or if all |
206 | | of the sub-tasks has been executed, and even |
207 | | retrieve the results in order. Progress bars, |
208 | | anyone? |
209 +-----------------+----------------------------------------------------+
210 | Made for Web | You can query status and results via URLs, |
211 | | enabling the ability to poll task status using |
212 | | Ajax. |
213 +-----------------+----------------------------------------------------+
233366e @ask Bumps version to 2.3.0a2
authored
214 | Error Emails | Can be configured to send emails to the |
e6fd05c Rendered README.rst
Ask Solem authored
215 | | administrators when tasks fails. |
216 +-----------------+----------------------------------------------------+
217
218
219 .. _`clustering`: http://www.rabbitmq.com/clustering.html
88d92ed @ask Bumps version to 2.2.0rc2
authored
220 .. _`HA`: http://www.rabbitmq.com/pacemaker.html
e6fd05c Rendered README.rst
Ask Solem authored
221 .. _`AMQP`: http://www.amqp.org/
222 .. _`Stomp`: http://stomp.codehaus.org/
83f164b README: Add mongodb to backend list
Ask Solem authored
223 .. _`Tokyo Tyrant`: http://tokyocabinet.sourceforge.net/
224
f6199fb Loads of documentation improvements
Ask Solem authored
225 .. _celery-documentation:
226
ad935fd Reorganized documentation
Ask Solem authored
227 Documentation
228 =============
912c78d README.rst is now the real textfile, while README is a symbolic link
Ask Solem authored
229
ad935fd Reorganized documentation
Ask Solem authored
230 The `latest documentation`_ with user guides, tutorials and API reference
231 is hosted at Github.
912c78d README.rst is now the real textfile, while README is a symbolic link
Ask Solem authored
232
ad935fd Reorganized documentation
Ask Solem authored
233 .. _`latest documentation`: http://ask.github.com/celery/
912c78d README.rst is now the real textfile, while README is a symbolic link
Ask Solem authored
234
f6199fb Loads of documentation improvements
Ask Solem authored
235 .. _celery-installation:
236
912c78d README.rst is now the real textfile, while README is a symbolic link
Ask Solem authored
237 Installation
f6199fb Loads of documentation improvements
Ask Solem authored
238 ============
912c78d README.rst is now the real textfile, while README is a symbolic link
Ask Solem authored
239
bd18d7f Bumped version to 2.2.0a2
Ask Solem authored
240 You can install Celery either via the Python Package Index (PyPI)
ac82259 sphinx-to-rst: Also process includes
Ask Solem authored
241 or from source.
242
0afa1ef Docs: Replaced all occurences of ``literal`` with `literal`
Ask Solem authored
243 To install using `pip`,::
ac82259 sphinx-to-rst: Also process includes
Ask Solem authored
244
94820f9 @ask Bumps version to 2.4.5
authored
245 $ pip install -U Celery
ac82259 sphinx-to-rst: Also process includes
Ask Solem authored
246
0afa1ef Docs: Replaced all occurences of ``literal`` with `literal`
Ask Solem authored
247 To install using `easy_install`,::
ac82259 sphinx-to-rst: Also process includes
Ask Solem authored
248
94820f9 @ask Bumps version to 2.4.5
authored
249 $ easy_install -U Celery
250
251 Bundles
252 -------
253
254 Celery also defines a group of bundles that can be used
255 to install Celery and the dependencies for a given feature.
256
257 The following bundles are available:
258
259 :`celery-with-redis`_:
260 for using Redis as a broker.
261
262 :`celery-with-mongodb`_:
263 for using MongoDB as a broker.
264
265 :`django-celery-with-redis`_:
266 for Django, and using Redis as a broker.
267
268 :`django-celery-with-mongodb`_:
269 for Django, and using MongoDB as a broker.
270
271 .. _`celery-with-redis`:
272 http://pypi.python.org/pypi/celery-with-redis/
273 .. _`celery-with-mongodb`:
274 http://pypi.python.org/pypi/celery-with-mongdb/
275 .. _`django-celery-with-redis`:
276 http://pypi.python.org/pypi/django-celery-with-redis/
277 .. _`django-celery-with-mongodb`:
278 http://pypi.python.org/pypi/django-celery-with-mongdb/
ac82259 sphinx-to-rst: Also process includes
Ask Solem authored
279
f6199fb Loads of documentation improvements
Ask Solem authored
280 .. _celery-installing-from-source:
281
ac82259 sphinx-to-rst: Also process includes
Ask Solem authored
282 Downloading and installing from source
283 --------------------------------------
284
bd18d7f Bumped version to 2.2.0a2
Ask Solem authored
285 Download the latest version of Celery from
ac82259 sphinx-to-rst: Also process includes
Ask Solem authored
286 http://pypi.python.org/pypi/celery/
287
288 You can install it by doing the following,::
289
290 $ tar xvfz celery-0.0.0.tar.gz
291 $ cd celery-0.0.0
292 $ python setup.py build
293 # python setup.py install # as root
294
f6199fb Loads of documentation improvements
Ask Solem authored
295 .. _celery-installing-from-git:
296
ac82259 sphinx-to-rst: Also process includes
Ask Solem authored
297 Using the development version
f6199fb Loads of documentation improvements
Ask Solem authored
298 -----------------------------
ac82259 sphinx-to-rst: Also process includes
Ask Solem authored
299
300 You can clone the repository by doing the following::
301
547c071 @ask Installation of git version should mention Kombu dep
authored
302 $ git clone https://github.com/ask/celery
303 $ cd celery
304 $ python setup.py develop
305
306 The development version will usually also depend on the development
307 version of `kombu`_, the messaging framework Celery uses
308 to send and receive messages, so you should also install that from git::
309
310 $ git clone https://github.com/ask/kombu
311 $ cd kombu
312 $ python setup.py develop
313
3219c2d @ask Ad copy of millions of tasks per day not so impressive anymore :)
authored
314 .. _`kombu`: http://kombu.readthedocs.org/en/latest/
6b74203 Autogenerated README.rst out of docs/introduction.rst
Ask Solem authored
315
f6199fb Loads of documentation improvements
Ask Solem authored
316 .. _getting-help:
b355abb Even more reorganization of the documentation
Ask Solem authored
317
912c78d README.rst is now the real textfile, while README is a symbolic link
Ask Solem authored
318 Getting Help
319 ============
320
f6199fb Loads of documentation improvements
Ask Solem authored
321 .. _mailing-list:
322
912c78d README.rst is now the real textfile, while README is a symbolic link
Ask Solem authored
323 Mailing list
324 ------------
325
326 For discussions about the usage, development, and future of celery,
bae2e49 Remove whitespace [Gun.io WhitespaceBot]
Gun.io Whitespace Robot authored
327 please join the `celery-users`_ mailing list.
912c78d README.rst is now the real textfile, while README is a symbolic link
Ask Solem authored
328
329 .. _`celery-users`: http://groups.google.com/group/celery-users/
330
f6199fb Loads of documentation improvements
Ask Solem authored
331 .. _irc-channel:
332
912c78d README.rst is now the real textfile, while README is a symbolic link
Ask Solem authored
333 IRC
334 ---
335
336 Come chat with us on IRC. The `#celery`_ channel is located at the `Freenode`_
337 network.
338
339 .. _`#celery`: irc://irc.freenode.net/celery
340 .. _`Freenode`: http://freenode.net
341
f6199fb Loads of documentation improvements
Ask Solem authored
342 .. _bug-tracker:
912c78d README.rst is now the real textfile, while README is a symbolic link
Ask Solem authored
343
344 Bug tracker
345 ===========
346
347 If you have any suggestions, bug reports or annoyances please report them
348 to our issue tracker at http://github.com/ask/celery/issues/
349
f6199fb Loads of documentation improvements
Ask Solem authored
350 .. _wiki:
351
7d1ddbe Rendered README.rst
Ask Solem authored
352 Wiki
353 ====
354
355 http://wiki.github.com/ask/celery/
356
6a50b1c Bumped version to 2.2.0a6
Ask Solem authored
357 .. _contributing-short:
f6199fb Loads of documentation improvements
Ask Solem authored
358
912c78d README.rst is now the real textfile, while README is a symbolic link
Ask Solem authored
359 Contributing
360 ============
361
0afa1ef Docs: Replaced all occurences of ``literal`` with `literal`
Ask Solem authored
362 Development of `celery` happens at Github: http://github.com/ask/celery
912c78d README.rst is now the real textfile, while README is a symbolic link
Ask Solem authored
363
364 You are highly encouraged to participate in the development
0afa1ef Docs: Replaced all occurences of ``literal`` with `literal`
Ask Solem authored
365 of `celery`. If you don't like Github (for some reason) you're welcome
912c78d README.rst is now the real textfile, while README is a symbolic link
Ask Solem authored
366 to send regular patches.
367
88d92ed @ask Bumps version to 2.2.0rc2
authored
368 Be sure to also read the `Contributing to Celery`_ section in the
369 documentation.
370
371 .. _`Contributing to Celery`: http://ask.github.com/celery/contributing.html
6a50b1c Bumped version to 2.2.0a6
Ask Solem authored
372
f6199fb Loads of documentation improvements
Ask Solem authored
373 .. _license:
374
912c78d README.rst is now the real textfile, while README is a symbolic link
Ask Solem authored
375 License
376 =======
377
0afa1ef Docs: Replaced all occurences of ``literal`` with `literal`
Ask Solem authored
378 This software is licensed under the `New BSD License`. See the ``LICENSE``
912c78d README.rst is now the real textfile, while README is a symbolic link
Ask Solem authored
379 file in the top distribution directory for the full license text.
380
381 .. # vim: syntax=rst expandtab tabstop=4 shiftwidth=4 shiftround
8f9e509 README.rst generated out of docs/introduction.rst
Ask Solem authored
382
Something went wrong with that request. Please try again.