Celery integration with Tornado
Python
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
demos
requirements
totoro
.gitignore
LICENSE
MANIFEST.in
README.rst
setup.cfg
setup.py

README.rst

Totoro

Celery integration with Tornado.

Installation

You can install Totoro either via the Python Package Index (PyPI) or from source.

To install using pip, simply:

$ sudo pip install totoro

or alternatively (you really should be using pip though):

$ sudo easy_install totoro

or from source:

$ sudo python setup.py install

Totoro can only support AMQP(RabbitMQ) by default. To use the Redis, you can specify the requirements on the pip comand-line by using brackets.

$ sudo pip install totoro[redis]

Hello, world

Here is a simple "Hello, world!" example for calling celery tasks from Tornado RequestHandler:

#!/usr/bin/env python

import tornado.httpserver
import tornado.ioloop
import tornado.web

from tornado import gen
import totoro
from totoro.test.celery_tasks import tasks


class MainHandler(tornado.web.RequestHandler):
    @gen.coroutine
    def get(self):
        response = yield gen.Task(tasks.echo.apply_async, args=['Hello world!'])
        self.write(response.result)


def main():
    totoro.setup_producer()
    application = tornado.web.Application([
        (r"/", MainHandler),
    ])
    http_server = tornado.httpserver.HTTPServer(application)
    http_server.listen(8888)
    tornado.ioloop.IOLoop.instance().start()


if __name__ == "__main__":
    main()

To run celery worker for the example:

$ python -m totoro.test.runtasks worker -l INFO

Configuration and defaults

TOTORO_AMQP_CONNECTION_POOL

New in version 0.1.2

The setting must a dict that used to configure the AMQP(RabbitMQ) connection pool. It supporting the following keys:

  • max_idle_connections - Max number of keeping connections. Defaults to 3.
  • max_open_connections - Max number of opened connections, 0 means no limit. Defaults to 10.
  • max_recycle_sec - How long connections are recycled. Defaults to 3600.

Example configuration:

celery = Celery("totoro_celery_tasks")
celery.conf.update(
    BROKER_URL='amqp://guest:guest@localhost:5672//',
    CELERY_TASK_SERIALIZER='json',
    CELERY_ACCEPT_CONTENT=['json'],  # Ignore other content
    CELERY_RESULT_SERIALIZER='json',
    CELERY_TIMEZONE='Asia/Shanghai',
    CELERY_ENABLE_UTC=True,
    TOTORO_AMQP_CONNECTION_POOL={
        'max_idle_connections': 1,
        'max_open_connections': 10,
        'max_recycle_sec': 3600
    },
)

Tests

To run the tests for the AMQP(broker/backend):

$ python -m totoro.test.runtasks worker -l INFO
$ python -m totoro.test.runtests

To run the tests for the AMQP broker with the Redis backend:

$ CELERY_RESULT_BACKEND=redis://127.0.0.1:6379/0  python -m totoro.test.runtasks worker -l INFO
$ CELERY_RESULT_BACKEND=redis://127.0.0.1:6379/0  python -m totoro.test.runtests

To run the tests for the Redis(broker/backend):

$ BROKER_URL=redis://127.0.0.1:6379/0  CELERY_RESULT_BACKEND=redis://127.0.0.1:6379/0 python -m totoro.test.runtasks worker -l INFO
$ BROKER_URL=redis://127.0.0.1:6379/0  CELERY_RESULT_BACKEND=redis://127.0.0.1:6379/0 python -m totoro.test.runtests