Skip to content

Commit

Permalink
Make installable.
Browse files Browse the repository at this point in the history
  • Loading branch information
clokep committed Oct 21, 2018
1 parent e3923f9 commit be71348
Show file tree
Hide file tree
Showing 3 changed files with 146 additions and 1 deletion.
14 changes: 13 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
@@ -1,8 +1,20 @@
*.pyc
__pycache__

# Testing
# Build related.
_build
build
dist
celery_batches.egg-info

# Testing related.
.cache
.pytest_cache
.tox

# Coverage related.
.coverage
htmlcov

# Vagrant
.vagrant
85 changes: 85 additions & 0 deletions README.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
Twisted-Celery
##############

Twisted-Celery is a connector to call Celery tasks (and receive results) from an
application running with Twisted.

The Problem
===========

Twisted and Celery are both "asynchronous", but have dramatically different
use-cases. Twisted is an event-drive networking framework, but doesn't perform
well for long lived CPU-intensive tasks. Celery is a distributed task runner
that excels at short-lived CPU bound work.

Sometimes you want to run a task from Twisted...but this is harder than it
seems. You might think you can just import your Celery app and run the tasks,
but this causes synchronous I/O to happen in the middle of your Twisted process.
Another solution might be to do all Celery calls inside of a thread (by using
``deferToThread`` and friends), while this is better than doing I/O on the
reactor thread you'll quickly cause thread-pool starvation.

Luckily, there's another way...

The Solution
============

Twisted-Celery uses the internals of Celery to create Celery-compatible
messages, but then uses Twisted to do distributed I/O to communicate with your
configured Celery broker and backend. It does this while exposing a
``send_task`` API identical to Celery's. Additionally, when calling a task it
returns a native ``Deferred`` instead of an ``AsyncResult`` so you can write
Twisted-compatible code easily.

Supported Features
==================

Currently the only supported broker and backend are over AMQP. Support for AMQP
in Twisted is provided via `pika <https://pika.readthedocs.io>`_.

If you'd like to use a different backend or broker, please contribute!

How it Works
============

Twisted-Celery sits somewhere in-between the Celery and Kombu layers of a
traditional Celery application. This is because much of Celery is really
implemented inside of Kombu (e.g. the serialization and message creation
facilities), while some of the higher level constructs are from Celery.

Where possible, calls into Celery and Kombu are made. Unfortunately much of the
internals of those frameworks assume synchonrous code (e.g.
``ensure_connection`` blocks if a new connection needs to be opened). The
easiest way to implement this was to copy portions of code out of Celery and
Kombu in order to make them asynchronous.

Getting Started
===============

After creating a Celery application as normal, import the ``TwistedCelery``
class and use it to wrap your Celery app. Use the ``send_task`` method provided
on this class within Twisted. It returns a ``Deferred`` that resolves to the
result of the task call.

.. code-block:: python
from twistedcelery import TwistedCelery
from twisted.internet import defer, task
from my_app import app
@defer.inlineCallbacks
def main(reactor):
# Create the Twisted Celery application.
tx_app = TwistedCelery(app)
# Execute a task (and get the result).
print("Sending task")
result = yield tx_app.send_task('my_app.tasks.some_task')
print("Got result: ", result)
tx_app.disconnect()
if __name__ == '__main__':
task.react(main)
48 changes: 48 additions & 0 deletions setup.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import codecs
import setuptools


def long_description():
try:
return codecs.open('README.rst', 'r', 'utf-8').read()
except IOError:
return 'Long description error: Missing README.rst file'


setuptools.setup(
name='twisted-celery',
packages=setuptools.find_packages(),
version='0.0.1',
description='Celery connector for Twisted.',
long_description=long_description(),
keywords='twisted celery',
author='Patrick Cloke',
author_email='clokep@patrick.cloke.us',
url='https://github.com/clokep/twisted-celery',
license='BSD',
platforms=['any'],
install_requires=[
'celery>=4.0,<5.0',
# Probably works with older versions.
'twisted>=18.7.0',
# Probably works with older versions.
'pika>=0.12.0',
],
classifiers=[
'Development Status :: 2 - Pre-Alpha',
'License :: OSI Approved :: BSD License',
'Topic :: System :: Distributed Computing',
'Programming Language :: Python',
'Programming Language :: Python :: 2',
'Programming Language :: Python :: 2.7',
'Programming Language :: Python :: 3',
'Programming Language :: Python :: 3.4',
'Programming Language :: Python :: 3.5',
'Programming Language :: Python :: 3.6',
'Programming Language :: Python :: Implementation :: CPython',
'Programming Language :: Python :: Implementation :: PyPy',
'Operating System :: OS Independent',
],
)

0 comments on commit be71348

Please sign in to comment.