-
Notifications
You must be signed in to change notification settings - Fork 1
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
3 changed files
with
146 additions
and
1 deletion.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,8 +1,20 @@ | ||
*.pyc | ||
__pycache__ | ||
|
||
# Testing | ||
# Build related. | ||
_build | ||
build | ||
dist | ||
celery_batches.egg-info | ||
|
||
# Testing related. | ||
.cache | ||
.pytest_cache | ||
.tox | ||
|
||
# Coverage related. | ||
.coverage | ||
htmlcov | ||
|
||
# Vagrant | ||
.vagrant |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,85 @@ | ||
Twisted-Celery | ||
############## | ||
|
||
Twisted-Celery is a connector to call Celery tasks (and receive results) from an | ||
application running with Twisted. | ||
|
||
The Problem | ||
=========== | ||
|
||
Twisted and Celery are both "asynchronous", but have dramatically different | ||
use-cases. Twisted is an event-drive networking framework, but doesn't perform | ||
well for long lived CPU-intensive tasks. Celery is a distributed task runner | ||
that excels at short-lived CPU bound work. | ||
|
||
Sometimes you want to run a task from Twisted...but this is harder than it | ||
seems. You might think you can just import your Celery app and run the tasks, | ||
but this causes synchronous I/O to happen in the middle of your Twisted process. | ||
Another solution might be to do all Celery calls inside of a thread (by using | ||
``deferToThread`` and friends), while this is better than doing I/O on the | ||
reactor thread you'll quickly cause thread-pool starvation. | ||
|
||
Luckily, there's another way... | ||
|
||
The Solution | ||
============ | ||
|
||
Twisted-Celery uses the internals of Celery to create Celery-compatible | ||
messages, but then uses Twisted to do distributed I/O to communicate with your | ||
configured Celery broker and backend. It does this while exposing a | ||
``send_task`` API identical to Celery's. Additionally, when calling a task it | ||
returns a native ``Deferred`` instead of an ``AsyncResult`` so you can write | ||
Twisted-compatible code easily. | ||
|
||
Supported Features | ||
================== | ||
|
||
Currently the only supported broker and backend are over AMQP. Support for AMQP | ||
in Twisted is provided via `pika <https://pika.readthedocs.io>`_. | ||
|
||
If you'd like to use a different backend or broker, please contribute! | ||
|
||
How it Works | ||
============ | ||
|
||
Twisted-Celery sits somewhere in-between the Celery and Kombu layers of a | ||
traditional Celery application. This is because much of Celery is really | ||
implemented inside of Kombu (e.g. the serialization and message creation | ||
facilities), while some of the higher level constructs are from Celery. | ||
|
||
Where possible, calls into Celery and Kombu are made. Unfortunately much of the | ||
internals of those frameworks assume synchonrous code (e.g. | ||
``ensure_connection`` blocks if a new connection needs to be opened). The | ||
easiest way to implement this was to copy portions of code out of Celery and | ||
Kombu in order to make them asynchronous. | ||
|
||
Getting Started | ||
=============== | ||
|
||
After creating a Celery application as normal, import the ``TwistedCelery`` | ||
class and use it to wrap your Celery app. Use the ``send_task`` method provided | ||
on this class within Twisted. It returns a ``Deferred`` that resolves to the | ||
result of the task call. | ||
|
||
.. code-block:: python | ||
from twistedcelery import TwistedCelery | ||
from twisted.internet import defer, task | ||
from my_app import app | ||
@defer.inlineCallbacks | ||
def main(reactor): | ||
# Create the Twisted Celery application. | ||
tx_app = TwistedCelery(app) | ||
# Execute a task (and get the result). | ||
print("Sending task") | ||
result = yield tx_app.send_task('my_app.tasks.some_task') | ||
print("Got result: ", result) | ||
tx_app.disconnect() | ||
if __name__ == '__main__': | ||
task.react(main) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,48 @@ | ||
#!/usr/bin/env python | ||
# -*- coding: utf-8 -*- | ||
import codecs | ||
import setuptools | ||
|
||
|
||
def long_description(): | ||
try: | ||
return codecs.open('README.rst', 'r', 'utf-8').read() | ||
except IOError: | ||
return 'Long description error: Missing README.rst file' | ||
|
||
|
||
setuptools.setup( | ||
name='twisted-celery', | ||
packages=setuptools.find_packages(), | ||
version='0.0.1', | ||
description='Celery connector for Twisted.', | ||
long_description=long_description(), | ||
keywords='twisted celery', | ||
author='Patrick Cloke', | ||
author_email='clokep@patrick.cloke.us', | ||
url='https://github.com/clokep/twisted-celery', | ||
license='BSD', | ||
platforms=['any'], | ||
install_requires=[ | ||
'celery>=4.0,<5.0', | ||
# Probably works with older versions. | ||
'twisted>=18.7.0', | ||
# Probably works with older versions. | ||
'pika>=0.12.0', | ||
], | ||
classifiers=[ | ||
'Development Status :: 2 - Pre-Alpha', | ||
'License :: OSI Approved :: BSD License', | ||
'Topic :: System :: Distributed Computing', | ||
'Programming Language :: Python', | ||
'Programming Language :: Python :: 2', | ||
'Programming Language :: Python :: 2.7', | ||
'Programming Language :: Python :: 3', | ||
'Programming Language :: Python :: 3.4', | ||
'Programming Language :: Python :: 3.5', | ||
'Programming Language :: Python :: 3.6', | ||
'Programming Language :: Python :: Implementation :: CPython', | ||
'Programming Language :: Python :: Implementation :: PyPy', | ||
'Operating System :: OS Independent', | ||
], | ||
) |