Python library for running asynchronous background jobs using Twisted
Python Makefile
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
db
examples
txque
.gitignore
Makefile
README.markdown
setup.py

README.markdown

txque: Python Async Background Jobs

This project allows you to run background jobs asynchronously much like Ruby's Sidekiq. It uses Python Twisted to handle thread pooling and all of the magic behind the scenes.

It can currently use the DBAPI modules MySQLdb, psycopg2, and sqlite3 - though new storage methods are easy to add via the dispatcher interface.

Installation

pip install txque

You'll need to set up a table in your DB for txque to store jobs. There are files in the db folder to initialize the table needed by txque. Make sure you run that SQL for your DB first before continuing.

Usage

Usage is pretty straightforward. To create a job definition, simply extend the Job class and define a "run" method.

from txque.work import Job

class MyBackgroundJob(Job):
    def run(self, anArgument, akeyword=avalue):
        # Now do some work.  This work can be synchronous, or it can return a deferred.
        someExpensiveFunction(anArgument, akeyword)

Then, you can queue a job anywhere you need to:

from twisted.internet import reactor
from txque.dispatchers.sql import MySQLDispatcher

# create a dispatcher
dispatcher = MySQLDispatcher(user="username", passwd="password", db="txque")

# queue the job, then stop the reactor once queued
d = dispatcher.queue(MyBackgroundJob('hi there'), queue='default', priority=10)
d.addCallback(lambda _: reactor.stop())

# start the reactor to start everything
reactor.run()

Then, you can run as many workers as you'd like. A sample worker application has been included in the examples folder (you'll want to edit it first before use to set your database config options). Running it is simple:

twistd -noy worker.tac

You can specify the queue and priority as well:

TXQUE_QUENAME=myque TXQUE_PRIORITY=5 twistd -noy worker.tac

Errors / Bugs / Contact

See github.