Mongo Task Queue (mtq)
mtq is a simple Python library for queueing jobs and processing them in the background with workers. It is backed by Mongodb and it is designed to have a low barrier to entry. It should be integrated in your web stack easily.
First, run a Mongod server:
$ mongod [options]
To put jobs on queues, define a regular python function:
import requests def count_words_at_url(url): """Just an example function that's called async.""" resp = requests.get(url) return len(resp.text.split())
Then, create a MTQ queue:
import mtq conn = mtq.default_connection() q = conn.queue()
And enqueue the function call:
from my_module import count_words_at_url result = q.enqueue(count_words_at_url, 'http://binstar.org')
For a more complete example, refer to the docs. But this is the essence.
To start executing enqueued function calls in the background, start a worker from your project's directory:
$ mtq-worker [info] Starting Main Loop worker=mr_chomps.local.67313 _id=51ffb3dd7d150a06f28b1e11 Got count_words_at_url('http://binstar.org') from default Job result = 818
Simply use the following command to install the latest released version:
pip install mtq