Clone this wiki locally
Pyres is a Resque clone built in python. Each job in the queue corresponds to a python class that needs to define a
Let’s say I have a blog which needs to check comments for spam. When the comment is saved in the DB we create a job in the queue with comment data. Looks something like.
class Comment(Model): name = Model.CharField() email = Model.EmailField() body = Model.TextField() spam = Model.BooleanField queue = "Spam" # @staticmethod def perform(comment_id): // You ORM or SQL to get the comment data from the comment_id x = urllib.open("http://apikey.rest.akismet.com/1.1/comment-check?comment_author="+self.name.. so forth if x == "true": // save the comment spam field to true. else: // comment is fine.
You can convert your existing class to be compatible with Pyres. All you need to do is add a
queuevariable and define a
performmethod on the class.
To insert a job into the queue you need to do something like this:
from pyres import ResQ r = Resq() r.enqueue(Comment, 23)
This puts a job into the queue
Spam. Now we need to fire off our workers. In the
scriptsfolder there is an executable
pyres_workerwhich is used to start a worker.
$ ./pyres_worker Spam
Just pass a comma seperated list of queues the worker should poll.
Tips and Tricks
While developing locally, it’s uncommon to need to rely on a separate worker process that watches the queues. Instead, it’s better to run the job synchronously. You can do this with a simple superclass that your jobs will inherit from:
from pyres import ResQ r = Resq() class JobBase(object): @classmethod def enqueue(cls, *args): if app.debug: cls.perform(*args) else: r.enqueue(cls, *args) class GenerateThumbnailsJob(JobBase): pass
From then on, you can enqueue jobs one way regardless if you’re in development, testing or production mode:
from jobs import GenerateThumbnailsJob GenerateThumbnailsJob.enqueue(*args)