Skip to content

Commit

Permalink
Updated README, and order of imported path.
Browse files Browse the repository at this point in the history
  • Loading branch information
Dan Lecocq committed Apr 29, 2012
1 parent e001bc8 commit 8ad7682
Show file tree
Hide file tree
Showing 2 changed files with 116 additions and 62 deletions.
176 changes: 115 additions & 61 deletions README.md
View file
Original file line number Original file line Diff line number Diff line change
@@ -1,61 +1,58 @@
qless-py qless-py
======== ========


Python bindings for [`qless`](https://github.com/seomoz/qless). Python bindings for [`qless`](https://github.com/seomoz/qless). Qless is a work
queue, based on Redis, and inspired by Resque, but with several key differences:

1. __Jobs can't get dropped__ -- jobs have to be checked back in as completed, and
long jobs can heartbeat, to ensure that they don't get quietly dropped on
the floor by a worker
1. __Stats__ -- qless automatically keeps summary statistics about how long jobs wait,
how long they take to run, etc.
1. __Scheduling__ -- qless supports scheduling jobs right out of the box
1. __Dependendies__ -- jobs can wait for another job(s) to complete before being
worked on
1. __History__ -- each job knows everything that's happend to it. From when it was
first enqueued, to when it was popped, failed, and completed
1. __Priority__ -- jobs aren't restricted to priorities of 'high', 'medium' and 'low',
but any integer. Jobs with the same priority are popped off in the order
they were inserted
1. __Tagging and Tracking__ -- jobs can be tagged in a searchable way, and flagged for
tracking, making it easy to keep tabs on important jobs (like jobs mentioned
in bug reports, for example).

With all these differences, it's like Resque in that it comes bundled with a web
app (available in the `qless` gem), and is high-performance.

Interest piqued? Then read on!


Installation Installation
============ ============


For qless, you'll need redis (and optionally hireds). Then qless can be installed You can install qless-py from source by checking it out from github, and checking out
from this repo with: the qless-core submodule:


# Install hiredis, redis git clone git://github.com/seomoz/qless-py.git
sudo pip install hiredis redis cd qless-py
sudo setup.py install # qless-core is a submodule
git submodule init
git submodule update
sudo python setup.py install


Lua Business Time!
--- ==============

Qless is a set of client language bindings, but the majority of the work is done in
a collection of Lua scripts that comprise the [core](https://github.com/seomoz/qless-core)
functionality. These scripts run on the Redis 2.6+ server atomically and allow for
portability with the same functionality guarantees. Consule the documentation for
`qless-core` to learn more about its internals.

Web Interface
-------------

`Qless` also comes with a web app for administrative tasks, like keeping tabs on the
progress of jobs, tracking specific jobs, retrying failed jobs, etc. It's available
in the [`qless`](https://github.com/seomoz/qless) library as a mountable
[`Sinatra`](http://www.sinatrarb.com/) app. The web app is language agnostic and was
one of the major desires out of this project, so you should consider using it even
if you're not planning on using the Ruby client.

Concepts and Philosphy
======================

Jobs are units of work that can be placed in queues. Jobs keep track of the history of
put / pop / fail events, as well as workers that have worked on a job. A job can appear
in only one queue at a time, and have a type and a JSON blob of user data associated
with it.

Workers can pop a job, and they get an exclusive lock on that job for a limited time.
This lock can be renewed by heartbeating the job to assure qless that the worker has
not disappeared and is indeed still working on it. The maximum allowable time between
heartbeats is configurable.

Usage
=====


You've read this far -- you probably want to write some code now and turn them into jobs.
Jobs are described essentially by two pieces of information -- a `class` and `data`. Jobs are described essentially by two pieces of information -- a `class` and `data`.
The class should have static methods that know how to process this type of job depending The class should have static methods that know how to process this type of job depending
on the queue it's in: on the queue it's in. For those thrown for a loop by this example, it's in refrence to
a [South Park](http://en.wikipedia.org/wiki/Gnomes_(South_Park\)) episode with a group of
enterprising gnomes set on world domination through three steps: 1) collect underpants,
2) ? 3) profit!


# In gnomes.py # In gnomes.py
class GnomesJob(object): class GnomesJob(object):
# This would be invoked when a GnomesJob is popped off # This would be invoked when a GnomesJob is popped off the 'underpants' queue
# the 'underpants' queue
@staticmethod @staticmethod
def underpants(job): def underpants(job):
# 1) Collect Underpants # 1) Collect Underpants
Expand Down Expand Up @@ -115,8 +112,7 @@ accessible through `__getitem__` and `__setitem__`. Otherwise, it's accessible t
if job['collected'] ...: if job['collected'] ...:
... ...


Great! With all this in place, you'd probably like to actually add some jobs now. Great! With all this in place, let's put them in the queue so that they can get run
First, you need to instantiate a qless client:


import qless import qless
# Connecting to localhost on 6379 # Connecting to localhost on 6379
Expand All @@ -137,7 +133,7 @@ create a job class in an interactive prompt, for example. You can _add_ jobs in
prompt, but just can't define new job types. prompt, but just can't define new job types.


Running Running
------- =======
All that remains is to have workers actually run these jobs. This distribution comes with a All that remains is to have workers actually run these jobs. This distribution comes with a
script to help with this: script to help with this:


Expand Down Expand Up @@ -184,15 +180,44 @@ particular working directory as the base,


which would yield sandboxes `/home/foo/awesome-project/qless-py-workers/sandbox-<k>`. which would yield sandboxes `/home/foo/awesome-project/qless-py-workers/sandbox-<k>`.


Gevent
------
Some jobs are I/O-bound, and might want to, say, make use of a greenlet pool. If you have
a class where you've, say, monkey-patched `socket`, you can ask qless to create a pool of
greenlets to run you job inside each process. To run 5 processes with 50 greenlets each:

qless-py-worker --workers 5 --greenlets 50

Debugging / Developing
======================
Whenever a job is processed, it checks to see if the file in which your job is defined has
been updated since its last import. If it has, it automatically reimports it. We think of
this as a feature.

With this in mind, when I start a new project and want to make use of qless, I first start
up the web app locally (see [`qless`](http://github.com/seomoz/qless) for more), take a
first pass, and enqueue a single job while the worker is running:

# Supposing that I have /my/awesome/project/awesomeproject.py
# In one terminal...
qless-py-worker --path /my/awesome/project --queue foo --workers 1 --interval 10 --verbose

# In another terminal...
>>> import qless
>>> import awesomeproject
>>> qless.client().queue('foo').put(awesomeproject.Job, {'key': 'value'))

From there, I watch the output on the worker, adjust my job class, save it, watch again, etc.,
but __without restarting the worker__ -- in general it shouldn't be necessary to restart the
worker.

Internals and Additional Features Internals and Additional Features
================================= =================================

While in many cases the above is sufficient, there are also many cases where you may need While in many cases the above is sufficient, there are also many cases where you may need
something more. Hopefully after this section many of your questions will be answered. something more. Hopefully after this section many of your questions will be answered.


Priority Priority
-------- ========

Jobs can optionally have priority associated with them. Jobs of equal priority are popped Jobs can optionally have priority associated with them. Jobs of equal priority are popped
in the order in which they were put in a queue. The lower the priority, the sooner it will in the order in which they were put in a queue. The lower the priority, the sooner it will
be processed (it's sort of like `nice`ness). If, for example, you get a new job to collect be processed (it's sort of like `nice`ness). If, for example, you get a new job to collect
Expand All @@ -201,17 +226,15 @@ some really valuable underpants, then:
queue.put(qless.gnomes.GnomesJob, {'address': '123 Brief St.'}, priority = -100) queue.put(qless.gnomes.GnomesJob, {'address': '123 Brief St.'}, priority = -100)


Tags Tags
---- ====

Jobs can have string tags associated with them. Currently, they're justs a piece of metadata Jobs can have string tags associated with them. Currently, they're justs a piece of metadata
that's associated with each job, but in the future, these will likely be indexed for quick that's associated with each job, but in the future, these will likely be indexed for quick
access. access.


queue.put(qless.gnomes.GnomesJob, {}, tags=['tidy', 'white', 'briefs']) queue.put(qless.gnomes.GnomesJob, {}, tags=['tidy', 'white', 'briefs'])


Delay Delay
----- =====

Jobs can also be scheduled for the future with a delay (in seconds). If for example, you just Jobs can also be scheduled for the future with a delay (in seconds). If for example, you just
learned of an underpants heist opportunity, but you have to wait until later: learned of an underpants heist opportunity, but you have to wait until later:


Expand All @@ -225,8 +248,7 @@ expires, you could also boost its priority:
queue.put(qless.gnomes.GnomesJob, {}, delay=3600, priority=-1000) queue.put(qless.gnomes.GnomesJob, {}, delay=3600, priority=-1000)


Retries Retries
------- =======

Workers sometimes die. That's an unfortunate reality of life. We try to mitigate the effects of Workers sometimes die. That's an unfortunate reality of life. We try to mitigate the effects of
this by insisting that workers heartbeat their jobs to ensure that they do not get dropped. That this by insisting that workers heartbeat their jobs to ensure that they do not get dropped. That
said, qless will automatically requeue jobs that do get 'stalled' up to the provided number of said, qless will automatically requeue jobs that do get 'stalled' up to the provided number of
Expand All @@ -236,8 +258,7 @@ a particular heist several times:
queue.put(qless.gnomes.GnomesJob, {}, retries=10) queue.put(qless.gnomes.GnomesJob, {}, retries=10)


Pop Pop
--- ===

A client pops one or more jobs from a queue: A client pops one or more jobs from a queue:


# Get a single job # Get a single job
Expand All @@ -246,8 +267,7 @@ A client pops one or more jobs from a queue:
jobs = queue.pop(20) jobs = queue.pop(20)


Heartbeating Heartbeating
------------ ============

Each job object has a notion of when you must either check in with a heartbeat or Each job object has a notion of when you must either check in with a heartbeat or
turn it in as completed. You can get the absolute time until it expires, or how turn it in as completed. You can get the absolute time until it expires, or how
long you have left: long you have left:
Expand All @@ -269,8 +289,7 @@ you are done, you should complete it so that the job can move on:
job.complete('anotherQueue') job.complete('anotherQueue')


Stats Stats
----- =====

One of the selling points of qless is that it keeps stats for you about your One of the selling points of qless is that it keeps stats for you about your
underpants hijinks. It tracks the average wait time, number of jobs that have underpants hijinks. It tracks the average wait time, number of jobs that have
waited in a queue, failures, retries, and average running time. It also keeps waited in a queue, failures, retries, and average running time. It also keeps
Expand All @@ -279,6 +298,41 @@ that took _x_ time to run.


Frankly, these are best viewed using the web app. Frankly, these are best viewed using the web app.


Web App
=======

`Qless` also comes with a web app for administrative tasks, like keeping tabs on the
progress of jobs, tracking specific jobs, retrying failed jobs, etc. It's available
in the [`qless`](https://github.com/seomoz/qless) library as a mountable
[`Sinatra`](http://www.sinatrarb.com/) app. The web app is language agnostic and was
one of the major desires out of this project, so you should consider using it even
if you're not planning on using the Ruby client.

Internals
=========

Lua
---

Qless is a set of client language bindings, but the majority of the work is done in
a collection of Lua scripts that comprise the [core](https://github.com/seomoz/qless-core)
functionality. These scripts run on the Redis 2.6+ server atomically and allow for
portability with the same functionality guarantees. Consult the documentation for
`qless-core` to learn more about its internals.

Concepts and Philosphy
======================

Jobs are units of work that can be placed in queues. Jobs keep track of the history of
put / pop / fail events, as well as workers that have worked on a job. A job can appear
in only one queue at a time, and have a type and a JSON blob of user data associated
with it.

Workers can pop a job, and they get an exclusive lock on that job for a limited time.
This lock can be renewed by heartbeating the job to assure qless that the worker has
not disappeared and is indeed still working on it. The maximum allowable time between
heartbeats is configurable.

Configuration Configuration
============= =============


Expand Down
2 changes: 1 addition & 1 deletion bin/qless-py-worker
View file
Original file line number Original file line Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ import qless
from qless import logger from qless import logger


# Now let's add each of the paths to the python search path # Now let's add each of the paths to the python search path
sys.path.extend([os.path.abspath(p) for p in args.paths]) sys.path = [os.path.abspath(p) for p in args.paths] + sys.path


if args.verbose: if args.verbose:
import logging import logging
Expand Down

0 comments on commit 8ad7682

Please sign in to comment.