Skip to content
Browse files

Updated README, and order of imported path.

  • Loading branch information...
1 parent e001bc8 commit 8ad7682d76b9c1b212a95103660b01abdf4aa246 Dan Lecocq committed
Showing with 116 additions and 62 deletions.
  1. +115 −61 README.md
  2. +1 −1 bin/qless-py-worker
View
176 README.md
@@ -1,61 +1,58 @@
qless-py
========
-Python bindings for [`qless`](https://github.com/seomoz/qless).
+Python bindings for [`qless`](https://github.com/seomoz/qless). Qless is a work
+queue, based on Redis, and inspired by Resque, but with several key differences:
+
+1. __Jobs can't get dropped__ -- jobs have to be checked back in as completed, and
+ long jobs can heartbeat, to ensure that they don't get quietly dropped on
+ the floor by a worker
+1. __Stats__ -- qless automatically keeps summary statistics about how long jobs wait,
+ how long they take to run, etc.
+1. __Scheduling__ -- qless supports scheduling jobs right out of the box
+1. __Dependendies__ -- jobs can wait for another job(s) to complete before being
+ worked on
+1. __History__ -- each job knows everything that's happend to it. From when it was
+ first enqueued, to when it was popped, failed, and completed
+1. __Priority__ -- jobs aren't restricted to priorities of 'high', 'medium' and 'low',
+ but any integer. Jobs with the same priority are popped off in the order
+ they were inserted
+1. __Tagging and Tracking__ -- jobs can be tagged in a searchable way, and flagged for
+ tracking, making it easy to keep tabs on important jobs (like jobs mentioned
+ in bug reports, for example).
+
+With all these differences, it's like Resque in that it comes bundled with a web
+app (available in the `qless` gem), and is high-performance.
+
+Interest piqued? Then read on!
Installation
============
-For qless, you'll need redis (and optionally hireds). Then qless can be installed
-from this repo with:
+You can install qless-py from source by checking it out from github, and checking out
+the qless-core submodule:
- # Install hiredis, redis
- sudo pip install hiredis redis
- sudo setup.py install
+ git clone git://github.com/seomoz/qless-py.git
+ cd qless-py
+ # qless-core is a submodule
+ git submodule init
+ git submodule update
+ sudo python setup.py install
-Lua
----
-
-Qless is a set of client language bindings, but the majority of the work is done in
-a collection of Lua scripts that comprise the [core](https://github.com/seomoz/qless-core)
-functionality. These scripts run on the Redis 2.6+ server atomically and allow for
-portability with the same functionality guarantees. Consule the documentation for
-`qless-core` to learn more about its internals.
-
-Web Interface
--------------
-
-`Qless` also comes with a web app for administrative tasks, like keeping tabs on the
-progress of jobs, tracking specific jobs, retrying failed jobs, etc. It's available
-in the [`qless`](https://github.com/seomoz/qless) library as a mountable
-[`Sinatra`](http://www.sinatrarb.com/) app. The web app is language agnostic and was
-one of the major desires out of this project, so you should consider using it even
-if you're not planning on using the Ruby client.
-
-Concepts and Philosphy
-======================
-
-Jobs are units of work that can be placed in queues. Jobs keep track of the history of
-put / pop / fail events, as well as workers that have worked on a job. A job can appear
-in only one queue at a time, and have a type and a JSON blob of user data associated
-with it.
-
-Workers can pop a job, and they get an exclusive lock on that job for a limited time.
-This lock can be renewed by heartbeating the job to assure qless that the worker has
-not disappeared and is indeed still working on it. The maximum allowable time between
-heartbeats is configurable.
-
-Usage
-=====
+Business Time!
+==============
+You've read this far -- you probably want to write some code now and turn them into jobs.
Jobs are described essentially by two pieces of information -- a `class` and `data`.
The class should have static methods that know how to process this type of job depending
-on the queue it's in:
+on the queue it's in. For those thrown for a loop by this example, it's in refrence to
+a [South Park](http://en.wikipedia.org/wiki/Gnomes_(South_Park\)) episode with a group of
+enterprising gnomes set on world domination through three steps: 1) collect underpants,
+2) ? 3) profit!
# In gnomes.py
class GnomesJob(object):
- # This would be invoked when a GnomesJob is popped off
- # the 'underpants' queue
+ # This would be invoked when a GnomesJob is popped off the 'underpants' queue
@staticmethod
def underpants(job):
# 1) Collect Underpants
@@ -115,8 +112,7 @@ accessible through `__getitem__` and `__setitem__`. Otherwise, it's accessible t
if job['collected'] ...:
...
-Great! With all this in place, you'd probably like to actually add some jobs now.
-First, you need to instantiate a qless client:
+Great! With all this in place, let's put them in the queue so that they can get run
import qless
# Connecting to localhost on 6379
@@ -137,7 +133,7 @@ create a job class in an interactive prompt, for example. You can _add_ jobs in
prompt, but just can't define new job types.
Running
--------
+=======
All that remains is to have workers actually run these jobs. This distribution comes with a
script to help with this:
@@ -184,15 +180,44 @@ particular working directory as the base,
which would yield sandboxes `/home/foo/awesome-project/qless-py-workers/sandbox-<k>`.
+Gevent
+------
+Some jobs are I/O-bound, and might want to, say, make use of a greenlet pool. If you have
+a class where you've, say, monkey-patched `socket`, you can ask qless to create a pool of
+greenlets to run you job inside each process. To run 5 processes with 50 greenlets each:
+
+ qless-py-worker --workers 5 --greenlets 50
+
+Debugging / Developing
+======================
+Whenever a job is processed, it checks to see if the file in which your job is defined has
+been updated since its last import. If it has, it automatically reimports it. We think of
+this as a feature.
+
+With this in mind, when I start a new project and want to make use of qless, I first start
+up the web app locally (see [`qless`](http://github.com/seomoz/qless) for more), take a
+first pass, and enqueue a single job while the worker is running:
+
+ # Supposing that I have /my/awesome/project/awesomeproject.py
+ # In one terminal...
+ qless-py-worker --path /my/awesome/project --queue foo --workers 1 --interval 10 --verbose
+
+ # In another terminal...
+ >>> import qless
+ >>> import awesomeproject
+ >>> qless.client().queue('foo').put(awesomeproject.Job, {'key': 'value'))
+
+From there, I watch the output on the worker, adjust my job class, save it, watch again, etc.,
+but __without restarting the worker__ -- in general it shouldn't be necessary to restart the
+worker.
+
Internals and Additional Features
=================================
-
While in many cases the above is sufficient, there are also many cases where you may need
something more. Hopefully after this section many of your questions will be answered.
Priority
---------
-
+========
Jobs can optionally have priority associated with them. Jobs of equal priority are popped
in the order in which they were put in a queue. The lower the priority, the sooner it will
be processed (it's sort of like `nice`ness). If, for example, you get a new job to collect
@@ -201,8 +226,7 @@ some really valuable underpants, then:
queue.put(qless.gnomes.GnomesJob, {'address': '123 Brief St.'}, priority = -100)
Tags
-----
-
+====
Jobs can have string tags associated with them. Currently, they're justs a piece of metadata
that's associated with each job, but in the future, these will likely be indexed for quick
access.
@@ -210,8 +234,7 @@ access.
queue.put(qless.gnomes.GnomesJob, {}, tags=['tidy', 'white', 'briefs'])
Delay
------
-
+=====
Jobs can also be scheduled for the future with a delay (in seconds). If for example, you just
learned of an underpants heist opportunity, but you have to wait until later:
@@ -225,8 +248,7 @@ expires, you could also boost its priority:
queue.put(qless.gnomes.GnomesJob, {}, delay=3600, priority=-1000)
Retries
--------
-
+=======
Workers sometimes die. That's an unfortunate reality of life. We try to mitigate the effects of
this by insisting that workers heartbeat their jobs to ensure that they do not get dropped. That
said, qless will automatically requeue jobs that do get 'stalled' up to the provided number of
@@ -236,8 +258,7 @@ a particular heist several times:
queue.put(qless.gnomes.GnomesJob, {}, retries=10)
Pop
----
-
+===
A client pops one or more jobs from a queue:
# Get a single job
@@ -246,8 +267,7 @@ A client pops one or more jobs from a queue:
jobs = queue.pop(20)
Heartbeating
-------------
-
+============
Each job object has a notion of when you must either check in with a heartbeat or
turn it in as completed. You can get the absolute time until it expires, or how
long you have left:
@@ -269,8 +289,7 @@ you are done, you should complete it so that the job can move on:
job.complete('anotherQueue')
Stats
------
-
+=====
One of the selling points of qless is that it keeps stats for you about your
underpants hijinks. It tracks the average wait time, number of jobs that have
waited in a queue, failures, retries, and average running time. It also keeps
@@ -279,6 +298,41 @@ that took _x_ time to run.
Frankly, these are best viewed using the web app.
+Web App
+=======
+
+`Qless` also comes with a web app for administrative tasks, like keeping tabs on the
+progress of jobs, tracking specific jobs, retrying failed jobs, etc. It's available
+in the [`qless`](https://github.com/seomoz/qless) library as a mountable
+[`Sinatra`](http://www.sinatrarb.com/) app. The web app is language agnostic and was
+one of the major desires out of this project, so you should consider using it even
+if you're not planning on using the Ruby client.
+
+Internals
+=========
+
+Lua
+---
+
+Qless is a set of client language bindings, but the majority of the work is done in
+a collection of Lua scripts that comprise the [core](https://github.com/seomoz/qless-core)
+functionality. These scripts run on the Redis 2.6+ server atomically and allow for
+portability with the same functionality guarantees. Consult the documentation for
+`qless-core` to learn more about its internals.
+
+Concepts and Philosphy
+======================
+
+Jobs are units of work that can be placed in queues. Jobs keep track of the history of
+put / pop / fail events, as well as workers that have worked on a job. A job can appear
+in only one queue at a time, and have a type and a JSON blob of user data associated
+with it.
+
+Workers can pop a job, and they get an exclusive lock on that job for a limited time.
+This lock can be renewed by heartbeating the job to assure qless that the worker has
+not disappeared and is indeed still working on it. The maximum allowable time between
+heartbeats is configurable.
+
Configuration
=============
View
2 bin/qless-py-worker
@@ -37,7 +37,7 @@ import qless
from qless import logger
# Now let's add each of the paths to the python search path
-sys.path.extend([os.path.abspath(p) for p in args.paths])
+sys.path = [os.path.abspath(p) for p in args.paths] + sys.path
if args.verbose:
import logging

0 comments on commit 8ad7682

Please sign in to comment.
Something went wrong with that request. Please try again.