oilcan is a Python job manager for Gearman. It is intended as an alternative for Celery, when your queuing system of choice is Gearman, and you want something simple.
Concept:
- You use a decorator to mark which python functions can be called as a task.
- You start oilcan, pointing it at a file cointaining tasks. It forks worker sub-processes.
- You send a message to Gearman to trigger one of the jobs.
oilcan is alpha software, not quite ready yet.
To define some tasks you do this (by convention in a tasks.py file):
from oilcan import task
@task
def example_task(self, workload):
# Workload is whatever string the client sent
do_something(workload)
To call that task:
# Setup
from gearman.libgearman import Client
client = Client()
for host, port in [("127.0.0.1", 4730), ("192.168.0.100", 4730)]:
client.add_server(host, port)
# Call
# Workload ('42') must be a string.
client.do_background('example_task', '42')
If do
or do_background
return a response code, it's an index into a gearman_return_t enum. To find out what it means, count into this list, starting from 0: http://gearman.org/docs/api/group__gearman__constants.html#g200f3d324fd4c4bfee16143c8f7b672a
-
Get the dependencies
Ubuntu 10.10 (Maverick) and later
sudo apt-get install gearman-job-server python-gearman.libgearman
Other Linuxes
Make sure you have the dependencies:
sudo apt-get install libevent-dev uuid-dev
The
python-gearman.libgearman
package isn't in earlier version of Ubuntu, and the PyPI version relies on a recent Gearman, so install both:wget http://launchpad.net/gearmand/trunk/0.14/+download/gearmand-0.14.tar.gz tar xvzf gearmand-0.14.tar.gz cd gearmand-0.14 ./configure --disable-libmemcached sudo make install sudo pip install python-libgearman
You might not need to disable libmemcached, but I got three of this type of error if I didn't: gearmand-0.14/gearmand/gearmand.c:193: undefined reference to `gearman_server_queue_libmemcached_conf'.
You might also need to:
sudo apt-get install libgearman2 ; sudo ldconfig
, but don't ask me why. -
If you're running Python 2.6 or earlier you need the argparse package. It's in the standard library for 2.7+:
sudo pip install argparse
-
Get oilcan (cd into a temporary directory first):
git clone git://github.com/grahamking/oilcan.git
-
Copy oilcan.py onto your python path:
sudo cp oilcan.py /usr/local/lib/python2.6/site-packages/
-
Link it from /usr/local/bin/:
sudo ln -s /usr/local/lib/python2.6/site-packages/oilcan.py /usr/local/bin/oilcan
-
Copy the upstart startup script into /etc/init/:
sudo cp oilcan.conf /etc/init/
-
Edit /etc/init/oilcan.conf and make it work for you.
vim /etc/init/oilcan.conf # Or your editor of choice
For help on this run:
/usr/local/bin/oilcan --help
-
Start the worker (Gearman must already be running):
sudo start oilcan
Oilcan ships with some unit tests in the tests file. To run them you need:
- Nose: python-nose in Ubuntu, or nose in PyPI.
- Coverage: python-coverage in Ubuntu, or coverage in PyPI.
Run like this, from the directory that contains tests.py and oilcan.py:
nosetests --with-coverage --cover-package=oilcan tests.py
If there is an error in your tasks.py oilcan will die as soon as it starts. To see what's going on, run oilcan in non-forked debug mode:
export DJANGO_SETTINGS_MODULE=settings # Only needed if using Django
/usr/local/bin/oilcan myapp.tasks --add-path /usr/local/myproj/ --no-fork --debug
If you prefer something other than upstart, simply copy the command from /etc/init/oilcan.conf
. Oilcan doesn't daemonize itself, so it should play nicely with any process manager you care to use.
If you are using MySQL InnoDB, you might get this error:
OperationalError: (1598, "Binary logging not possible. Message: Transaction level 'READ-COMMITTED' in InnoDB is not safe for binlog mode 'STATEMENT'")
Because oilcan runs for a long time, it changes MySQL's transaction isolation mode to READ-COMMITTED, so that it sees changes. InnoDB's binary log needs to be in ROW mode to support this. In /etc/mysql/my.cnf
add or edit this row:
binlog-format = ROW