sheep is a dead-simple worker process manager.
Contrary to what you may have heard or even expressed yourself, sheep are not stupid. (An Introduction to Sheep Behavior - Richard Cobb).
Writing worker processes that do background processing is a fairly common task.
At the time sheep was created there was no available solution to the requirements.
sheep must be able to:
- Easy to use configuration (sheep uses derpconf);
- Easy to fork (spawning multiple workers with a single command);
- Reliability when running an unit-of-work (must not die even in the face of SEGFAULT);
- Ease-of-use (Implementing a new worker must be really simple).
Without further ado, let's see how to implement and use sheep.
Installing is as easy as:
$ pip install sheep
The class required to run sheep is the Shepherd. All you need to do is subclass it like this:
from sheep import Shepherd
class MyWorker(Shepherd):
def do_work(self):
# do some heavy work
print("Done")
if __name__ == "__main__":
MyWorker.run()Save this in a file called my_worker.py. Then running your new worker is as easy as:
$ python my_worker.py
You can see all the options with --help:
$ python my_worker.py --help
Since sheep uses derpconf, all you need to do is create a file with the configuration keys for your application. We'll call this file my_worker.conf for this example:
# the configuration names MUST be in uppercase,
# otherwise derpconf will ignore them
CONFIG1 = "test"
CONFIG2 = "other"When invoking your Shepherd, just use the --config option:
$ python my_worker.py --config=./my_worker.conf
Then in your Shepherd those become available via the config property:
from sheep import Shepherd
class MyWorker(Shepherd):
def do_work(self):
# do some heavy work
print(self.config.CONFIG1) # prints "test"
print(self.config.CONFIG2) # prints "other"
if __name__ == "__main__":
MyWorker.run()Subclassing derpconf's Config class you can specify defaults for your keys. Just create a config.py file with something like this:
from derpconf.config import Config
Config.define('foo', 'fooval', 'Foo is always a foo', 'FooValues')Now, all that you need to do is tell your Shepherd where he can find your Config class:
from sheep import Shepherd
from config import Config
class MyWorker(Shepherd):
def get_config_class(self):
return Config # just return the class type and
# Shepherd will take care of the rest
def do_work(self):
# do some heavy work
print(self.config.foo) # prints "fooval" even if no config
# file specified or key not found in
# config file
if __name__ == "__main__":
MyWorker.run()Many times you'll need to initialize some service, like connecting to a database. Sheep got you covered. Just override the initialize method in your Shepherd:
from sheep import Shepherd
from db import connect
class MyWorker(Shepherd):
def initialize(self):
self.db = connect(self.config.CONNECTION_STRING)
def do_work(self):
# do some heavy work
self.db.execute('SELECT baz FROM foo') # or something like it
if __name__ == "__main__":
MyWorker.run()Since initialize is called in the end of your Shepherd's constructor, the configuration keys will be available to you already.
You should override the get_description method in your Shepherd in order to get better logging messages when running your worker.
Just return whatever text you want to be used as the name of your parent worker.
from sheep import Shepherd
class MyWorker(Shepherd):
def get_description(self):
return "My Funky Worker v1.0.0"
def do_work(self):
# do some heavy work
print("Done")
if __name__ == "__main__":
MyWorker.run()As with the arguments you can already pass to your Shepherd, you can add more of your own. Just override the config_parser method:
from sheep import Shepherd
class MyWorker(Shepherd):
def config_parser(self, parser):
# parser is an instance of argparse.ArgumentParser
# just use the regular argparse configuration methods
parser.add_argument('--foo', help='foo help')
def do_work(self):
# do some heavy work
print("Done")
if __name__ == "__main__":
MyWorker.run()Now my_worker.py can be called with --foo:
$ python my_worker.py --foo
If your do_work method raises any exceptions, your Shepherd will log it to the standard output and run do_work again. Sheep won't let your worker die in the face of an exception.
"What if it gets a SEGFAULT error while doing its work?", you'll ask. Well, we are prepared for that. If the fork that's running your work dies, sheep will revive it and start running it again.
Assuming it's a trappable error (not a SEGFAULT), you can use the handle_error method in your Shepherd class to do something with the error, like logging it to an error collector like Sentry.
from sheep import Shepherd
class MyWorker(Shepherd):
def handle_error(self, exc_type, exc_value, tb):
# do something with the exception information
pass
def do_work(self):
# do some heavy work
raise RuntimeError('woot?')
if __name__ == "__main__":
MyWorker.run()Fork, branch, pull request. Same ol', same ol'.
Look at the LICENSE file.