Little process-based workers to do your bidding.
Pull request Compare This branch is 13 commits behind toastdriven:master.
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.



Little process-based workers to do your bidding.

Deliberately minimalist, you provide the number of workers to use & a list of commands (to be executed at the shell) & littleworkers will eat through the list as fast as it can.

Why littleworkers?

littleworkers shines when you just want to parallelize something without a lot of fuss & when you care more about the data/commands to be run.

  • Tiny source
  • Easy to queue a set of actions
  • Works with any runnable commands
  • Uses processes
  • Non-blocking

Seriously, it's not a replacement for threading or multiprocessing if your application needs to share a ton of data with the children.


Usage is trivial:

from littleworkers import Pool

# Define your commands.
commands = [
    'ls -al',
    'cd /tmp && mkdir foo',
    'echo "Hello There."',
    'sleep 2 && echo "Done."'

# Setup a pool. Since I have two cores, I'll use two workers.
lil = Pool(workers=2)

# Run!

You want the stdout back:

import subprocess
from littleworkers import Pool

class MyPool(Pool):
    def __init__(self, *args, **kwargs):
        super(MyPool, self).__init__(*args, **kwargs)
        self.collected_output = []

    def create_process(self, command):
        logging.debug("Starting process to handle command '%s'." % command)
        return subprocess.Popen(command, shell=True, stdout=subprocess.PIPE)

    def remove_from_pool(self, pid):
        return super(MyPool, self).remove_from_pool(pid)

You want to use a Queue instead of the default list:

from Queue import Queue, Empty
from littleworkers import Pool

class QueuePool(Pool):
    def __init__(self, *args, **kwargs):
        super(QueuePool, self).__init__(*args, **kwargs)
        self.commands = Queue()

    def prepare_commands(self, commands):
        for command in commands:

    def command_count(self):
        return self.commands.qsize()

    def next_command(self):
            return self.commands.get()
        except Empty:
            return None


  • Python 2.6+
author:Daniel Lindsley