Little process-based workers to do your bidding.
Deliberately minimalist, you provide the number of
workers to use &
a list of commands (to be executed at the shell) &
littleworkers will eat
through the list as fast as it can.
littleworkers shines when you just want to parallelize something without a
lot of fuss & when you care more about the data/commands to be run.
- Tiny source
- Easy to queue a set of actions
- Works with any runnable commands
- Uses processes
Seriously, it's not a replacement for threading or multiprocessing if your application needs to share a ton of data with the children.
Usage is trivial:
from littleworkers import Pool # Define your commands. commands = [ 'ls -al', 'cd /tmp && mkdir foo', 'date', 'echo "Hello There."', 'sleep 2 && echo "Done."' ] # Setup a pool. Since I have two cores, I'll use two workers. lil = Pool(workers=2) # Run! lil.run(commands)
You want the stdout back:
import subprocess from littleworkers import Pool class MyPool(Pool): def __init__(self, *args, **kwargs): super(MyPool, self).__init__(*args, **kwargs) self.collected_output =  def create_process(self, command): logging.debug("Starting process to handle command '%s'." % command) return subprocess.Popen(command, shell=True, stdout=subprocess.PIPE) def remove_from_pool(self, pid): self.collected_output.append(self.pool[pid].stdout.read()) return super(MyPool, self).remove_from_pool(pid)
You want to use a
Queue instead of the default
from Queue import Queue, Empty from littleworkers import Pool class QueuePool(Pool): def __init__(self, *args, **kwargs): super(QueuePool, self).__init__(*args, **kwargs) self.commands = Queue() def prepare_commands(self, commands): for command in commands: self.commands.put(command) def command_count(self): return self.commands.qsize() def next_command(self): try: return self.commands.get() except Empty: return None
- Python 2.6+