Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
Actor model based parallel processing with python multiprocessing
Python
branch: master

Fetching latest commit…

Cannot retrieve the latest commit at this time

Failed to load latest commit information.
swarm
test
.gitignore
AUTHOR
COPYING
LICENSE
MANIFEST.in
README.rst
TODO
URL
VERSION
setup.py

README.rst

DESCRIPTION

Parallel processing for humans

OUTLINE

multiprocessing based framework that uses codeblocks.py atomization layer.

Main area of swarm application is web crawlers, mass parsers or email senders with complex internal behaviour.

EXAMPLE

import logging
import sys
import time
import random
from swarm import BaseSwarm as Swarm, define_swarm, atomic, swarm

class config:
    # number of pool workers. Defaults to `multiprocessing.cpu_count()`
    COMPETITORS = 10
    # just set log level and and enjoy
    # `swarm.log.[debug|info|error]('Message')` inside pool workers
    LOG_LEVEL = logging.INFO

calculator = Swarm(__name__)
calculator.config.from_object(config)

define_swarm.start()
@calculator.sync
def squares_of_range(x):
    #current_process() == <MainProcess>
    for i in range(1, x):
        #code inside `atomic()` context pushed into pool asyncronously
        with atomic() << _:
            # current_process() == <PoolWorker-X>

            # transparently takes `i` variable from outer world
            # and sends all generated items to items queue of top @sync function
            yield i*i

            # internally required modules are also imported into PoolWorker
            time.sleep(round(random.random(),3))

            #message that exceeds log level does not leave process bounds
            swarm.log.debug('Calculated result for %s'%i)

            if i > 3:
                swarm.log.error('Oh no! %s is too much for me!'%i)
    #this function will only finish after all `atomic()`s finished
define_swarm.finish()

if __name__ == '__main__':
    for x in squares_of_range(int(sys.argv[1])):
        print '>', x

FEATURES

  • nested atomic()-s
  • recursion of atomic() calling functions
  • transparent serialization of closures
  • pause/restore
  • explicit namespace management with codeblocks.py namespace management
  • werkzeug like context stacks. Manager and atom level context stacks
  • multiprocessing as parallelization backend
  • wide use of blinker signals for infill customization

WARNING

Currently this is alpha version.

Something went wrong with that request. Please try again.