distribution.rst

Using Multiple Processors

This section of the tutorial shows all the work that is needed to paralellize operations in deap.

Distributed Task Manager

Distributing tasks on multiple computers is taken care of by the distributed task manager module :mod:`~deap.dtm`. Its API similar to the multiprocessing module allows it to be very easy to use. In the :ref:`last section <using-tools>` a complete algorithm was exposed with the :func:`toolbox.map` left to the default :func:`map`. In order to parallelize the evaluation the operation to do is to replace this map with the one provided by the dtm module and tell to dtm which function is the main program here it is the :func:`main` function.

from deap import dtm

toolbox.register("map", dtm.map)

def main():
    # My evolutionary algorithm
    pass

if __name__ == "__main__":
    dtm.start(main)

That's it. The map operation contained in the toolbox will now be parallel. The next time you run the algorithm, it will run on the number of cores specified to the mpirun command used to run the python script. The usual bash command to use dtm will be :

$ mpirun [options] python my_script.py

Multiprocessing Module

Using the :mod:`multiprocessing` module is exactly similar to using the distributed task manager. The only operation to do is to replace in the toolbox the appropriate function by the parallel one.

import multiprocessing

pool = multiprocessing.pool()
toolbox.register("map", pool.map)

# Continue on with the evolutionary algorithm

Note

While Python 2.6 is required for the multiprocessing module, the pickling of partial function is possible only since Python 2.7 (or 3.1), earlier version of Python may throw some strange errors when using partial function in the multiprocessing :func:`multiprocessing.Pool.map`. This may be avoided by creating local function outside of the toolbox (in Python version 2.6).

Note

The pickling of lambda function is not yet available in Python.