Archipelago initialisation in same thread, without parallelisation

[mghijs]

An issue in pagmo 1.x.x already referenced this issue.
Are there any plans to create a parallel initialisation?
(I am using Python 2, and a pygmo.ipyparallel_island as user-defined island)

[bluescarni]

It's planned, but we don't have a timeline for it yet.

[Argysh]

meanwhile something along these lines works good enough for me:

import multiprocessing as mp
import pygmo as pg


def pop_init(nPop, seed):
    prob_def = YourProblem()
    prob = pg.problem(prob_def)
    return pg.population(prob, nPop, seed=seed)


if __name__ == "__main__":
    # since pygmo does not use multi threading for the initialisation and the initialisation calls fitness
    # this may take forever if done on a single thread. By initialising the populations in a separate mp.pool
    # and adding them to the archipelago afterwards this can be compensated
    nPop     = 16
    nGen     = 8
    nWorker  = 32
    nIslands = 32

    # create all populations
    with mp.Pool(nWorker) as pool:
        populations = pool.starmap(pop_init, [(nPop, seed) for seed, nPop in enumerate([nPop] * nIslands)])

    # add them to new islands in the otherwise empty archipelago
    archipelago = pg.archipelago()
    for pop in populations:
        archipelago.push_back(algo=pg.algorithm(pg.sade(nGen)), pop=pop, udi=pg.mp_island())
    archipelago.wait()

    # 'the work' must continue!

[bluescarni]

There is now something we call a "batch fitness evaluation" scheme in pagmo which is used, among other things, to parallel init populations, islands and archipelagos in C++.

We are currently lacking the capability in Python, but it is not difficult to implement and all the necessary pieces are there. The code won't be long, and it will be very similar to the workarounds @Argysh and other have posted in the past.

We will get there with Python batch fitness evaluators, it's just a matter of manpower as usual, and we are busy with many other things in pagmo at this particular time (migration, topology, new algorithms, etc.).

If anyone wants to take the lead, this should be a good entry-level contribution to pagmo, and we would be willing to offer all the assistance needed. A good starting point would be the documentation of the bfe class in Python:

https://esa.github.io/pagmo2/docs/python/py_bfe.html

As usual, we are available on gitter for discussing possible contributions and answering questions in (almost!) real time:

https://gitter.im/pagmo2/Lobby

[bluescarni]

The batch fitness evaluation framework has now been completed on the Python side with the release of pagmo 2.13. I will close this report.