Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Archipelago initialisation in same thread, without parallelisation #135

Closed
mghijs opened this issue Oct 16, 2017 · 4 comments
Closed

Archipelago initialisation in same thread, without parallelisation #135

mghijs opened this issue Oct 16, 2017 · 4 comments
Assignees

Comments

@mghijs
Copy link

mghijs commented Oct 16, 2017

An issue in pagmo 1.x.x already referenced this issue.
Are there any plans to create a parallel initialisation?
(I am using Python 2, and a pygmo.ipyparallel_island as user-defined island)

@bluescarni
Copy link
Member

It's planned, but we don't have a timeline for it yet.

@Argysh
Copy link

Argysh commented Jul 29, 2019

meanwhile something along these lines works good enough for me:

import multiprocessing as mp
import pygmo as pg


def pop_init(nPop, seed):
    prob_def = YourProblem()
    prob = pg.problem(prob_def)
    return pg.population(prob, nPop, seed=seed)


if __name__ == "__main__":
    # since pygmo does not use multi threading for the initialisation and the initialisation calls fitness
    # this may take forever if done on a single thread. By initialising the populations in a separate mp.pool
    # and adding them to the archipelago afterwards this can be compensated
    nPop     = 16
    nGen     = 8
    nWorker  = 32
    nIslands = 32

    # create all populations
    with mp.Pool(nWorker) as pool:
        populations = pool.starmap(pop_init, [(nPop, seed) for seed, nPop in enumerate([nPop] * nIslands)])

    # add them to new islands in the otherwise empty archipelago
    archipelago = pg.archipelago()
    for pop in populations:
        archipelago.push_back(algo=pg.algorithm(pg.sade(nGen)), pop=pop, udi=pg.mp_island())
    archipelago.wait()

    # 'the work' must continue!

@bluescarni
Copy link
Member

bluescarni commented Jul 29, 2019

There is now something we call a "batch fitness evaluation" scheme in pagmo which is used, among other things, to parallel init populations, islands and archipelagos in C++.

We are currently lacking the capability in Python, but it is not difficult to implement and all the necessary pieces are there. The code won't be long, and it will be very similar to the workarounds @Argysh and other have posted in the past.

We will get there with Python batch fitness evaluators, it's just a matter of manpower as usual, and we are busy with many other things in pagmo at this particular time (migration, topology, new algorithms, etc.).

If anyone wants to take the lead, this should be a good entry-level contribution to pagmo, and we would be willing to offer all the assistance needed. A good starting point would be the documentation of the bfe class in Python:

https://esa.github.io/pagmo2/docs/python/py_bfe.html

As usual, we are available on gitter for discussing possible contributions and answering questions in (almost!) real time:

https://gitter.im/pagmo2/Lobby

@bluescarni
Copy link
Member

The batch fitness evaluation framework has now been completed on the Python side with the release of pagmo 2.13. I will close this report.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants