Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dask Integration [FEATURE] #101

Open
franciscblubaugh opened this issue Jun 16, 2022 · 8 comments
Open

Dask Integration [FEATURE] #101

franciscblubaugh opened this issue Jun 16, 2022 · 8 comments
Labels
enhancement New feature or request

Comments

@franciscblubaugh
Copy link

I recently came across this library in a technical talk. I frequently use the Dask parallel processing engine to scale my work across multiple machines. Is there any plans to expand the multiprocessing tasking to leverage something like Dask or MPI for cluster based optimization?

@franciscblubaugh franciscblubaugh added the enhancement New feature or request label Jun 16, 2022
@flemmel
Copy link

flemmel commented Oct 1, 2022

Hi @franciscblubaugh,

We are successfully using Dask with Pygmo2 in our project Pyxel (https://gitlab.com/esa/pyxel and https://esa.gitlab.io/pyxel/).
It works well in a single computer and a grid of computers (84 cores)

We have developed our own user-defined BFE (Batch Fitness Evaluator) and user-defined Island using Dask.

Since January our project is open-source (MIT License), you can find these user-defined BFE and Island here https://gitlab.com/esa/pyxel/-/blob/master/pyxel/calibration/user_defined.py

This code could/should be integrated in Pygmo2.

What do Pygmo contributor think ?

@bluescarni
Copy link
Member

What do Pygmo contributor think ?

We would certainly welcome PRs in this sense :)

@flemmel
Copy link

flemmel commented Oct 3, 2022

Nice !

I will create a Pull Request !

@IvoSteiner
Copy link

I am successfully using Pygmo2 on my local PC (single machine) and am very satisfied with the parallel optimization performance. As part of my master’s thesis, I intend to conduct parallel optimizations using Pygmo2 on the university's HPC (multi machine). I have explored the Dask integration / extension in Pygmo2 as described in Pyxel. I have a general understanding of the process, but I still have various difficulties with the implementation in my code.

@flemmel and @bluescarni

  • An official Dask integration in Pygmo2 would be highly desirable. Is it still planned?
  • Do you have basic Dask Pygmo2 integration / extension examples other than Pyxel.py itself?

@bluescarni
I am a beginner regarding parallelizing Python code. Based on the Pygmo2 capabilities description, I assumed that the library already runs natively on HPCs (Multi Machine).

  • Are there approaches with less overhead than Dask to execute Pygmo2 on HPCs?
  • How do you work with Pygmo2 on an HPC? Any simple examples are appreciated.

Thanks for your help!

@bluescarni
Copy link
Member

bluescarni commented Dec 4, 2023

* An _official_ Dask integration in Pygmo2 would be highly desirable. Is it still planned?

No concrete plans at the moment.

* Do you have basic Dask Pygmo2 _integration / extension_ examples other than Pyxel.py itself?

Dask integration would mean implementing a user-defined island that distributes the evolutions via Dask. We have several user-defined islands implemented in pygmo already:

https://github.com/esa/pygmo2/blob/master/pygmo/_py_islands.py

See also the island documentation for information on the API that a user-defined island needs to implement:

https://esa.github.io/pygmo2/island.html

* Are there approaches with less overhead than Dask to execute Pygmo2 on HPCs?
* How do you work with Pygmo2 on an HPC? Any simple examples are appreciated.

We have an ipyparallel island which can be used on HPC setups:

https://esa.github.io/pygmo2/islands.html#pygmo.ipyparallel_island

We don't have however much experience/user feedback regarding HPC deployments...

@IvoSteiner
Copy link

Thank you for the prompt response. I will take a closer look at the concepts you mentioned. I will reach out again if I have any new insights regarding the HPC deployment. However, unfortunately, it no longer has the highest priority in my thesis.

@erl987
Copy link

erl987 commented Jul 28, 2024

I want to upvote a Dask integration as a built-in island. It does not seem too hard to try the approaches mentioned in this issue.

But the competing frameworks jmetalpy and pymmo provide Dask support out of the box. For distributed computing a clear advantage.

@flemmel
Copy link

flemmel commented Aug 6, 2024

Really good, I will try to add a ‘Dask’ island and a ‘Dask’ batch fitness evaluator for end of August !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

5 participants