-
Notifications
You must be signed in to change notification settings - Fork 56
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Dask Integration [FEATURE] #101
Comments
We are successfully using Dask with Pygmo2 in our project Pyxel (https://gitlab.com/esa/pyxel and https://esa.gitlab.io/pyxel/). We have developed our own user-defined BFE (Batch Fitness Evaluator) and user-defined Island using Dask. Since January our project is open-source (MIT License), you can find these user-defined BFE and Island here https://gitlab.com/esa/pyxel/-/blob/master/pyxel/calibration/user_defined.py This code could/should be integrated in Pygmo2. What do Pygmo contributor think ? |
We would certainly welcome PRs in this sense :) |
Nice ! I will create a Pull Request ! |
I am successfully using Pygmo2 on my local PC (single machine) and am very satisfied with the parallel optimization performance. As part of my master’s thesis, I intend to conduct parallel optimizations using Pygmo2 on the university's HPC (multi machine). I have explored the Dask integration / extension in Pygmo2 as described in Pyxel. I have a general understanding of the process, but I still have various difficulties with the implementation in my code. @flemmel and @bluescarni
@bluescarni
Thanks for your help! |
No concrete plans at the moment.
Dask integration would mean implementing a user-defined island that distributes the evolutions via Dask. We have several user-defined islands implemented in pygmo already: https://github.com/esa/pygmo2/blob/master/pygmo/_py_islands.py See also the island documentation for information on the API that a user-defined island needs to implement: https://esa.github.io/pygmo2/island.html
We have an ipyparallel island which can be used on HPC setups: https://esa.github.io/pygmo2/islands.html#pygmo.ipyparallel_island We don't have however much experience/user feedback regarding HPC deployments... |
Thank you for the prompt response. I will take a closer look at the concepts you mentioned. I will reach out again if I have any new insights regarding the HPC deployment. However, unfortunately, it no longer has the highest priority in my thesis. |
I want to upvote a But the competing frameworks |
Really good, I will try to add a ‘Dask’ island and a ‘Dask’ batch fitness evaluator for end of August ! |
I recently came across this library in a technical talk. I frequently use the Dask parallel processing engine to scale my work across multiple machines. Is there any plans to expand the multiprocessing tasking to leverage something like Dask or MPI for cluster based optimization?
The text was updated successfully, but these errors were encountered: