New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve usage under pure dask cluster mode. #535

Merged
merged 12 commits into from Dec 11, 2018

Conversation

Projects
None yet
1 participant
@rainwoodman
Copy link
Member

rainwoodman commented Dec 6, 2018

Combined with several threading fixes in bigfile, this PR improves the support with dask clusters.

Two new functions are added: nbodykit.use_distributed and nbodykit.use_mpi. For dask clusters, initialize using use_distributed. for MPI clustrers, initialize using use_mpi. The MPI cluster mode is the default.

The global cache context manager is removed. The opportunistic caching it is now always enabled in MPI mode (when dask's synchronous scheduler is used); and always disabled in distributed mode, when the distributed scheduler is used.

The pickling trick was from mpi4py's bitbucket issue : https://bitbucket.org/mpi4py/mpi4py/issues/104/#comment-47225639

The singleton initialization needs to be revisited after the distributed PR dask/distributed#2391 is merged.

rainwoodman added some commits Nov 26, 2018

Add use_distributed and use_mpi.
use_distributed will use the distributed model from distributed and
restrict MPI to MPI_COMM_SELF.
Remove GlobalCache context manager in compute.
Always enable the oppurtunity cache, and allow changing the size.

This should work around the threading race condition mentioned at

dask/distributed#2372 (comment)

@rainwoodman rainwoodman requested a review from nickhand Dec 6, 2018

@rainwoodman

This comment has been minimized.

Copy link
Member Author

rainwoodman commented Dec 11, 2018

I need to merge this to use it with another branch..

@rainwoodman rainwoodman merged commit db2610f into master Dec 11, 2018

3 checks passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details
coverage/coveralls Coverage decreased (-0.2%) to 94.703%
Details

@rainwoodman rainwoodman deleted the mpiself branch Dec 11, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment