Join GitHub today
GitHub is home to over 31 million developers working together to host and review code, manage projects, and build software together.Sign up
Improve usage under pure dask cluster mode. #535
Combined with several threading fixes in bigfile, this PR improves the support with dask clusters.
Two new functions are added: nbodykit.use_distributed and nbodykit.use_mpi. For dask clusters, initialize using use_distributed. for MPI clustrers, initialize using use_mpi. The MPI cluster mode is the default.
The global cache context manager is removed. The opportunistic caching it is now always enabled in MPI mode (when dask's synchronous scheduler is used); and always disabled in distributed mode, when the distributed scheduler is used.
The pickling trick was from mpi4py's bitbucket issue : https://bitbucket.org/mpi4py/mpi4py/issues/104/#comment-47225639
The singleton initialization needs to be revisited after the distributed PR dask/distributed#2391 is merged.