Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

updated docs with explicit scheduler setting example #78

Closed
orbeckst opened this issue Nov 1, 2018 · 3 comments
Closed

updated docs with explicit scheduler setting example #78

orbeckst opened this issue Nov 1, 2018 · 3 comments
Assignees

Comments

@orbeckst
Copy link
Member

orbeckst commented Nov 1, 2018

Expected behaviour

PMDA should run with "best performance" out of the box.

At a minimum, the docs should be clear what one has to do.

The dask docs recommend distributed for GIL-bound code so in the following I use it as the default, but we should benchmark the single machine schedulers

  • thread
  • processes
  • distributed

For using distributed:

from dask.distributed import Client
client = Client()

import pmda
...

or configure the scheduler

dask.config.set(scheduler='distributed')

Actual behaviour

With PMDA now using dask's preferred way to select a scheduler (#66), we now default to Dask's default scheduler. For delayed(), this is the threads scheduler, which does not work well with our Python based code: the GIL serializes the tasks and I expect that performance is poor out of the box.

@orbeckst
Copy link
Member Author

orbeckst commented Nov 1, 2018

@orbeckst
Copy link
Member Author

orbeckst commented Nov 2, 2018

I overlooked https://github.com/MDAnalysis/pmda/blob/master/pmda/parallel.py#L311 – we still default to 'multiprocessing'.

I am closing this as not super-urgent. More docs on how to set other schedulers are good, but can wait.

@orbeckst orbeckst closed this as completed Nov 2, 2018
@orbeckst
Copy link
Member Author

orbeckst commented Nov 2, 2018

Well, the docs need some updating... so I re-open so that I can close an issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant