Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Limit concurrency when gathering many times #2303

Merged
merged 2 commits into from Oct 9, 2018

Conversation

mrocklin
Copy link
Member

@mrocklin mrocklin commented Oct 9, 2018

Previously requests like the following would generate many independent
requests to the scheduler or workers

results = [future.result() for future in futures]

We would try to push people away from this and recommend that they use
gather, but this didn't always happen, especially when they were using
as_completed

for future, result in as_completed(futures, with_results=True)

So now we just impose a bit of flow control around the Client.gather
method (which all of these use) to allow only five concurrent
outgoing communications at once.

Fixes #1858
Fixes dask/dask#4011

Previously requests like the following would generate many independent
requests to the scheduler or workers

    results = [future.result() for future in futures]

We would try to push people away from this and recommend that they use
gather, but this didn't always happen, especially when they were using
as_completed

    for future, result in as_completed(futures, with_results=True)

So now we just impose a bit of flow control around the `Client.gather`
method (which all of these use) to allow only five concurrent
outgoing communications at once.

Fixes dask#1858
Fixes dask/dask#4011
@mrocklin mrocklin merged commit 8e67438 into dask:master Oct 9, 2018
@mrocklin mrocklin deleted the concurrent-gather branch October 9, 2018 20:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant