Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

What to do when threads_per_worker is 0 #3986

Merged
merged 2 commits into from
Jul 24, 2020

Conversation

jsignell
Copy link
Member

tl;dr Maybe threads_per_worker should throw an error if set to 0.

While working on #3984 I ran into some odd behaviors when threads_per_worker is set to 0 in LocalCluster. I am not sure what the intended behavior should be or if this PR makes things better.

Some examples:

On master the following results in a divide by 0 error. This PR sets n_workers to 0 which I think is a good solution:

from distributed import LocalCluster, Client
cluster = LocalCluster(threads_per_worker=0)
client = Client(cluster)

client
# <Client: 'tcp://127.0.0.1:46121' processes=0 threads=0, memory=0 B> 

For the next case, I am not sure that the behavior should be. Here the nthreads is being passed int the worker_kwargs as 0. So we don't really have control over what happens after that. In this case it sets nthreads for each worker to CPU_COUNT

from distributed import LocalCluster, Client

cluster = LocalCluster(n_workers=2, threads_per_worker=0)
client = Client(cluster)

client
# <Client: 'tcp://127.0.0.1:46509' processes=2 threads=24, memory=33.37 GB>

@jsignell
Copy link
Member Author

@gforsyth who I was looking at this with.

@mrocklin
Copy link
Member

I think that we should raise if threads_per_worker < 1

When testing in the distributed repository async tests are preferred, if they're easy to use. You'll also want to use the cleanup fixture.

@jsignell
Copy link
Member Author

I think that we should raise if threads_per_worker < 1

That makes sense to me and should result in unambiguous behavior.

@jsignell
Copy link
Member Author

@mrocklin I made it a warning instead of an error since it is easy enough to set it to None.

@mrocklin mrocklin merged commit ccc9d2a into dask:master Jul 24, 2020
@mrocklin
Copy link
Member

Thanks @jsignell ! Merged.

@jsignell jsignell deleted the threads_per_worker_0 branch July 24, 2020 18:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants