Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Create new threadpool when operating from thread #1487

Merged
merged 4 commits into from
Aug 23, 2016
Merged

Create new threadpool when operating from thread #1487

merged 4 commits into from
Aug 23, 2016

Conversation

mrocklin
Copy link
Member

@mrocklin mrocklin commented Aug 19, 2016 •
edited
Loading

This cleans up some issues in the threaded scheduler when operating from other threads.

  1. Add test for calling dask.threaded.get in Worker distributed#441 shows that dask.threaded.get can block when being called from other threads
  2. We were creating but not cleaning up threadpools when using num_workers= keyword argument

Now we keep a pool of pools around, keyed by thread and num_workers. These get cleaned up at every call to get.

Reported by @AlbertDeFusco

@mrocklin
Create new threadpool when operating from thread
aceb84c 
When we call dask.threaded.get from a separate thread we shouldn't use
the default threadpool.  ThreadPool objects are not safe in this manner.

Now, if we call dask.threaded.get from the non-master thread we create
a new ThreadPool.

This needs a test and some more careful thought.  WIP
@mrocklin
Copy link
Member Author

I have a test for this in the distributed repository. I'll create something here soon as well.

@mrocklin mrocklin force-pushed the thread-threadpool branch 2 times, most recently from fa5cc21 to 9936b26 Compare August 20, 2016 12:10
@mrocklin
Copy link
Member Author

@jcrist @shoyer this change to the threaded scheduler could use your review if you have the time

shoyer
shoyer reviewed Aug 21, 2016
View reviewed changes
dask/threaded.py
pool = pools[thread][num_workers]
else:
pool = ThreadPool(num_workers)
pools[thread][num_workers] = pool
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't you need a lock here? I'm imagining a race condition where multiple threads try to create thread pools at the same time, e.g., there wasn't an existing pool when the check on line 61 was run, but there is when you overwrite the dictionary key.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've added a lock fairly conservatively around the all pools management code

@mrocklin
Copy link
Member Author

Merging soon if no comments

@shoyer
Copy link
Member

shoyer commented Aug 22, 2016

This seems sane to me now.

On Mon, Aug 22, 2016 at 9:58 AM, Matthew Rocklin notifications@github.com
wrote:

Merging soon if no comments

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#1487 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABKS1qLVoU9n5NztCxCuCqg3G8EvB8-vks5qidUngaJpZM4Jog2B
.

@mrocklin mrocklin merged commit 229957d into dask:master Aug 23, 2016
@mrocklin mrocklin deleted the thread-threadpool branch August 23, 2016 01:50
@sinhrks sinhrks added this to the 0.11.1 milestone Oct 9, 2016
@rmancy-smg rmancy-smg mentioned this pull request Feb 6, 2017
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
0.11.1
Development

Successfully merging this pull request may close these issues.
3 participants
@mrocklin @shoyer @sinhrks