Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bugfix: deadlock #2

Merged
merged 2 commits into from
Apr 29, 2018
Merged

bugfix: deadlock #2

merged 2 commits into from
Apr 29, 2018

Conversation

SayreBlades
Copy link
Owner

@SayreBlades SayreBlades commented Apr 29, 2018

notes

ughh, this was a nightmare:

  1. gdb stacks did not indicate deadlock
  2. It was actually an issue with aws. boto3 client was throwing an exception intermittently. These exceptions were being swallowed and the thread would be re-allocated to the pool.
  3. bubbling up these exceptions to main thread is not documented anywhere in dask

red herrings:

boto/boto3#454
boto/boto3#801

canonical boto3 multithreading:

http://boto3.readthedocs.io/en/latest/guide/resources.html#multithreading-multiprocessing.

100 million prime numbers

I also dropped in a new prime number search example... shit is so hot:

$ AWS_PROFILE=personal AWS_DEFAULT_REGION=us-east-1 python examples/primes.py
[########################################] | 100% Completed | 43.7s
5761457

try:
res = func_wrapper(*args, **kwds)
except BaseException as e:
result = e, sys.exc_info()[2]
Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the handler for exceptions that happen outside your ad-hoc client code. I.e. your client function may divide by zero, and that is handled by dask internal machinery execute_task (https://github.com/dask/dask/blob/master/dask/local.py#L280). However if there is an issue with boto3 or networking to the server, these errors also need to propogate up to the dask task executer.

@SayreBlades SayreBlades merged commit 348d2af into master Apr 29, 2018
@SayreBlades SayreBlades deleted the bugfix/deadlock branch April 29, 2018 04:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant