New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
BUG: multiprocessing scheduler silently hangs with lock task argument #1342
Comments
Multiprocessing locks aren't serializable, but What is your expected behavior here? A lock exists only in a process - serializing it between processes (if that worked) wouldn't have inter-process locking. A few options:
If you only need locking in each process, but don't care if many processes access the same resource concurrently, then a small wrapper class around |
Thanks of the in-depth explanation! This became an issue in #1293, and the desired behavior there was to lock multiple processes from using the same resource (an hdf file) simultaneously. AFAIK using multiprocessing.Lock is supposed to be share-able between multiple processes by using a single global lock instead of serializing it (see this stackoverflow question) which is what i did in a workaround in #1293. I now see your recommendation of using The purpose of reporting this issue is mostly to make this problem visible and documented if it ever comes up, and maybe adding a explicit exception or warning to the code in the future. |
That's a good point. If you serialize the lock file path, and create a new I fixed the deadlock problem in #1343, so the scheduler won't lock up on deserialization errors. |
Thanks for the extremely fast fix! This makes #1341 redundant, right? Should I close or would you like to keep the tests anyway? |
We could ping the locket guy. He's pretty responsive. |
When passing a multiprocessing lock and using the multiprocessing scheduler in a dask graph (without even using the lock) dask silently hangs when
multiprocessing.get
is called.PR #1341 adds a xfail test that proves this point
The text was updated successfully, but these errors were encountered: