You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee=Noneclosed_at=<Date2018-07-17.16:51:46.885>created_at=<Date2018-07-17.14:21:19.056>labels= ['type-bug', 'invalid']
title='Possible multiprocessing deadlock when placing too many objects in Queue()'updated_at=<Date2018-07-17.16:51:46.883>user='https://github.com/fhstoica'
I am trying to use the multiprocessing module for a simulation on a spherical lattice, but the process hangs when the lattice is too large.
In the file IssuesWithQueueMultiProcessing.py, the method createLattice(), use either "return(I4)" for the small lattice or "return(I5)" for the large lattice.
Running the script when using the large lattice causes the process to hang while for the small lattice it works fine. I have tested with Python 3.5.2 and 3.6.1 and the behavior is the same in both versions.
The problem is you're joining the child processes before draining the queue in the parent.
Generally, instead of building your own kind of synchronization like this, I would recommend you use the higher-level abstractions provided by multiprocessing.Pool or concurrent.futures.ProcessPoolExecutor.
By the way, this issue is mentioned precisely in the documentation:
"""
As mentioned above, if a child process has put items on a queue (and it has not used JoinableQueue.cancel_join_thread), then that process will not terminate until all buffered items have been flushed to the pipe.
This means that if you try joining that process you may get a deadlock unless you are sure that all items which have been put on the queue have been consumed. Similarly, if the child process is non-daemonic then the parent process may hang on exit when it tries to join all its non-daemonic children.
"""
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: