New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
hang and/or leaked processes with multiprocessing.Pool(...).imap(...) #79810
Comments
This simple program causes a hang / leaked processes (easiest to run in an interactive shell): import multiprocessing
tuple(multiprocessing.Pool(4).imap(print, (1, 2, 3))) $ python3.6
Python 3.6.7 (default, Oct 22 2018, 11:32:17)
[GCC 8.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import multiprocessing
>>> tuple(multiprocessing.Pool(4).imap(print, (1, 2, 3)))
1
2
3
<<<hang>>>
^CProcess ForkPoolWorker-1:
Traceback (most recent call last):
Process ForkPoolWorker-2:
Process ForkPoolWorker-3:
Process ForkPoolWorker-4:
File "/usr/lib/python3.6/multiprocessing/pool.py", line 746, in next
item = self._items.popleft()
IndexError: pop from an empty deque
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.6/multiprocessing/pool.py", line 750, in next
self._cond.wait(timeout)
File "/usr/lib/python3.6/threading.py", line 295, in wait
waiter.acquire()
KeyboardInterrupt
$ python3.7
Python 3.7.2 (default, Dec 25 2018, 03:50:46)
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import multiprocessing
>>> tuple(multiprocessing.Pool(4).imap(print, (1, 2, 3)))
1
2
3
(None, None, None)
>>>
KeyboardInterrupt
Process ForkPoolWorker-3:
Process ForkPoolWorker-1:
Process ForkPoolWorker-2:
Process ForkPoolWorker-4:
>>> Traceback (most recent call last):
File "/usr/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
self.run()
File "/usr/lib/python3.7/multiprocessing/process.py", line 99, in run
self._target(*self._args, **self._kwargs)
Traceback (most recent call last):
File "/usr/lib/python3.7/multiprocessing/pool.py", line 110, in worker
task = get()
File "/usr/lib/python3.7/multiprocessing/queues.py", line 351, in get
with self._rlock:
File "/usr/lib/python3.7/multiprocessing/synchronize.py", line 95, in __enter__
return self._semlock.__enter__()
KeyboardInterrupt
File "/usr/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
self.run()
File "/usr/lib/python3.7/multiprocessing/process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib/python3.7/multiprocessing/pool.py", line 110, in worker
task = get()
File "/usr/lib/python3.7/multiprocessing/queues.py", line 351, in get
with self._rlock:
File "/usr/lib/python3.7/multiprocessing/synchronize.py", line 95, in __enter__
return self._semlock.__enter__()
KeyboardInterrupt
Traceback (most recent call last):
File "/usr/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
self.run()
File "/usr/lib/python3.7/multiprocessing/process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib/python3.7/multiprocessing/pool.py", line 110, in worker
task = get()
File "/usr/lib/python3.7/multiprocessing/queues.py", line 351, in get
with self._rlock:
File "/usr/lib/python3.7/multiprocessing/synchronize.py", line 95, in __enter__
return self._semlock.__enter__()
KeyboardInterrupt
Traceback (most recent call last):
File "/usr/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
self.run()
File "/usr/lib/python3.7/multiprocessing/process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib/python3.7/multiprocessing/pool.py", line 110, in worker
task = get()
File "/usr/lib/python3.7/multiprocessing/queues.py", line 352, in get
res = self._reader.recv_bytes()
File "/usr/lib/python3.7/multiprocessing/connection.py", line 216, in recv_bytes
buf = self._recv_bytes(maxlength)
File "/usr/lib/python3.7/multiprocessing/connection.py", line 407, in _recv_bytes
buf = self._recv(4)
File "/usr/lib/python3.7/multiprocessing/connection.py", line 379, in _recv
chunk = read(handle, remaining)
KeyboardInterrupt (python3.8 shows the same behaviour as python3.7) python2.7 also has similar behaviour. I'm told this more reliably hangs on windows, though I don't have windows on hand. I've "fixed" my code to explicitly open / close the pool: with contextlib.closing(multiprocessing.Pool(jobs)) as pool:
tuple(pool.imap(...)) I suspect a refcounting / gc bug |
Weirdly enough, it works with iPython: $ ipython3
Python 3.7.1 (default, Nov 6 2018, 18:49:54)
Type 'copyright', 'credits' or 'license' for more information
IPython 6.4.0 -- An enhanced Interactive Python. Type '?' for help.
|
I believe that this is similar to https://bugs.python.org/issue35378 on which @pablogsal is working. You were right, the issue steems from a refcount bug. Until the resolution you can avoid the issue by explictly keeping a reference on the pool: >>> import multiprocessing
>>> d = multiprocessing.Pool(4)
>>> tuple(d.imap(print, (1, 2, 3)))
1
2
3
(None, None, None)
>>> |
Indeed, looks like a duplicate. |
I suggest you to write: with multiprocessing.Pool(4) as pool: result = tuple(pool.imap(print, (1, 2, 3))) On Python 3.8, your example will now log a resource warning since you don't close/terminate explicitly the pool. |
If you see the bottom of my issue, I've suggested (nearly) the same thing -- though I require python2.x compatibility so I'm using |
Oh, I missed that: good! |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: