Deadlock in test_concurrent_futures #56573
Comments
[271/356/1] test_concurrent_futures
Traceback (most recent call last):
File "/home2/buildbot/slave/3.x.loewis-sun/build/Lib/multiprocessing/queues.py", line 268, in _feed
send(obj)
File "/home2/buildbot/slave/3.x.loewis-sun/build/Lib/multiprocessing/connection.py", line 229, in send
self._send_bytes(memoryview(buf))
File "/home2/buildbot/slave/3.x.loewis-sun/build/Lib/multiprocessing/connection.py", line 423, in _send_bytes
self._send(struct.pack("=i", len(buf)))
File "/home2/buildbot/slave/3.x.loewis-sun/build/Lib/multiprocessing/connection.py", line 392, in _send
n = write(self._handle, buf)
OSError: [Errno 32] Broken pipe
Timeout (1:00:00)!
Thread 0x00000954:
File "/home2/buildbot/slave/3.x.loewis-sun/build/Lib/threading.py", line 237 in wait
File "/home2/buildbot/slave/3.x.loewis-sun/build/Lib/multiprocessing/queues.py", line 252 in _feed
File "/home2/buildbot/slave/3.x.loewis-sun/build/Lib/threading.py", line 690 in run
File "/home2/buildbot/slave/3.x.loewis-sun/build/Lib/threading.py", line 737 in _bootstrap_inner
File "/home2/buildbot/slave/3.x.loewis-sun/build/Lib/threading.py", line 710 in _bootstrapThread 0x00000953: Thread 0x00000001: See commit e6e7e42efdc2 of the issue bpo-12310. |
|
Message on a stackoverflow thread: "I have suffered from the same problem, even if connecting on localhost in python 2.7.1. After a day of debugging i found the cause and a workaround: Cause: BaseProxy class has thread local storage which caches the connection, which is reused for future connections causing "broken pipe" errors even on creating a new Manager Workaround: Delete the cached connection before reconnecting if address in BaseProxy._address_to_local:
del BaseProxy._address_to_local[self.address][0].connection"--- See also maybe the (closed) issue bpo-11663: multiprocessing doesn't detect killed processes |
|
Connection._send_bytes() has a comment about broken pipes: def _send_bytes(self, buf):
# For wire compatibility with 3.2 and lower
n = len(buf)
self._send(struct.pack("=i", len(buf)))
# The condition is necessary to avoid "broken pipe" errors
# when sending a 0-length buffer if the other end closed the pipe.
if n > 0:
self._send(buf)But the OSError(32, "Broken pipe") occurs on sending the buffer size (a chunk of 4 bytes: self._send(struct.pack("=i", len(buf)))), not on sending the buffer content. See also maybe the (closed) issue bpo-9205: Parent process hanging in multiprocessing if children terminate unexpectedly |
|
Ah, submit a new task after the manager shutdown fails with OSError(32, 'Broken pipe'). Example: from multiprocessing.managers import BaseManager
class MathsClass(object):
def foo(self):
return 42
class MyManager(BaseManager):
pass
MyManager.register('Maths', MathsClass)
if __name__ == '__main__':
manager = MyManager()
manager.start()
maths = manager.Maths()
maths.foo()
manager.shutdown()
try:
maths.foo()
finally:
manager.shutdown()This example doesn't hang, but this issue is about concurrent.futures, not multiprocessing. |
|
Oh, I think that I found a deadlock (or something like that): import concurrent.futures
import faulthandler
import os
import signal
import time
def work(n):
time.sleep(0.1)
def main():
faulthandler.register(signal.SIGUSR1)
print("pid: %s" % os.getpid())
with concurrent.futures.ProcessPoolExecutor() as executor:
for number, prime in executor.map(work, range(100)):
print("shutdown")
executor.shutdown()
print("shutdown--")
if __name__ == '__main__':
main()Trace: Thread 0x00007fbfc8bbe700: Current thread 0x00007fbfcc2e3700: |
vstinner
changed the title
|
Retrieving the result of a future after the executor has been shut down can cause a hang. It seems like this regression was introduced in a76257a99636. This regression exists only for ProcessPoolExecutor. The problem is that even if there are pending work items, the processes are still signaled to shut down leaving the pending work items permanently unfinished. The patch simply removes the call to shut down the processes when there are pending work items. Attached is a patch. |
|
Well I was sure I had added this code for a reason, but the tests seem to run without... |
|
New changeset 26389e9efa9c by Ross Lagerwall in branch '3.2': New changeset 25f879011102 by Ross Lagerwall in branch 'default': |
|
Thanks! |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: