New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ProcessPoolExecutor documentation reports wrong exception being raised #78967
Comments
https://docs.python.org/3.8/library/concurrent.futures.html "initializer is an optional callable that is called at the start of each worker process; initargs is a tuple of arguments passed to the initializer. Should initializer raise an exception, all currently pending jobs will raise a BrokenThreadPool, as well any attempt to submit more jobs to the pool." This should be BrokenProcessPool? |
I think it raises BrokenThreadPool . A sample program that I tried as below raising an exception in the initializer. Maybe I am wrong here. Can you please attach a script that triggers BrokenProcessPool? # bpo-34786.py import concurrent.futures
import time
def bar(i):
raise Exception(i) # Raise exception from the initializer
def foo(i):
time.sleep(i)
return "1" with concurrent.futures.ThreadPoolExecutor(max_workers=5, # Run the program ./python.exe ../backups/bpo34786.py
Exception in initializer:
Traceback (most recent call last):
File "/Users/karthikeyansingaravelan/stuff/python/cpython/Lib/concurrent/futures/thread.py", line 69, in _worker
initializer(*initargs)
File "../backups/bpo34786.py", line 5, in bar
raise Exception(i)
Exception: 1
Exception in initializer:
Traceback (most recent call last):
File "/Users/karthikeyansingaravelan/stuff/python/cpython/Lib/concurrent/futures/thread.py", line 69, in _worker
initializer(*initargs)
File "../backups/bpo34786.py", line 5, in bar
raise Exception(i)
Exception: 1
Traceback (most recent call last):
File "../backups/bpo34786.py", line 13, in <module>
future_to_url = {executor.submit(foo, i, 60): i for i in range(10)}
File "../backups/bpo34786.py", line 13, in <dictcomp>
future_to_url = {executor.submit(foo, i, 60): i for i in range(10)}
File "/Users/karthikeyansingaravelan/stuff/python/cpython/Lib/concurrent/futures/thread.py", line 148, in submit
raise BrokenThreadPool(self._broken)
concurrent.futures.thread.BrokenThreadPool: A thread initializer failed, the thread pool is not usable anymore Thanks |
Perhaps I wasn't clear that this considered ProcessPoolExecutor documentation, not ThreadPoolExecutor. Taking the example from documentation and adding an initializer like this: import concurrent.futures
import math
PRIMES = [
112272535095293,
112582705942171,
112272535095293,
115280095190773,
115797848077099,
1099726899285419]
def is_prime(n):
if n % 2 == 0:
return False
sqrt_n = int(math.floor(math.sqrt(n)))
for i in range(3, sqrt_n + 1, 2):
if n % i == 0:
return False
return True
def init():
raise Exception()
def main():
with concurrent.futures.ProcessPoolExecutor(initializer=init) as executor:
for number, prime in zip(PRIMES, executor.map(is_prime, PRIMES)):
print('%d is prime: %s' % (number, prime))
if __name__ == '__main__':
main() ...yields the following: Exception in initializer:
Traceback (most recent call last):
File "/Users/developer/cpython/Lib/concurrent/futures/process.py", line 219, in _process_worker
initializer(*initargs)
File "/tmp/bpo34786.py", line 25, in init
raise Exception()
Exception
Exception in initializer:
Traceback (most recent call last):
File "/Users/developer/cpython/Lib/concurrent/futures/process.py", line 219, in _process_worker
initializer(*initargs)
File "/tmp/bpo34786.py", line 25, in init
raise Exception()
Exception
Exception in initializer:
Traceback (most recent call last):
File "/Users/developer/cpython/Lib/concurrent/futures/process.py", line 219, in _process_worker
initializer(*initargs)
File "/tmp/bpo34786.py", line 25, in init
raise Exception()
Exception
Exception in initializer:
Traceback (most recent call last):
File "/Users/developer/cpython/Lib/concurrent/futures/process.py", line 219, in _process_worker
initializer(*initargs)
File "/tmp/bpo34786.py", line 25, in init
raise Exception()
Exception
Traceback (most recent call last):
File "/tmp/bpo34786.py", line 33, in <module>
main()
File "/tmp/bpo34786.py", line 29, in main
for number, prime in zip(PRIMES, executor.map(is_prime, PRIMES)):
File "/Users/developer/cpython/Lib/concurrent/futures/process.py", line 476, in _chain_from_iterable_of_lists
for element in iterable:
File "/Users/developer/cpython/Lib/concurrent/futures/_base.py", line 594, in result_iterator
yield fs.pop().result()
File "/Users/developer/cpython/Lib/concurrent/futures/_base.py", line 436, in result
return self.__get_result()
File "/Users/developer/cpython/Lib/concurrent/futures/_base.py", line 388, in __get_result
raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending. |
Got it. Sorry about that I copy pasted the quoted string and assumed ThreadPoolExecutor overlooking the PR. The change is reasonable to me. Thanks |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: