ProcessPoolExecutor in interactive shell doesn't work in Windows

[Decade]

BPO	17874
Nosy	@brianquinlan
Superseder	bpo-11161: futures.ProcessPoolExecutor hangs

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2013-11-11.12:58:20.298>
created_at = <Date 2013-04-30.03:16:37.629>
labels = ['type-bug', 'OS-windows']
title = "ProcessPoolExecutor in interactive shell doesn't work in Windows"
updated_at = <Date 2013-11-11.12:58:20.296>
user = 'https://bugs.python.org/Decade'

bugs.python.org fields:

activity = <Date 2013-11-11.12:58:20.296>
actor = 'sbt'
assignee = 'none'
closed = True
closed_date = <Date 2013-11-11.12:58:20.298>
closer = 'sbt'
components = ['Windows']
creation = <Date 2013-04-30.03:16:37.629>
creator = 'Decade'
dependencies = []
files = []
hgrepos = []
issue_num = 17874
keywords = []
message_count = 7.0
messages = ['188122', '188123', '188150', '188199', '188205', '201274', '202624']
nosy_count = 3.0
nosy_names = ['bquinlan', 'sbt', 'Decade']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = '11161'
type = 'behavior'
url = 'https://bugs.python.org/issue17874'
versions = ['Python 3.3']

[Decade]

ProcessPoolExecutor doesn't work in an interactive shell in Windows, such as IDLE or the command prompt. It does work in Unix, and it works if I use the ThreadPoolExecutor instead.

For example, let's use the tutorial at http://eli.thegreenplace.net/2013/01/16/python-paralellizing-cpu-bound-tasks-with-concurrent-futures/

I get:
>>> pool_factorizer_chunked([1,2,3,456,7,8],8)
Exception in thread Thread-1:
Traceback (most recent call last):
  File "C:\Python33\lib\threading.py", line 639, in _bootstrap_inner
    self.run()
  File "C:\Python33\lib\threading.py", line 596, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Python33\lib\concurrent\futures\process.py", line 248, in _queue_management_worker
    shutdown_worker()
  File "C:\Python33\lib\concurrent\futures\process.py", line 208, in shutdown_worker
    call_queue.put_nowait(None)
  File "C:\Python33\lib\multiprocessing\queues.py", line 132, in put_nowait
    return self.put(obj, False)
  File "C:\Python33\lib\multiprocessing\queues.py", line 79, in put
    raise Full
queue.Full

Traceback (most recent call last):
  File "<pyshell#10>", line 1, in <module>
    pool_factorizer_chunked([1,2,3,456,7,8],8)
  File "<pyshell#5>", line 14, in pool_factorizer_chunked
    resultdict.update(f.result())
  File "C:\Python33\lib\concurrent\futures\_base.py", line 392, in result
    return self.__get_result()
  File "C:\Python33\lib\concurrent\futures\_base.py", line 351, in __get_result
    raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.
>>> pool_factorizer_map([1,2,3,456,7,8],8)
Exception in thread Thread-2:
Traceback (most recent call last):
  File "C:\Python33\lib\threading.py", line 639, in _bootstrap_inner
    self.run()
  File "C:\Python33\lib\threading.py", line 596, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Python33\lib\concurrent\futures\process.py", line 248, in _queue_management_worker
    shutdown_worker()
  File "C:\Python33\lib\concurrent\futures\process.py", line 208, in shutdown_worker
    call_queue.put_nowait(None)
  File "C:\Python33\lib\multiprocessing\queues.py", line 132, in put_nowait
    return self.put(obj, False)
  File "C:\Python33\lib\multiprocessing\queues.py", line 79, in put
    raise Full
queue.Full

Traceback (most recent call last):
  File "<pyshell#13>", line 1, in <module>
    pool_factorizer_map([1,2,3,456,7,8],8)
  File "<pyshell#12>", line 6, in pool_factorizer_map
    executor.map(factorize_naive, nums))}
  File "<pyshell#12>", line 4, in <dictcomp>
    return {num:factors for num, factors in
  File "C:\Python33\lib\concurrent\futures\_base.py", line 546, in result_iterator
    yield future.result()
  File "C:\Python33\lib\concurrent\futures\_base.py", line 399, in result
    return self.__get_result()
  File "C:\Python33\lib\concurrent\futures\_base.py", line 351, in __get_result
    raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.
>>>

[Decade]

Oh, yeah, using Python 3.3.1 (v3.3.1:d9893d13c628, Apr 6 2013, 20:30:21) [MSC v.1600 64 bit (AMD64)] on win32

[sbt]

Some sort of error is expected.

On Windows any functions or classes used in a task sent the executor must be picklable/unpicklable. This means that they must be defined in an importable module rather than being defined in the interactive shell.

On Unix, you can use a definition made in the interactive shell, but *only* if it was defined before starting the executor.

[Decade]

Ah. Then, a documentation error. The error message ("queue.Full"?) and the documentation are totally not clear about that. Does ProcessPoolExecutor just not require tasks to be picklable in Unix?

Also, this raises questions about what exactly "picklable" means, and why it's not defined in the documentation.

[sbt]

Ah. Then, a documentation error. The error message ("queue.Full"?) and
the documentation are totally not clear about that.

Once something goes wrong you are likely to get a cascade of errors, and the first one reported is not necessarily the original cause.

Does ProcessPoolExecutor just not require tasks to be picklable in Unix?

On Unix the main process forks using os.fork() when the executor is created. The forked processes inherit all the definitions previously created in the main process.

Also, this raises questions about what exactly "picklable" means, and why
it's not defined in the documentation.

Since concurrent.futures uses multiprocessing the following section applies

http://docs.python.org/dev/library/multiprocessing.html#windows

[brianquinlan]

Take a look at http://bugs.python.org/issue11161 and its fix. Do you think that just saying that ProcessPoolExecutor will not work from the interactive shell is sufficient?

[sbt]

Fixed by bpo-11161.