multiprocessing.Pool from concurrent threads failure on 3.9.0rc1

[drougge]

BPO	41567
Nosy	@atsuoishimoto, @drougge, @aeros, @iritkatriel, @cebtenzzre
Superseder	bpo-43517: Fix false positives in circular import detection with from-imports
Files	pool_error_on_3.9.py: repro example

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-03-16.16:16:54.899>
created_at = <Date 2020-08-17.11:37:30.505>
labels = ['type-bug', 'library', '3.9', '3.10']
title = 'multiprocessing.Pool from concurrent threads failure on 3.9.0rc1'
updated_at = <Date 2021-03-16.16:16:54.899>
user = 'https://github.com/drougge'

bugs.python.org fields:

activity = <Date 2021-03-16.16:16:54.899>
actor = 'pitrou'
assignee = 'none'
closed = True
closed_date = <Date 2021-03-16.16:16:54.899>
closer = 'pitrou'
components = ['Library (Lib)']
creation = <Date 2020-08-17.11:37:30.505>
creator = 'drougge'
dependencies = []
files = ['49401']
hgrepos = []
issue_num = 41567
keywords = []
message_count = 6.0
messages = ['375542', '375544', '375556', '379400', '379401', '388771']
nosy_count = 6.0
nosy_names = ['ishimoto', 'drougge', 'aeros', 'iritkatriel', 'cebtenzzre', 'doublex']
pr_nums = []
priority = 'normal'
resolution = 'duplicate'
stage = 'resolved'
status = 'closed'
superseder = '43517'
type = 'behavior'
url = 'https://bugs.python.org/issue41567'
versions = ['Python 3.9', 'Python 3.10']

[drougge]

If several threads try to start a multiprocessing.Pool at the same time when no pool has been started before this often fails with an exception like this (the exact import varies):

Exception in thread Thread-2:
Traceback (most recent call last):
  File "/tmp/py3.9.0rc1/lib/python3.9/threading.py", line 950, in _bootstrap_inner
    self.run()
  File "/tmp/py3.9.0rc1/lib/python3.9/threading.py", line 888, in run
    self._target(*self._args, **self._kwargs)
  File "/tmp/py3.9.0rc1/lib/python3.9/multiprocessing/context.py", line 118, in Pool
    from .pool import Pool
ImportError: cannot import name 'Pool' from partially initialized module 'multiprocessing.pool' (most likely due to a circular import) (/tmp/py3.9.0rc1/lib/python3.9/multiprocessing/pool.py)

This happens even if Pool was imported before starting the threads and is new in 3.9. It's easy to work around by starting a pool in the main thread before starting the other threads.

I have attached a minimal example that triggers it. Tested on Debian stable and FreeBSD 11.3.

[iritkatriel]

I see the same in Python 3.10 on windows 10.

If I change the relative imports to absolute imports in a couple of functions in multiprocessing.context as below, the attached (pool_error_on_3.9.py) script not longer raises the exception.

    def SimpleQueue(self):
        '''Returns a queue object'''
        from multiprocessing.queues import SimpleQueue
        return SimpleQueue(ctx=self.get_context())

    def Pool(self, processes=None, initializer=None, initargs=(),
             maxtasksperchild=None):
        '''Returns a process pool object'''
        from multiprocessing.pool import Pool
        return Pool(processes, initializer, initargs, maxtasksperchild,
                    context=self.get_context())

[iritkatriel]

I think this is no a bug, on the basis that multiprocessing.Pool is not thread-safe.

[atsuoishimoto]

multiprocessing module used to work multithread environment until https://bugs.python.org/issue35943 merged.

I guess we can make multiprocessing thread-safe again if we move local imports to global.
Does it make sense?

[atsuoishimoto]

Some my observation at https://discuss.python.org/t/differences-between-3-8-and-3-9-in-importing-module/5520 .

[doublex]

Example code (fails):

import os, concurrent.futures

def parallel_callback( arg ):
    return os.getpid()

def parallel( *args ):
    def thread_callback( param ):
        with concurrent.futures.ProcessPoolExecutor(max_workers=1) as executor:
            future = executor.submit( parallel_callback, param )
            pid = future.result()
            print( 'pid:', pid )
            return pid
    with concurrent.futures.ThreadPoolExecutor(max_workers=len(args)) as executor:
        future = executor.map( thread_callback, args )
        results = list(future)
    print( 'DONE' )

parallel( 1, 2, 3 )