Multiprocessing worker functions not terminating with a large number of processes and a manager

[EricG]

BPO	32937
Nosy	@ronaldoussoren, @pitrou, @applio, @akulakov

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-08-16.23:22:57.149>
created_at = <Date 2018-02-24.14:17:03.214>
labels = ['OS-mac', 'type-bug', 'library']
title = 'Multiprocessing worker functions not terminating with a large number of processes and a manager'
updated_at = <Date 2021-08-16.23:22:57.149>
user = 'https://bugs.python.org/Ericg'

bugs.python.org fields:

activity = <Date 2021-08-16.23:22:57.149>
actor = 'ned.deily'
assignee = 'none'
closed = True
closed_date = <Date 2021-08-16.23:22:57.149>
closer = 'ned.deily'
components = ['Library (Lib)', 'macOS']
creation = <Date 2018-02-24.14:17:03.214>
creator = 'Ericg'
dependencies = []
files = []
hgrepos = []
issue_num = 32937
keywords = []
message_count = 9.0
messages = ['312718', '312747', '312748', '312751', '312924', '312950', '313108', '380058', '398224']
nosy_count = 5.0
nosy_names = ['ronaldoussoren', 'pitrou', 'davin', 'Ericg', 'andrei.avk']
pr_nums = []
priority = 'normal'
resolution = 'out of date'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue32937'
versions = ['Python 3.6']

[EricG]

I have the following code:

    import multiprocessing
    from multiprocessing import Pool, Manager
    import time
    import random

    def worker_function( index, messages ):

        print( "%d: Entered" % index )
        time.sleep( random.randint( 3, 15 ) )
        messages.put( "From: %d" % index )
        print( "%d: Exited" % index )

    manager = Manager()
    messages = manager.Queue()

    with Pool( processes = None ) as pool:

        for x in range( 30 ):
            pool.apply_async( worker_function, [ x, messages ] )

        pool.close()
        pool.join()

It does not terminate -- all entered messages are printed, but not all exited messages are printed.

If I remove all the code related to the Manager and Queue, it will terminate properly with all messages printed.

If I assign processes explicitly, I can continue to increase the number assigned to processes and have it continue to work until I reach a value of 20 or 21. > 20, it fails all of the time. With a value == 20 it fails some of the time. With a value of < 20, it always succeeds.

multiprocessing.cpu_count() returns 24 for my MacPro.

[pitrou]

What happens if you add another process that calls get() on the queue? You should not try to put data on a queue if you don't ever plan to consume it, as the queue's background thread will eventually block until something gets consumed.

For example, this blocks here on Linux:

$ ./python -c "import multiprocessing as mp; q = mp.Queue(); [q.put(None) for i in range(50000)]"

[EricG]

I do plan to consume the messages on the queue, but only after all worker functions are complete...after pool.join() returns. Is this not ok?

I can certainly spawn a thread on the main process which will consume the queue entries and insert them into a list or queue which can then be accessed after join returns. Is that the correct way this code should be written?

[pitrou]

I'm not sure... I cannot reproduce your problem on Linux, even with 50 processes and 10000 iterations, on Python 3.6.4.

Which exact version are you using?
What happens if you replace the manager Queue with a non-manager Queue?

[EricG]

If I do:

    from queue import Queue

    messages = Queue()

No messages are printed. I believe this is expected as a regular Queue cannot be shared between processes. It was a problem that the manager was designed to solve.

I am using a MacPro running 10.3.2
Python 3.6.4

It would not surprise me if this were an OS specific issue. To reproduce it may requiring using a Mac with a high number of cores.

It is trivially reproducible for me.

[EricG]

Making some further observations, when I set processes = 12, for example, I can see 12 separate python processes + 4 additional processes also created which I assume are setup for the manager and, perhaps, other purposes.

Now, what makes these 4 additional processes interesting is that for me, multiprocessing.cpu_count() returns 24. And, it is when I set processes = 20 that the python code will sometimes terminate successfully. 20 + 4 = 24...so I am using every single cpu in that situation.

However, as noted, when I set processes = 19, it will always terminate successfully. 19 + 4 < 24...there is at least one cpu not assigned any work.

Perhaps there some some kind of race condition or swapping around of data structures or something that only happens on macOS when every cpu is in use by python for this purpose.

[pitrou]

I don't know, maybe someone with a Mac wants to try and debug it?

[ronaldoussoren]

This issue can probably be closed as out of date:

Using: Python3.9, installer from python.org. On my laptop os.cpu_count() == 8.

"python -c 'import repro; repro.main()' works for me (it runs a while and exists when all workers have exited). The repro module contains the code below, this is the code from the original message with some changes because of changes to the multiprocessing launcher strategy on macOS.

# repro.py
import multiprocessing
from multiprocessing import Pool, Manager
import time
import random

def worker_function( index, messages ):

    print( "%d: Entered" % index )
    time.sleep( random.randint( 3, 15 ) )
    messages.put( "From: %d" % index )
    print( "%d: Exited" % index )


def main():
    manager = Manager()
    messages = manager.Queue()
    with Pool( processes = None ) as pool:

        for x in range( 30 ):
            pool.apply_async( worker_function, [ x, messages ] )

        pool.close()
        pool.join()
# EOF

[akulakov]

I have confirmed the reproducer posted by Ronald also works for me, on 3.11 compiled from dev branch, also with cpu count == 8; on the new M1 silicon (that shouldn't matter though).

I agree this can be closed.