Multiprocessing maxtasksperchild results in hang

[Jimbofbx]

BPO	10332
Nosy	@pitrou
Files	pool_lifetime_close-1.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2011-10-24.18:06:42.824>
created_at = <Date 2010-11-05.22:22:28.334>
labels = ['type-bug', 'library']
title = 'Multiprocessing maxtasksperchild results in hang'
updated_at = <Date 2011-10-24.18:06:42.822>
user = 'https://bugs.python.org/Jimbofbx'

bugs.python.org fields:

activity = <Date 2011-10-24.18:06:42.822>
actor = 'neologix'
assignee = 'none'
closed = True
closed_date = <Date 2011-10-24.18:06:42.824>
closer = 'neologix'
components = ['Library (Lib)']
creation = <Date 2010-11-05.22:22:28.334>
creator = 'Jimbofbx'
dependencies = []
files = ['23489']
hgrepos = []
issue_num = 10332
keywords = ['patch', 'needs review']
message_count = 8.0
messages = ['120547', '133003', '133648', '133667', '146119', '146268', '146308', '146313']
nosy_count = 7.0
nosy_names = ['pitrou', 'jnoller', 'asksol', 'neologix', 'Jimbofbx', 'python-dev', 'jkeating']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue10332'
versions = ['Python 2.7', 'Python 3.2', 'Python 3.3']

[Jimbofbx]

v.3.2a3

If the maxtasksperchild argument is used, the program will just hang after whatever that value is rather than working as expected. Tested in Windows XP 32-bit

test code:

import multiprocessing

def f(x):
    return 0;

if __name__ == '__main__':
    pool = multiprocessing.Pool(processes=2,maxtasksperchild=1);
    results = list();
    for i in range(10):
        results.append(pool.apply_async(f, (i)));
    pool.close();
    pool.join();
    for r in results:
        print(r);
    print("Done");

[jkeating]

I can duplicate this using python-2.7-8.fc14.1 on Fedora 14, and using map_async with the pool.

[neologix]

This problem arises because the pool's close method is called before all the tasks have completed. Putting a sleep(1) before pool.close() won't exhibit this lockup.
The root cause is that close makes the workers handler thread exit: since the maxtasksperchild argument is used, workers exit when they've processed their max number of tasks. But since the workers handler thread exited, it doesn't maintain the pool of workers anymore, and thus the remaining tasks are not treated anymore, and the task handler thread waits indefinitely (since it waits until the cache is empty).
The solution is to prevent the worker handler thread from exiting until the cache has been drained (unless the pool is terminated in which case it must exit right away).
Attached is a patch and relevant test.

Note: I noticed that there are some thread-unsafe operations (the cache that can be modified from different threads, and thread states are modified also from different threads). While this isn't an issue with the current cPython implementation (GIL), I wonder if this should be fixed.

[jnoller]

Note: I noticed that there are some thread-unsafe operations (the cache that can be modified from different threads, and thread states are modified also from different threads). While this isn't an issue with the current cPython implementation (GIL), I wonder if this should be fixed.

Yes. We should fix those.

[neologix]

Here's an updated patch.
I'll open a separate issue for the thread-safety.

[pitrou]

The patch looks good to me, thanks.

[python-dev]

New changeset 3465a9b2d25c by Charles-François Natali in branch '2.7':
Issue bpo-10332: multiprocessing: fix a race condition when a Pool is closed
http://hg.python.org/cpython/rev/3465a9b2d25c

New changeset 52c98a729a71 by Charles-François Natali in branch '3.2':
Issue bpo-10332: multiprocessing: fix a race condition when a Pool is closed
http://hg.python.org/cpython/rev/52c98a729a71

New changeset c2cdabc44665 by Charles-François Natali in branch 'default':
Issue bpo-10332: multiprocessing: fix a race condition when a Pool is closed
http://hg.python.org/cpython/rev/c2cdabc44665

[neologix]

James, thanks for the report!