multiprocessing sentinel resource leak

[quick-b]

BPO	26732
Nosy	@pitrou, @vstinner, @applio
PRs	bpo-26732: fix too many fds in processes started with the "forkserver" method #2813

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2016-04-10.21:58:21.521>
labels = ['3.7', 'library', 'performance']
title = 'multiprocessing sentinel resource leak'
updated_at = <Date 2017-07-31.00:12:15.278>
user = 'https://bugs.python.org/quick-b'

bugs.python.org fields:

activity = <Date 2017-07-31.00:12:15.278>
actor = 'quick-b'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2016-04-10.21:58:21.521>
creator = 'quick-b'
dependencies = []
files = []
hgrepos = []
issue_num = 26732
keywords = []
message_count = 13.0
messages = ['263153', '263154', '263282', '298844', '298846', '299051', '299052', '299157', '299174', '299418', '299419', '299424', '299531']
nosy_count = 8.0
nosy_names = ['holdenweb', 'pitrou', 'vstinner', 'jnoller', 'sbt', 'davin', 'Winterflower', 'quick-b']
pr_nums = ['2813']
priority = 'normal'
resolution = None
stage = 'resolved'
status = 'open'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue26732'
versions = ['Python 3.6', 'Python 3.7']

[quick-b]

The sentinel creates a named pipe, but the parent's end of the pipe is inherited by subsequently created children.

import multiprocessing,signal,sys
def sproc(x): signal.pause()
for each in range(int(sys.argv[1])):
  multiprocessing.Process(target=sproc, args=(each,)).start()
signal.pause()

Running the above on Linux with varying numbers of child processes (expressed as the argument to the above) and using techniques like "$ sudo ls /proc/NNNN/fd" it is possible to see an ever growing number of pipe connections for subsequent children.

[quick-b]

(Sorry, an unnamed pipe, but a pipe nonetheless.)

[vstinner]

I confirm the issue with Python 3.6 on Linux.

File descriptors of the parent process:

haypo@selma$ ls -l /proc/31564/fd/
lrwx------. 1 haypo haypo 64 13 avril 00:55 0 -> /dev/pts/0
lrwx------. 1 haypo haypo 64 13 avril 00:55 1 -> /dev/pts/0
lr-x------. 1 haypo haypo 64 13 avril 00:55 10 -> pipe:[697354]
lr-x------. 1 haypo haypo 64 13 avril 00:55 11 -> pipe:[697355]
lr-x------. 1 haypo haypo 64 13 avril 00:55 12 -> pipe:[697356]
lrwx------. 1 haypo haypo 64 13 avril 00:54 2 -> /dev/pts/0
lr-x------. 1 haypo haypo 64 13 avril 00:55 3 -> pipe:[697347]
lr-x------. 1 haypo haypo 64 13 avril 00:55 4 -> pipe:[697348]
lr-x------. 1 haypo haypo 64 13 avril 00:55 5 -> pipe:[697349]
lr-x------. 1 haypo haypo 64 13 avril 00:55 6 -> pipe:[697350]
lr-x------. 1 haypo haypo 64 13 avril 00:55 7 -> pipe:[697351]
lr-x------. 1 haypo haypo 64 13 avril 00:55 8 -> pipe:[697352]
lr-x------. 1 haypo haypo 64 13 avril 00:55 9 -> pipe:[697353]

File descriptors of the first child process:

haypo@selma$ ls -l /proc/31565/fd/
lrwx------. 1 haypo haypo 64 13 avril 00:54 0 -> /dev/pts/0
lrwx------. 1 haypo haypo 64 13 avril 00:54 1 -> /dev/pts/0
lrwx------. 1 haypo haypo 64 13 avril 00:54 2 -> /dev/pts/0
lr-x------. 1 haypo haypo 64 13 avril 00:54 3 -> /dev/null
l-wx------. 1 haypo haypo 64 13 avril 00:54 4 -> pipe:[697347]

File descriptors of the second child process:

haypo@selma$ ls -l /proc/31566/fd/
lrwx------. 1 haypo haypo 64 13 avril 00:54 0 -> /dev/pts/0
lrwx------. 1 haypo haypo 64 13 avril 00:54 1 -> /dev/pts/0
lrwx------. 1 haypo haypo 64 13 avril 00:54 2 -> /dev/pts/0
lr-x------. 1 haypo haypo 64 13 avril 00:54 3 -> pipe:[697347]
lr-x------. 1 haypo haypo 64 13 avril 00:54 4 -> /dev/null
l-wx------. 1 haypo haypo 64 13 avril 00:54 5 -> pipe:[697348]

(...)

File descriptors of the last child process:

haypo@selma$ ls -l /proc/31574/fd/
lrwx------. 1 haypo haypo 64 13 avril 00:57 0 -> /dev/pts/0
lrwx------. 1 haypo haypo 64 13 avril 00:57 1 -> /dev/pts/0
lr-x------. 1 haypo haypo 64 13 avril 00:57 10 -> pipe:[697354]
lr-x------. 1 haypo haypo 64 13 avril 00:57 11 -> pipe:[697355]
lr-x------. 1 haypo haypo 64 13 avril 00:57 12 -> /dev/null
l-wx------. 1 haypo haypo 64 13 avril 00:57 13 -> pipe:[697356]
lrwx------. 1 haypo haypo 64 13 avril 00:54 2 -> /dev/pts/0
lr-x------. 1 haypo haypo 64 13 avril 00:57 3 -> pipe:[697347]
lr-x------. 1 haypo haypo 64 13 avril 00:57 4 -> pipe:[697348]
lr-x------. 1 haypo haypo 64 13 avril 00:57 5 -> pipe:[697349]
lr-x------. 1 haypo haypo 64 13 avril 00:57 6 -> pipe:[697350]
lr-x------. 1 haypo haypo 64 13 avril 00:57 7 -> pipe:[697351]
lr-x------. 1 haypo haypo 64 13 avril 00:57 8 -> pipe:[697352]
lr-x------. 1 haypo haypo 64 13 avril 00:57 9 -> pipe:[697353]

[pitrou]

This is expected with the "fork" method, but also happens with the "forkserver" method.

[pitrou]

New changeset 896145d by Antoine Pitrou in branch 'master':
bpo-26732: fix too many fds in processes started with the "forkserver" method (bpo-2813)
896145d

[vstinner]

The newly added test now hangs on almost all Windows buildbots, see:
http://bugs.python.org/issue31009#msg299050

[vstinner]

By the way, 3.6 is marked as affected, but the change wasn't backported to 3.6?

[quick-b]

Thank you for the fix, pitrou!

Regarding your comment regarding the behavior being expected with the "fork" method, I would suggest a documentation modification in the description of the fork method to add something to the effect of "All resources of the parent are inherited by the child process; this includes the ability of later children to act as the parent of earlier children via the various functions in this package."

[vstinner]

Antoine: Do you want to backport your change 896145d to Python 3.6? If yes, don't forget to include my fix 302bbbe for Windows (bpo-31009).

If you prefer, I can handle the backport.

[pitrou]

Victor, you can try to backport the fix, it should be interesting, though I suspect you will run into conflicts :-)

[pitrou]

Kevin:

Regarding your comment regarding the behavior being expected with the "fork" method, I would suggest a documentation modification in the description of the fork method to add something to the effect of "All resources of the parent are inherited by the child process; this includes the ability of later children to act as the parent of earlier children via the various functions in this package."

I would rather keep the latter part ("the ability of later children to act as the parent of earlier children via the various functions in this package") an implementation detail rather than a documented feature. Who knows, perhaps some day we'll change the implementation so that those are not inherited anymore.

[holdenweb]

IIRC sockets can be marked as inheritable or not. It seems to me it would be a useful enhancement to allow the same determination for the pipes.

[quick-b]

Hi Antoine,

... an implementation detail rather than a documented feature

I understand your desire, but this is a leaky abstraction and I would still suggest that it should be more clear that the fork method will inherit *all* resources from the parent, including multiprocessing internal resources related to other processes. This is potentially important if there are (a) large numbers of processes to be created and (b) for security aspects.

Regards,
Kevin

[iritkatriel]

Backport to 3.6 is no longer relevant. Is there anything left to do here?