subprocess: optional auto-reaping fixing os.wait() lossage

[yorick]

BPO	1187312
Nosy	@loewis
Files	subprocess.patch: subprocess.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-03-26.17:08:48.160>
created_at = <Date 2005-04-21.11:40:17.000>
labels = ['library']
title = 'subprocess: optional auto-reaping fixing os.wait() lossage'
updated_at = <Date 2020-03-26.17:08:48.160>
user = 'https://bugs.python.org/yorick'

bugs.python.org fields:

activity = <Date 2020-03-26.17:08:48.160>
actor = 'Justin.Lebar'
assignee = 'none'
closed = True
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2005-04-21.11:40:17.000>
creator = 'yorick'
dependencies = []
files = ['6629']
hgrepos = []
issue_num = 1187312
keywords = ['patch']
message_count = 5.0
messages = ['48250', '48251', '48252', '48253', '365096']
nosy_count = 3.0
nosy_names = ['loewis', 'yorick', 'Justin.Lebar']
pr_nums = []
priority = 'normal'
resolution = 'out of date'
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue1187312'
versions = ['Python 2.4']

[yorick]

The subprocess module automatically reaps child
processes. It maintains a list of Popen instances, and
each time a new Popen is created, the list is traversed
and a non-polling wait is done for each instance.

I discussed this with the author, Peter Åstrand, and
this behaviour was inherited from the older popen2
code, and is intended to avoid a limitless accretion of
zombies when the user does not take care to wait for
the processes.

However, the auto-reaping interacts badly with
os.wait()/waitpid() since the user is not aware that
the module is reaping children behind her back. In
particular, os.wait(), which is very useful when a
blocking wait for many children is desired, may not
work at all, which caused me to look at the problem in
the first case.

The solution is to allow the user to create Popen
instances that are not auto-reaped. The interface is
otherwise unchanged, and existing code will see no
change in behaviour.

This patch does three things:

• Adds an autoreap parameter to the Popen constructor,
  defaulting to True (the previous behaviour)
• Documents the auto-reaper and its interaction with
  os.wait()/waitpid(), which was previously missing
• Changes the list of instances to a set, to avoid O(N)
  element removal.

For completeness, here is a test case:

import os, subprocess, time
p = subprocess.Popen(["/bin/true"]).pid
time.sleep(1)
subprocess.call(["/bin/false"])
(pid, status) = os.wait()
print "got", pid, "expected", p

The above code will throw an exception. With the patch,
it will work as expected if autoreap=False is added to
the Popen call.

[yorick]

Logged In: YES
user_id=432579

and a non-polling wait is done for each instance.

Sorry, this should be "non-blocking wait".

[yorick]

Logged In: YES
user_id=432579

Revised patch, using a dict instead of a set (for
compatibility with python 2.2, following PEP-291), and
rename autoreap parameter to "autowait", after discussion
with Peter Åstrand.

[loewis]

Logged In: YES
user_id=21627

This has been fixed in the subversion trunk in a different
way: processes are added to _active only inside __del__. So
as long as the application keeps a reference to the
subprocess object, it can wait for it; auto-reaping only
starts when the last reference was dropped. If you still see
a problem in that approach, please submit a new patch
(relative to svn trunk, preferably).

Marking this one as outdated.

[JustinLebar]

15 years later, it seems asyncio subprocesses may have the same issue. :)

https://bugs.python.org/issue40078