Reading from a killed shell script with popen* under linux

[boukthor]

BPO	1194328

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2006-02-23.19:02:49.000>
created_at = <Date 2005-05-03.09:44:06.000>
labels = ['invalid', 'library']
title = 'Reading from a killed shell script with popen* under linux'
updated_at = <Date 2006-02-23.19:02:49.000>
user = 'https://bugs.python.org/boukthor'

bugs.python.org fields:

activity = <Date 2006-02-23.19:02:49.000>
actor = 'boukthor'
assignee = 'none'
closed = True
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2005-05-03.09:44:06.000>
creator = 'boukthor'
dependencies = []
files = []
hgrepos = []
issue_num = 1194328
keywords = []
message_count = 4.0
messages = ['25220', '25221', '25222', '25223']
nosy_count = 1.0
nosy_names = ['boukthor']
pr_nums = []
priority = 'normal'
resolution = 'not a bug'
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue1194328'
versions = []

[boukthor]

These are problems I've run into under linux (Mandrake
10.0 through 2006.0, Debian 3.1 and Gentoo) with python
versions ranging from 2.3.3 to 2.4:

If you run this:

import os
pout = os.popen("/bin/sleep 999")
print pout.read()
print pout.close()

and kill the "sleep", you get the desired behaviour:
the read returns an empty string when the process gets
killed and the close returns the killing signal.
However, if you put the "sleep" command in a shell
script, for example this simple "/tmp/test.sh":

#!/bin/sh
sleep 999

and run the script through popen (or one of the popen*
method of the os and popen2 module)

import os
pout = os.popen("/tmp/test.sh")
print pout.read()
print pout.close()

then kill the sh running "test.sh", the read never
returns and the shell remains as a zombie. You can get
some strange behaviour with the Popen* classes too
(whether bypassing the shell intervention or not). For
example, run this (and kill the subshell):

import popen2
sub = popen2.Popen3("/tmp/test.sh")
print sub.wait()
print sub.fromchild.read()

this time the wait correctly returns the signal that
killed the shell and removes it from the process table,
but the read hangs again.

As an added oddity, if you run the popen command from
an interactive prompt and raise a KeyboardInterrupt by
pressing Ctrl-C before trying to read from the pipe,
you kill the subprocess with the SIGINT...

I have a final suggestion: if you try reading the
output of a popen* method with a "for" statement like this:

import os
pout = os.popen("for i in $(seq 1 10); do echo $i;
sleep 1; done")
for l in pout: print l,

it waits for the subprocess to complete before going
into the loop. To get the output as it is piped, you
have to use the poll method of the Popen* classes
(which is not available under OSes other than Unix):

sub = popen2.Popen3("for i in $(seq 1 10); do echo $i;
sleep 1; done")
while (-1 == sub.poll()): print sub.fromchild.readline()

I don't know if this last behaviour can be fixed or not
but I'd suggest some mention of this in the manual if
it can't.

[boukthor]

Logged In: YES
user_id=638508

Oops, just saw the report bpo-1180160. I missed it because I
searched for popen, not Popen before posting. My bad. I'll
cross-reference the two.

[boukthor]

Logged In: YES
user_id=638508

The report bpo-1180160 was only loosely related to the above
problem. In fact it is probably closer to the following :
761888 popen2.Popen3 and popen2.Popen4 leaks filedescriptors
892939 Race condition in popen2
998246 Popen3.poll race condition
1183780 Popen4 wait() fails sporadically with threads

NB : This bug is very incapacitating for me since I can't
figure any workaround and though I understand that
developers may have other priorities, I'd appreciate some
acknowledgement that somebody at least read this report...

[boukthor]

Logged In: YES
user_id=638508

This is not a bug after all...

When you run a command through a shell script, kill the
script and waiting for it to return, any subcommands of the
script get reattached to init until they complete. If these
subcommands didn't close them, they still have descriptors
to the popen pipe through which they may write data. It is
then normal that reading the pipe should block until
completion, not only of the command invoqued but of all its
subprocesses as well. Very annoying for my purpose, but
definitively not a python bug.

Oh well, I thinks I'll stop talking to myself, now. I must
say I'm a bit disappointed at the nonexistent support and
help I got from this python bugzilla, though...