subprocess.Popen pipeline example code in the documentation is lacking

[stevenkwong]

BPO	7678
Nosy	@gpshead, @merwok

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/gpshead'
closed_at = <Date 2011-02-05.21:49:27.952>
created_at = <Date 2010-01-11.21:26:43.924>
labels = ['type-bug', 'docs']
title = 'subprocess.Popen pipeline example code in the documentation is lacking'
updated_at = <Date 2011-02-05.21:49:27.951>
user = 'https://bugs.python.org/stevenkwong'

bugs.python.org fields:

activity = <Date 2011-02-05.21:49:27.951>
actor = 'gregory.p.smith'
assignee = 'gregory.p.smith'
closed = True
closed_date = <Date 2011-02-05.21:49:27.952>
closer = 'gregory.p.smith'
components = ['Documentation']
creation = <Date 2010-01-11.21:26:43.924>
creator = 'steven.k.wong'
dependencies = []
files = []
hgrepos = []
issue_num = 7678
keywords = []
message_count = 4.0
messages = ['97606', '124657', '127787', '128028']
nosy_count = 5.0
nosy_names = ['gregory.p.smith', 'eric.araujo', 'steven.k.wong', 'docs@python', 'rosslagerwall']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'needs patch'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue7678'
versions = ['Python 3.1', 'Python 2.7', 'Python 3.2']

[stevenkwong]

The example code at http://www.python.org/doc/2.6.2/library/subprocess.html#replacing-shell-pipeline should be updated to add the close() call noted below:

output=dmesg | grep hda
==>
p1 = Popen(["dmesg"], stdout=PIPE)
p2 = Popen(["grep", "hda"], stdin=p1.stdout, stdout=PIPE)
p1.stdout.close() # <----- this line is not in the doc, but we should add it
output = p2.communicate()[0]

The parent process does not use p1.stdout after passing it to p2, so it should close p1.stdout after that. If it is not closed, there is a possibility for p1's process to never terminate if p2's process fails. The following code demonstrate this situation:

from subprocess import Popen, PIPE
import time
# p1 is a program that writes lots of data to the pipe
cmd = ['cat', 'a_large_file']
p1 = Popen(cmd, stdout=PIPE)
# p2 is a program that fails without reading much data from the pipe
cmd = ['python', '-c', 'import time; time.sleep(5); asdf']
p2 = Popen(cmd, stdin=p1.stdout, stdout=PIPE)
output = p2.communicate()[0]
while p1.poll() is None:  # it loops forever
    print 'sleep a bit'
    time.sleep(1)

[merwok]

As a non-expert user of subprocess, calling close before communicate seems strange to me. Does the example code with a bug work if close is called after communicate?

In the 3.2 docs, we could update the example to use a with statement, unless it is deemed a distraction for this simple introductory section.

[rosslagerwall]

The docs should be updated. This has been noted in msg54949 and http://www.enricozini.org/2009/debian/python-pipes/

Perhaps this example will make it clear:
import subprocess

p1 = subprocess.Popen(["yes"], stdout=subprocess.PIPE)
p2 = subprocess.Popen(["head"], stdin=p1.stdout, stdout=subprocess.PIPE)
#p1.stdout.close()
p1.wait()

This example hangs. "yes" writes to "head" and head reads the first 10 lines and then exits. But, "yes" does not receive a SIGPIPE because the python process still has a p1.stdout open. Thus, p1.stdout should be closed after being passed to p2.

[gpshead]

documentation updated in r88352. thanks!