Better support for pipe I/O encoding in subprocess

[ncoghlan]

BPO	13442
Nosy	@ncoghlan, @vstinner
Superseder	bpo-6135: subprocess seems to use local encoding and give no choice

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2011-11-21.01:32:25.909>
created_at = <Date 2011-11-21.01:22:48.293>
labels = ['docs']
title = 'Better support for pipe I/O encoding in subprocess'
updated_at = <Date 2011-11-21.01:32:25.908>
user = 'https://github.com/ncoghlan'

bugs.python.org fields:

activity = <Date 2011-11-21.01:32:25.908>
actor = 'ncoghlan'
assignee = 'none'
closed = True
closed_date = <Date 2011-11-21.01:32:25.909>
closer = 'ncoghlan'
components = ['Documentation']
creation = <Date 2011-11-21.01:22:48.293>
creator = 'ncoghlan'
dependencies = []
files = []
hgrepos = []
issue_num = 13442
keywords = []
message_count = 3.0
messages = ['148022', '148023', '148024']
nosy_count = 3.0
nosy_names = ['ncoghlan', 'vstinner', 'docs@python']
pr_nums = []
priority = 'normal'
resolution = 'duplicate'
stage = 'resolved'
status = 'closed'
superseder = '6135'
type = None
url = 'https://bugs.python.org/issue13442'
versions = ['Python 3.3']

[ncoghlan]

Currently, pipes in the subprocess module work strictly with bytes I/O, *unless* you set "universal newlines=True". In that case, it assumes an output encoding of UTF-8 for stdout and stderr and applies universal newlines process.

When stdin/out/err are remapped to ordinary I/O streams then 'encoding' and 'errors' can be specified as usual, but it is currently challenging to do this for pipes. Since they're created internally by the subprocess module, user code doesn't get the opportunity to wrap them when using the convenience APIs. When using Popen objects, you have to create the object, then wrap each stream individually (rebinding the attributes as you go).

My suggestion is that we add a new option for the stdin/out/err arguments:

    class TextPipe:
        def __init__(self, encoding, errors='strict'):
            self.encoding = encoding
            self.errors = errors

So to read UTF-8 encoded data from a subprocess, you could just do:

    data = check_stdout(cmd, stdout=TextPipe('utf-8'), stderr=STDOUT)

There are at least a couple of other alternatives here:

• separate out the pipe creation logic from the Popen logic so it is possible to create and wrap the pipe objects explicitly and then pass the wrapped pipe object to the subprocess invocation APIs. 'TextPipe' would then actually be such a wrapped pipe, rather than merely instructions to tell Popen what kind of pipe to create.
• instead of adding 'TextPipe', just re-use the PIPE name (with the class itself still being used as a marker constant to request implicit creation of a binary PIPE)

[vstinner]

This issue looks as a duplicate of bpo-6135.

[ncoghlan]

Indeed, I'll add my suggestions over there.