subprocess seems to use local encoding and give no choice

[mark-summerfield]

BPO	6135
Nosy	@gpshead, @amauryfa, @ncoghlan, @pitrou, @vstinner, @mark-summerfield, @merwok, @bitdancer, @mightyiam, @andyclegg, @cjerdonek, @vadmium, @eryksun, @zooba, @davispuh
PRs	bpo-6135: Fix subprocess.check_output() doc to mention changes in 3.6 #5564 [3.7] bpo-6135: Fix subprocess.check_output doc to mention changes in 3.6 (GH-5564) #5572 [3.6] bpo-6135: Fix subprocess.check_output doc to mention changes in 3.6 (GH-5564) #5573
Files	subprocess.patch: Add encoding and errors to subprocess.Popen. Based against trunk. subprocess3.patch: Add encoding and errors to subprocess.Popen. Based against py3k. test_subprocess3.py.patch: unittests for encoding parameter subProcessTest.py: Workaround example. Calls itself as a sub-process. 6135_1.patch 6135_2.patch 6135_3.patch 6135_4.patch 6135_5.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/zooba'
closed_at = <Date 2016-09-07.03:18:30.672>
created_at = <Date 2009-05-28.07:08:42.275>
labels = ['type-feature', 'library']
title = 'subprocess seems to use local encoding and give no choice'
updated_at = <Date 2018-02-07.02:11:33.939>
user = 'https://github.com/mark-summerfield'

bugs.python.org fields:

activity = <Date 2018-02-07.02:11:33.939>
actor = 'gregory.p.smith'
assignee = 'steve.dower'
closed = True
closed_date = <Date 2016-09-07.03:18:30.672>
closer = 'steve.dower'
components = ['Library (Lib)']
creation = <Date 2009-05-28.07:08:42.275>
creator = 'mark'
dependencies = []
files = ['14293', '14294', '14295', '28760', '44386', '44389', '44398', '44404', '44408']
hgrepos = []
issue_num = 6135
keywords = ['patch']
message_count = 66.0
messages = ['88466', '89094', '89293', '89322', '89325', '89332', '89333', '97090', '97092', '97093', '111181', '111466', '123020', '123024', '148025', '148066', '148484', '148494', '148495', '148496', '148498', '168191', '168213', '180157', '265785', '265793', '265812', '265822', '274467', '274468', '274469', '274481', '274489', '274492', '274497', '274509', '274510', '274517', '274518', '274519', '274561', '274562', '274566', '274574', '274642', '274644', '274646', '274648', '274649', '274651', '274653', '274665', '274699', '274736', '274737', '282291', '282294', '282298', '304049', '304060', '304118', '304119', '304126', '311756', '311757', '311760']
nosy_count = 20.0
nosy_names = ['gregory.p.smith', 'amaury.forgeotdarc', 'ncoghlan', 'pitrou', 'vstinner', 'mark', 'eric.araujo', 'segfaulthunter', 'Arfrever', 'r.david.murray', 'srid', 'mightyiam', 'andrewclegg', 'chris.jerdonek', 'python-dev', 'martin.panter', 'eryksun', 'steve.dower', 'berwin22', 'davispuh']
pr_nums = ['5564', '5572', '5573']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue6135'
versions = ['Python 3.6']

[mark-summerfield]

When I start a process with subprocess.Popen() and pipe the stdin and
stdout, it always seems to use the local 8-bit encoding.

I tried setting process.stdin.encoding = "utf8" and the same for stdout
(where process is the subprocess object), but to no avail.

I also tried using shell=True since on Mac, Terminal.app is fine with
Unicode, but that didn't work.

So basically, I have programs that output Unicode and run fine on the
Mac terminal, but that cannot be executed by subprocess because
subprocess uses the mac_roman encoding instead of Unicode.

I wish it were possible to specify the stdin and stdout encoding that is
used; then I could use the same one on all platforms. (But perhaps it is
possible, and I just haven't figured out how?)

[srid]

Related discussion thread: https://answers.launchpad.net/bzr/+question/63601

[amauryfa]

I propose to add two parameters (encoding, error) to the
subprocess.Popen function.

• python 2.x could build and return codecs.StreamReader objects
• python 3.x would just pass these parameters to io.TextIOWrapper

I'll try to come with a patch.

[segfaulthunter]

I wrote a patch to add encoding and error to subprocess.Popen in Python
2.7 (trunk).

[segfaulthunter]

Cosmetic update.

[pitrou]

Two things:

1. The argument should be called errors for consistency with open()
   and TextIOWrapper(), not error
2. You should add some unit tests.

[segfaulthunter]

Should we also cover the unusual case where stdout, stderr and stdin
have different encodings, because now we are assuming the are all the same.

[mark-summerfield]

I agree with Florian Mayer that the encoding handling should be
stream-specific. You could easily be reading the stdout of some third
party program that uses, say, latin1, but want to do your own output in,
say, utf-8.

One solution that builds on what Amaury Forgeot d'Arc has done (i.e.,
the encoding and errors parameters) by allowing those parameters to
accept either a single string (as now), or a dict with keys 'stdin',
'stdout', 'stderr'. Of course it is possible that the client might not
specify all the dict's keys in which case those would use the normal
default (local 8-bit etc.)

[amauryfa]

I don't understand. How is the subprocess stdout related to the main
program output?
Stream-specific encoding could be useful for subprocesses that expect
latin-1 from stdin but write utf-8 to stdout. I'm not sure we should
support this.

[mark-summerfield]

On Thu, Dec 31, 2009 at 1:30 PM, Amaury Forgeot d'Arc
<report@bugs.python.org> wrote:

Amaury Forgeot d'Arc <amauryfa@gmail.com> added the comment:

I don't understand. How is the subprocess stdout related to the main
program output?
Stream-specific encoding could be useful for subprocesses that expect
latin-1 from stdin but write utf-8 to stdout. I'm not sure we should
support this.

Yes, you're right.

(What I had in mind was a scenario where you read one process's stdout
and wrote to another process's stdin; but of course using your errors
& encoding arguments this will work because there'll be two separate
process objects each of which can have its encoding and errors set
separately.)

[BreamoreBoy]

Ran new unit test before and after patching subprocess on Windows Vista against 3.1 debug maintenance release, all ok apart from this at end of latter.

File "test\test_subprocess.py", line 568, in test_encoded_stderr
self.assertEqual(p.stderr.read(), send)
AssertionError: 'ï[32943 refs]\r\n' != 'ï'

I'm sure I've seen a ref to this somewhere, can anyone remember where?

[BreamoreBoy]

In 2.7 and py3k test_subprocess has a class BaseTestCase which has a assertStderrEqual method. These don't exist in 3.1. I believe that the py3k code will need to be backported to 3.1. Can this be done on this issue, or do we need a new one to keep things clean?

[vstinner]

About the topic:

subprocess seems to use local 8-bit encoding and gives no choice
I don't understand that: by default, Python 2 and Python 3 use byte strings, so there is no encoding (nor error handler).

I don't see how you can get unicode from a process only using subprocess. But with Python 3, you can get unicode if you set universal_newlines option to True.

So for Python 2, it's a new feature (get unicode), and for Python 3, it's a new option to specify the encoding. The title should be changed to something like "subprocess: add an option to specify stdin, stdout and/or stderr encoding and errors" and the type should be changed to "feature request".

Or am I completly wrong?

[vstinner]

... it always seems to use the local 8-bit encoding

The locale encoding is not necessary a 8-bit encoding, it can by a multibyte like... UTF-8 :-)

--

subprocess.patch: You should maybe use io.open(process.stdout.fileno(), encoding=..., errors=...) instead of codecs.getreader/getwriter. The code will be closer to Python 3. I think that the io module has a better support of unicode than codec reader/writer objects and a nicer API. See:
http://bugs.python.org/issue8796#msg106339

--

... allowing [encoding and errors] to accept either a single string
(as now), or a dict with keys 'stdin', 'stdout', 'stderr'

I like this idea. But what about other TextIOWrapper (or other file classes) options: buffer size, newline, line_buffering, etc.?

Why not using a dict for existing stdin, stdout and stderr arguments? Dummy example:

process = Popen(
   command,
   stdin={'file': PIPE, 'encoding': 'iso-8859-1', 'newline': False},
   stdout={'file': PIPE', 'encoding': 'utf-8', 'buffering': 0, 'line_buffering': False},
   ...)

If stdin, stdout or stderr is a dict: the default value of its 'file' key can be set to PIPE. I don't think that it's possible to choose the encoding, buffer size, or anything else if stdin, stdout or stderr is not a pipe.

With this solution, you cannot specify the encoding for stdin, stdout and stderr at once. You have at least to repeat the encoding for stdin and stdout (and use stderr=STDOUT).

--

I still hesitate to accept this feature request. Is it really needed to add extra arguments for TextIOWrapper? Can't the developer create its own TextIOWrapper object with all interesting options?

In Python 3, be able to specify stdout encoding is an interesting feature. Control the buffers size is also an interesting option.

My problem is maybe the usage of a dict to specify various options. I'm not sure that it is extensible to support future needs.

[ncoghlan]

I discovered this same problem recently when updating the subprocess docs, and also in working on the improved shell invocation support I am proposing for 3.3 (bpo-13238).

I initially posted an earlier variant this suggestion as a new issue (bpo-13442), but Victor redirected me here.

Firstly, I don't think it makes any sense to set encoding information globally for the Popen object. As a simple example, consider using Python to write a test suite for the iconv command line tool: there's only one Popen instance (for the iconv call), but different encodings for stdin and stdout.

Really, we want to be able to make full use of Python 3's layered I/O model, but we want the subprocess pipe instances to be slotted in at the lowest layer rather than creating them ourselves.

The easiest way to do that is to have a separate class that specifies the additional options for pipe creation and does the wrapping:

    class TextPipe:
        def __init__(self, *args, **kwds):
            self.args = args
            self.kwds = kwds
        def wrap_pipe(self, pipe):
            return io.TextIOWrapper(pipe, *self.args, **self.kwds)

The stream creation process would then include a new "wrap = getattr(stream_arg, 'wrap_pipe', None)" check that is similar to the existing check for subprocess.PIPE, but invokes the method to wrap the pipe after creating it.

So to read UTF-8 encoded data from a subprocess, you could just do:

    data = check_stdout(cmd, stdout=TextPipe('utf-8'), stderr=STDOUT)

[pitrou]

Firstly, I don't think it makes any sense to set encoding information
globally for the Popen object. As a simple example, consider using
Python to write a test suite for the iconv command line tool: there's
only one Popen instance (for the iconv call), but different encodings
for stdin and stdout.

Isn't that the exception rather than the rule? I think it actually makes
sense, in at least 99.83% of cases ;-), to have a common encoding
setting for all streams.
(I'm not sure about the "errors" setting, though: should we use strict
for stdin/stdout and backslashreplace for stderr, as the interpreter
does?)

Perhaps the common case should be made extra easy.

[merwok]

Is subprocess affected by PYTHONIOENCODING?

[vstinner]

Is subprocess affected by PYTHONIOENCODING?

Yes, as any Python process.

[merwok]

So the users can control the encoding, and this is a doc bug.

[bitdancer]

If you decide this is only a doc bug, please see also related bpo-12832.

[pitrou]

So the users can control the encoding, and this is a doc bug.

Not really. People can control the encoding in the child process (and only if it's a Python 3 process of course).
They can't control the encoding in the parent's subprocess pipes and that's what the request (& patch) is about.

[cjerdonek]

> only one Popen instance (for the iconv call), but different encodings
> for stdin and stdout.
Isn't that the exception rather than the rule? I think it actually makes
sense, in at least 99.83% of cases ;-), to have a common encoding
setting for all streams.

FWIW, I recently encountered a scenario (albeit in a test situation) where the ability to set different encodings for stdout and stderr would have been useful to me. It was while creating a test case for bpo-15595. I was changing the locale encoding for stdout, but I also wanted to leave it unchanged for stderr because there didn't seem to be a way to control the encoding that the child used for stderr.

[cjerdonek]

To my previous comment, bpo-15648 shows the case where I was able to change the encoding for stdout in the child process but not stderr (which would require supporting two encodings in Popen to handle).

[berwin22]

I've found a workaround by specifying the enviroment variable:

my_env = os.environ
my_env['PYTHONIOENCODING'] = 'utf-8'
p = subprocess.Popen(shlex.split(command), stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE, env=my_env)

I've attached an example script for testing. It calls itself recursively 10 times.
Pleased note the 'fix' variable.

[vadmium]

This seems like a request for extra feature(s), rather than a bug report.

FWIW I agree with Victor’s hesitation in <https://bugs.python.org/issue6135#msg123024\>. I doubt it is worth adding special support to have multiple pipes that all use text mode (universal_newlines=True) but with different encoding settings. If you really want this, just add an external TextIOWrapper, or string encoding:

process = Popen(command, stdin=PIPE, stdout=PIPE, ...)

# Manually wrap with TextIOWrapper. Probably risks deadlocking in the general case, due to buffering, so this is a bad idea IMO.
input_writer = io.TextIOWrapper(process.stdin, "iso-8859-1")
output_reader = io.TextIOWrapper(process.stdout, "utf-8")

# Better: use communicate(), or a custom select() loop or similar:
input_string = input_string.encode("iso-8859-1")
[output_string, _] = process.communicate(input_string)
output_string = output_string.decode("utf-8")

Also I’m not enthusiastic about adding encoding etc parameters for a single PIPE scenario. What is wrong with leaving it in binary mode in the Popen call, and encoding, decoding, or using TextIOWrapper as a separate step?

[vstinner]

Martin Panter added the comment:

Also, should encoding=... or errors=... be an error if universal_newlines=False (the default)?

Right. But if encoding or errors is used, universal_newlines value
should be set automatically to True.

For example, I expect Unicode when writing:

output = subprocess.call(cmd, stdout=subprocess.PIPE, encoding='utf8').stdout

[vstinner]

Nick Coghlan added the comment:

a. How do folks feel about providing a new "text" parameter to replace the cryptic "universal_newlines=True" that would explicitly be equivalent to "universal newlines with sys.getdefaultencoding()"?

If it's just text=True, I don't see the point of having two options
with the same purpose.

b. Given (a), what if the new "text" parameter also accepted a new "subprocess.TextConfig" object in addition to the base behaviour of doing a plain bool(text) check to activate the defaults?

Can you give an example? I don't see how the API would be used.

[zooba]

Given specifying an encoding will do the same thing as universal_newlines would have, should I just "hide" references to universal_newlines in the doc? (i.e. only mention it under a versionchanged banner, rather than front-and-centre)

[ncoghlan]

Ah, excellent point about "encoding='utf-8'" already being a less cryptic replacement for universal_newlines=True, so consider my questions withdrawn :)

As far as universal_newlines goes, that needs to remain documented for the sake of folks reading code that uses it, and folks that need compatibility with older versions, but +1 for recommending specifying an encoding over using that flag for new 3.6+ code.

[vstinner]

Steve Dower added the comment:

Given specifying an encoding will do the same thing as universal_newlines would have, should I just "hide" references to universal_newlines in the doc? (i.e. only mention it under a versionchanged banner, rather than front-and-centre)

No. Don't hide universal_newlines, it's different than encoding.
universal_newlines uses the locale encoding which is a good choice in
most cases.

[zooba]

universal_newlines uses the locale encoding which is a good choice in
most cases.

Well, some cases, and only really by accident. I won't hide the parameter, but it is promoted as "frequently used" and I'll make sure to document encoding before universal_newlines. Call it a very weak deprecation.

The new patch also removes the openers, which I think were solving a problem we don't have right now.

[zooba]

Addressed more feedback.

[vstinner]

6135_4.patch: Hum, what if you only set errors? I suggest to use TextIOWrapper but use the locale encoding:

if (universal_newlines or errors) and not encoding:
  encoding = getpreferredencoding(False)

[zooba]

Sure, that's easy enough. Any other concerns once I've made that change?

[vstinner]

Steve Dower added the comment:

Sure, that's easy enough. Any other concerns once I've made that change?

If errors enables Unicode and the doc is updated, the patch will LTGM :-)

[eryksun]

Why do you need to call getpreferredencoding()? Isn't that already the default if you call TextIOWrapper with encoding as None? For example:

    text_mode = encoding or errors or universal_newlines

    self.stdin = io.open(p2cwrite, 'wb', bufsize)
    if text_mode:
        self.stdin = io.TextIOWrapper(self.stdin, write_through=True,
                                      line_buffering=(bufsize == 1),
                                      encoding=encoding, errors=errors)

[vstinner]

Why do you need to call getpreferredencoding()?

I proposed to do that, but I prefer your simple flag:

text_mode = encoding or errors or universal_newlines

Steve: please use this simpler flag to avoid TextIOWrapper details in subprocess.py ;-)

[zooba]

Steve: please use this simpler flag to avoid TextIOWrapper details in subprocess.py

The TextIOWrapper details are already specified in multiple places in the documentation. Should we remove all of those and write it more like:

"if *encoding*, *errors* or *universal_newlines* are specified, the streams will be opened in text mode using TextIOWrapper."

That way we make the class part of the interface, but not the interpretation of the arguments. (Currently it's a big mess of encodings and newline transformations.)

[zooba]

More doc updates - shouldn't be a difficult review this time, but I always like getting multiple opinions on doc changes.

[vadmium]

Maybe good to adjust the other mentions of universal_newlines, e.g. for check_output(). The Posix version of the multiple-pipe _communicate() method probably needs adjusting too. Test case:

>>> check_output(("true",), encoding="ascii", input="")  # Should be text
b''
>>> check_output(("true",), encoding="ascii")  # Correct result
''

[python-dev]

New changeset 720f0cf580e2 by Steve Dower in branch 'default':
Issue bpo-6135: Adds encoding and errors parameters to subprocess
https://hg.python.org/cpython/rev/720f0cf580e2

[zooba]

I also added more tests, specifically for ansi and oem encodings on Windows and the errors argument on all platforms.

[davispuh]

Still can't specify encoding for getstatusoutput and getoutput. Also it uses wrong encoding by default, most of time it will be console's codepage instead of ANSI codepage which is used now.

[davispuh]

And looks like only way to specify console's codepage is with
encoding=os.device_encoding(2) which will have to be used for 99% of cases. I don't see other way...

[zooba]

looks like only way to specify console's codepage is with
encoding=os.device_encoding(2) which will have to be used for 99% of cases

That's true, but it only applies when the subprocess is using the same call (GetConsoleCP) to determine what encoding to use to write to something *other* than the console, which is incorrect. The defaults in this cause ought to be ACP, but it depends entirely on the target application and there is no good "99%" default.

To see this demonstrated, check the difference between these two commands (with 3.5 or earlier):

python -c "import sys; print(sys.stdin.encoding)"
python -c "import sys; print(sys.stdin.encoding)" < textfile.txt

When stdin is redirected, the default encoding is not the console encoding (unless it happens to match your ACP). And ACP is not the correct encoding for a file anyway - it just happens to be the default for TextIOWrapper - so to do it properly you need to detect that it's not a console (sys.stdin.isatty()) and then use the raw stream rather than the default wrapper. (Or you specify that your program requires ACP files, or that PYTHONIOENCODING should be set, or that it requires any other encoding and you explicitly switch to that one.)

In short, character encoding is hard, and the only way to get it right is for the developer to be aware of it. No good defaults exist.

[andyclegg]

The commit for this bug (720f0cf580e2) introduces encoding and errors arguments but doesn't actually document what the values of these should be. In the case of the encoding it could be reasonably guessed, but the only way to determine what the value of 'errors' should be is to make the logical leap and look at the TextIOWrapper doc page.

And in reply to message bpo-274510, there certainly should be a 'text=True' argument added. There are countless use cases where text rather than bytes is the expected behaviour, and currently this has to be triggered by using a more obscure option. In any case, it could be argued that text should be returned when shell=True since this mimics the behaviour of any shell program.

(Sorry if adding a comment to a closed bug is poor etiquette; it seemed like the best place to put this comment)

[zooba]

The commit for this bug (720f0cf580e2) introduces encoding and errors arguments but doesn't actually document what the values of these should be

Do you mean it doesn't document that they take the normal encoding/errors arguments that all the other functions that do character encoding take? Or they don't specify which encoding you should use? The latter is not feasible, and if you mean the former then feel free to open a new issue for documentation.

Having a single "text" parameter is essentially what we had before - a magic option that modifies your data without giving you any ability to do it correctly. This change brought Popen in line with all the other places where we translate OS-level bytes into Python str, and it's done it in exactly the same way.

Also see msg274562 where Nick withdraws his own suggestion of a "text" parameter.

[andyclegg]

I meant the former; I'll look a bit more at the documentation and submit an issue/patch.

As regards the 'text' flag - universal_newlines is actually exactly that already. I've just checked the code of subprocess.py and the universal_newlines argument is read in only two places:

• as one of the deciding factors in whether to use the text mode
• in a backwards compatibility clause in check_output

So subprocess *already* has a text mode and a 'magic option' to trigger it. It works well, and is useful in most cases. When the default encoding guess is incorrect, it can easily be corrected by supplying the correct encoding. This is a good situation!

What is not so good is the API. I'm teaching a Python course for scientists at the moment. Retrieving text from external processes is an extremely common use case. I would rather not teach them to just have to use 'encoding=utf-8', because arguably teaching a user to supply an encoding without knowing if it's correct is worse than the system guessing. Equally, teaching 'universal_newlines=True' is a bit obscure.

The way forward seems to be:

• Add a text=True/False argument that is essentially the same as universal_newlines, to improve the API.
• Clearly document that this implies that the encoding will be guessed, and that an explicit encoding can be given if the guess is wrong
• Optionally, and I have no strong feelings either way on this, remove/deprecate the universal_newlines argument

[ncoghlan]

This discussion should probably be submitted as a new RFE (requesting "text" as a more obvious and beginner friendly alias for universal_newlines), but I'll also add a note regarding precedent for a simple binary/text toggle: the mode settings for open().

There, the default is text (with the default encoding determined by the system), and you have to include "b" in the mode settings to say "I want raw binary".

For the subprocess APIs, the default is different (i.e. binary), so the natural counterpart to the "b" mode flag in open() would be an explicit "text" mode flag.

[andyclegg]

RFE submitted as bpo-31756 , thanks

[gpshead]

New changeset fc1ce81 by Gregory P. Smith (Brice Gros) in branch 'master':
bpo-6135: Fix subprocess.check_output doc to mention changes in 3.6 (GH-5564)
fc1ce81

[gpshead]

New changeset 4e7a964 by Gregory P. Smith (Miss Islington (bot)) in branch '3.7':
bpo-6135: Fix subprocess.check_output doc to mention changes in 3.6 (GH-5564) (GH-5572)
4e7a964

[gpshead]

New changeset 7f95c8c by Gregory P. Smith (Miss Islington (bot)) in branch '3.6':
bpo-6135: Fix subprocess.check_output doc to mention changes in 3.6 (GH-5564) (GH-5573)
7f95c8c