stdout.encoding not set when redirecting windows command line output

[Redoute]

BPO	14192
Nosy	@loewis

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2012-03-04.12:05:48.594>
created_at = <Date 2012-03-04.09:42:29.739>
labels = ['type-bug', 'OS-windows']
title = 'stdout.encoding not set when redirecting windows command line output'
updated_at = <Date 2012-03-04.14:54:44.096>
user = 'https://bugs.python.org/Redoute'

bugs.python.org fields:

activity = <Date 2012-03-04.14:54:44.096>
actor = 'loewis'
assignee = 'none'
closed = True
closed_date = <Date 2012-03-04.12:05:48.594>
closer = 'loewis'
components = ['Windows']
creation = <Date 2012-03-04.09:42:29.739>
creator = 'Redoute'
dependencies = []
files = []
hgrepos = []
issue_num = 14192
keywords = []
message_count = 4.0
messages = ['154885', '154888', '154892', '154898']
nosy_count = 2.0
nosy_names = ['loewis', 'Redoute']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue14192'
versions = ['Python 2.7']

[Redoute]

When running a python script from windows command line (cmd.exe) and redirecting its output, stdout.encoding is set to None and printing non-ascii chars fails. Encoding should be the same as without redirecting.
Example:

[Code unictest.py]
# -- coding: utf-8 --

from sys import stdout, stderr
print >> stderr, 'stdout.encoding: ', stdout.encoding
print u'äöüß'
[/Code]

[windows command prompt]
C:\Daten>testunic.py
stdout.encoding: cp850
äöüß

C:\Daten>testunic.py > testunic.txt
stdout.encoding:  None
Traceback (most recent call last):
  File "C:\Daten\Cmd\testunic.py", line 5, in <module>
    print u'├ñ├Â├╝├ƒ'
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-3: ordin
al not in range(128)
[/windows command prompt]

[loewis]

That is impossible to implement. When outputting to a terminal window, the terminal's encoding is queried and used. When the Python script is run on its own, there may not even exist a terminal encoding.

In any case, this issue is resolved in Python 3 (not by setting the encoding to the terminal's encoding, but to the system code page). For 2.7, changing this would be a new feature, so I'm closing this as resolved - please upgrade to Python 3.

[Redoute]

Martin, thanks for your response.

I have to take it, although I am not really convinced. The system codepage probably differs from the console codepage (here 1152 vs. 850). Print statements on the other hand are typical for console programs. Calling python.exe (vs. pythonw.exe) means console program means print output has to use console codepage as default.

It is disappointing that output changes and programs fail because of a command line redirection.

[loewis]

I have to take it, although I am not really convinced. The system
codepage probably differs from the console codepage (here 1152 vs.
850). Print statements on the other hand are typical for console
programs. Calling python.exe (vs. pythonw.exe) means console program
means print output has to use console codepage as default.

I think in practice, this statement is incorrect for most applications.
When they use print, they want to create text files, and those will
be opened with notepad (or some other text editor on Windows), and will
have to be in cp1252. So guessing that it should be cp850 will be incorrect
most of the time.

In any case, you have perfectly described why the initial design decision
in Python 2 was to refuse to guess.