sys.stdout fails to use default encoding as advertised

[stevendaprano]

BPO	4947
Nosy	@malemburg, @birkenfeld, @pitrou, @vstinner, @ezio-melotti, @merwok, @stevendaprano, @ssbarnea
Files	file_write-2.7-v3.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/birkenfeld'
closed_at = <Date 2010-09-08.10:51:52.849>
created_at = <Date 2009-01-14.11:18:47.856>
labels = ['interpreter-core', 'type-bug', 'docs']
title = 'sys.stdout fails to use default encoding as advertised'
updated_at = <Date 2010-09-08.10:51:52.848>
user = 'https://github.com/stevendaprano'

bugs.python.org fields:

activity = <Date 2010-09-08.10:51:52.848>
actor = 'vstinner'
assignee = 'georg.brandl'
closed = True
closed_date = <Date 2010-09-08.10:51:52.849>
closer = 'vstinner'
components = ['Documentation', 'Interpreter Core']
creation = <Date 2009-01-14.11:18:47.856>
creator = 'steven.daprano'
dependencies = []
files = ['18524']
hgrepos = []
issue_num = 4947
keywords = ['patch']
message_count = 12.0
messages = ['79849', '79852', '79887', '87635', '87637', '87638', '113890', '113891', '113892', '113900', '113901', '115857']
nosy_count = 14.0
nosy_names = ['lemburg', 'georg.brandl', 'ggenellina', 'pitrou', 'vstinner', 'hongqn', 'ezio.melotti', 'eric.araujo', 'steven.daprano', 'zuo', 'ssbarnea', 'cvrebert', 'srid', 'denilsonsa']
pr_nums = []
priority = 'high'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue4947'
versions = ['Python 2.6', 'Python 2.7']

[stevendaprano]

Documentation for files states that when writing to a file, unicode
strings are converted to byte strings using the encoding specified by
file.encoding.
http://docs.python.org/library/stdtypes.html#file.encoding

sys.stdout is a file, but it does not behave as stated above:

>>> type(sys.stdout)
<type 'file'>
>>> sys.stdout.encoding
'UTF-8'
>>> u = u"\u554a"
>>> print u
啊
>>> sys.stdout.write(u)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character u'\u554a' in 
position 0: ordinal not in range(128)
>>> sys.stdout.write(u.encode('utf-8'))
啊

[pitrou]

It probably uses sys.getdefaultencoding() instead.

[ggenellina]

It probably uses sys.getdefaultencoding() instead.

That would be wrong too, according to the cited documentation.

file.encoding is a read only attribute; it can be set in C code using
PyFile_SetEncoding. Apart from its definition in fileobject.c, it is
*only* used by PyInitialize when sys.stdin/stdout/stderr are created.
There are no tests, nor any other use of it anywhere. Apparently the
attribute *is* checked when writing unicode objects, but it does not
work.

I'm guessing now, but probably the original intent was to make file
objects behave like the wrapper returned by codecs.open works now --
later it was deemed impractical and forgotten. Now, the "declarative"
meaning of file.encoding survives, but the "behavior" is broken.

I don't know what would be the right thing to do. The encoding used by
stdin/stdout/stderr is valuable information so the attribute should
remain. Fixing the behavior is like having a crippled
StreamReaderWriter and I don't see the point. But StreamReaderWriter
has an "encoding" attribute too, and it "works", so one cannot rely on
having such attribute to know whether the stream automatically encodes
its data or not.

[zuo]

The matter had been discussed (and not once...), IMO without
satisfactory conclusion -- see:

• http://bugs.python.org/issue612627 (the feature added)
• http://bugs.python.org/issue1214889 (another feature rejected)
• http://bugs.python.org/issue1099364 (problems reported)
• http://bugs.python.org/issue967986 (problems reported)
• http://mail.python.org/pipermail/python-list/2008-December/693601.html
• http://mail.python.org/pipermail/python-dev/2008-December/084362.html
• and probably in many other places...

Anyway, it's definitely a bug -- either in the language/implementation
or in the documentation.

[zuo]

PS. The main problem is not a lack of feature but that inconsistency,
and that's not documented if File type docs:

print >>my_file, my_unicode # <- is encoded with my_file.encoding
my_file.write(my_unicode) # <- is encoded with my_file.encoding

# and on the other hand:
print my_unicode -- works # <- is encoded with my_file.encoding
sys.stdout.write(my_unicode) # <- is encoded with what is returned by
sys.getdefaultencoding()

[zuo]

s / if File / in File
s / -- works # <- is encoded with my_file.encoding / # <- is encoded
with sys.stdout.encoding

(sorry, too little sleep)

[vstinner]

Attached patch fixes this old and annoying issue. The issue only concerns sys.std* files, because Python only set the encoding and errors attributes for these files.

[vstinner]

Oh, I forgot to write that my patch uses also the errors attribute. Update the patch to add tests on errors: file_write-2.7-v2.patch.

[pitrou]

Your patch threatens to break compatibility. I think it would be better to simply change the "encoding" and "errors" attributes of standard streams.

[vstinner]

Your patch threatens to break compatibility

Yes it does. But I think that nobody relies on this bug. If your terminal uses something that utf-8, you will see strange characters if you write something else than ascii characters. I supopse that anybody facing this problem uses a workaround like replacing sys.stdout object, encode manually each string with the right encoding or something else.

[vstinner]

3rd version of the patch: accept character buffer objects without reencoding them. Add also tests on character buffer objects.