[Windows] Python 2 mishandles console code page after setlocale

[segevfiner]

BPO	34283
Nosy	@pfmoore, @tjguk, @zware, @segevfiner

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-04-26.03:19:09.051>
created_at = <Date 2018-07-30.17:49:31.737>
labels = ['type-bug', 'OS-windows']
title = '[Windows] Python 2 mishandles console code page after setlocale'
updated_at = <Date 2020-04-26.03:19:09.051>
user = 'https://github.com/segevfiner'

bugs.python.org fields:

activity = <Date 2020-04-26.03:19:09.051>
actor = 'zach.ware'
assignee = 'none'
closed = True
closed_date = <Date 2020-04-26.03:19:09.051>
closer = 'zach.ware'
components = ['Windows']
creation = <Date 2018-07-30.17:49:31.737>
creator = 'Segev Finer'
dependencies = []
files = []
hgrepos = []
issue_num = 34283
keywords = []
message_count = 3.0
messages = ['322683', '322689', '367291']
nosy_count = 4.0
nosy_names = ['paul.moore', 'tim.golden', 'zach.ware', 'Segev Finer']
pr_nums = []
priority = 'normal'
resolution = 'out of date'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue34283'
versions = ['Python 2.7']

[segevfiner]

Found by trying to use pip: pypa/pip#5665.

This is likely affected by the console code page.

Python version: 2.7.15 64 bit
OS: Windows 10.0.17134.165 x64
The console locale is set to cp872.
The console font is consolas.

Apparently, msvcrt does charset conversion when writing to its file descriptors based on the set locale! and it's even special cased to handle the OEM console code page (You can see this in crt/src/write.c:_write_nolock if you have MSVC 2008).

When the "C" locale is set, no conversion is done. Python encodes to the OEM code page, and it passes through to the console unscathed. But once you do setlocale than the CRT expects you to use the ANSI code page, but Python will be encoding to the OEM code page which will result in this error from fwrite.

file.encoding in Python 2 is also not settable directly from Python (C API only), it's only used for stdio and set internally on startup: Python/pythonrun.c:349-378.

I found this describing this: Why printf can display non-ASCII characters when “C” locale is used?.

    #!/usr/bin/env python2
    from __future__ import print_function
    import locale

    print(u' |\u2588')  # Works
    locale.setlocale(locale.LC_ALL, '')
    print(u' |\u2588')  # IOError: [Errno 42] Illegal byte sequence

[zooba]

I have no intention of changing how encodings work in 2.7. Use codecs.open if you want to specify text IO encoding.

Someone else may be willing to look at it.

[zware]

As this issue is specific to 2.7, I'm closing it.