Force console stdout to use UTF8 on Windows

[pfmoore]

BPO	23901
Nosy	@pfmoore, @vstinner, @tjguk, @bitdancer, @vadmium, @zware, @zooba
Superseder	bpo-1602: windows console doesn't print or input Unicode

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/pfmoore'
closed_at = <Date 2017-03-26.06:19:27.808>
created_at = <Date 2015-04-09.17:22:37.562>
labels = ['OS-windows']
title = 'Force console stdout to use UTF8 on Windows'
updated_at = <Date 2017-03-26.06:19:27.806>
user = 'https://github.com/pfmoore'

bugs.python.org fields:

activity = <Date 2017-03-26.06:19:27.806>
actor = 'martin.panter'
assignee = 'paul.moore'
closed = True
closed_date = <Date 2017-03-26.06:19:27.808>
closer = 'martin.panter'
components = ['Windows']
creation = <Date 2015-04-09.17:22:37.562>
creator = 'paul.moore'
dependencies = []
files = []
hgrepos = []
issue_num = 23901
keywords = []
message_count = 8.0
messages = ['240354', '240375', '240378', '240386', '240388', '240694', '241465', '290507']
nosy_count = 8.0
nosy_names = ['paul.moore', 'vstinner', 'tim.golden', 'r.david.murray', 'martin.panter', 'piotr.dobrogost', 'zach.ware', 'steve.dower']
pr_nums = []
priority = 'normal'
resolution = 'duplicate'
stage = 'resolved'
status = 'closed'
superseder = '1602'
type = None
url = 'https://bugs.python.org/issue23901'
versions = ['Python 3.5']

[pfmoore]

Console code page issues are a consistent source of problems on Windows. It would be nice, given that the Windows console has Unicode support, if Python could write the full range of Unicode to the console by default.

The MSVC runtime appears to have a flag that can be set via _setmode(), _O_U8TEXT, which "enables Unicode mode" (see https://msdn.microsoft.com/en-us/library/tw4k6df8%28v=vs.100%29.aspx?f=255&MSPPError=-2147217396, in particular the second example). It seems as if Python could set U8TEXT mode on sys.stdout on startup (assuming it's a console) and set the encoding to UTF8, and then Unicode output would "just work".

I don't have code that implements this yet, but if I can get my head round the IO stack and the Python startup code, I'll give it a go.

Steve - any comments on whether this might work? I've never seen any real-world code using U8TEXT which makes me wonder if it's reliable (doing msvcrt.setmode(sys.stdout.fileno(), 0x40000) in Python 3.4 causes Python to crash, which is worrying, but it works in 3.5...).

[bitdancer]

There are a lot of issues in this tracker (for some definition of a lot) that indicate that the console does *not* support unicode. So if you are writing utf-8 I wouldn't expect this to work. (If it were an API taking unicode directly, that might be a different story). But the amount I know about windows is pretty small, so I sure hope you are right

[vstinner]

There are a lot of issues in this tracker (for some definition of a lot) that indicate that the console does *not* support unicode.

The main issue is the issue bpo-1602.

[pfmoore]

Generally, my understanding is that the console does pretty badly at supporting Unicode via the normal WriteFile APIs and the "code page" support (mainly because support for the UTF8 code page is rubbish). But the WriteConsole API does, I believe, have pretty solid Unicode support (it's what Powershell uses, for example). Typically, attempts to support Unicode for Python console output (e.g., win_unicode_console on PyPI) deal with this by making a file-like object that calls WriteConsole under the hood, and replaces sys.stdout with this. The problem with this approach is that it isn't a "normal" text stream object (there's no underlying raw bytes buffer), so the result isn't seamless (although win_unicode_console is pretty good).

What I noticed is that the C runtime supports an _O_U8TEXT mode for console file descriptors, at the (bytes) write() level. So that could be seamlessly integrated into the bytes IO layer of the Python IO stack.

As far as I can tell from the description, the way it works is to treat a block of bytes written via write() as a UTF8 string, encode it to Unicode and write it to the console via WriteConsole(). (I haven't checked the CRT source, but that seems like the most likely implementation).

Code speaks louder than words, obviously, and I do intend to produce a trial implementation. But that'll take a bit of time because I need to understand how the IO stack hangs together first.

An alternative approach would be a RawIOBase implementation that wrote bytes to the console by (re-)decoding them from UTF8 and using WriteConsole, then wrapping that in the usual buffered IO and text IO layers (with the text IO layer using UTF8 encoding). That may well be implementable in pure Python, and make a good prototype implementation. Hmm...

[pfmoore]

Doh. That latter approach (a RawIOBase implementation) is *precisely* what win_unicode_console does for stdout (using utf16le rather than utf8 as that's the native Windows encoding used by WriteConsole). So (a) yes it would work, and (b) it has already demonstrated in the wild that the approach is viable.

(Actually, a C implementation of this approach might be a better way of implementing this anyway, rather than relying on a relatively obscure C runtime feature).

[pfmoore]

My proof-of-concept attempt to use _O_U8TEXT resulted in some very bizarre behaviour - odd buffering of the interactive interpreter output and what appear to be Chinese characters being displayed for normal (ASCII) interactions.

I suspect there is some oddity around how _O_U8TEXT works. The approach looks too fragile to pursue. I'll look further into the RawIOBase option.

[vstinner]

If sys.stdout is modified, it must be carefully tested in various scenario:

• Windows console, default config
• Windows console, TrueType font
• PowerShell => see bpo-21927, it looks like PowerShell has its own set of Unicode issues
• Redirect output into a file
• etc.

Very good articles by Michael S. Kaplan on Windows stdout/console:

• "Conventional wisdom is retarded, aka What the @#%&* is _O_U16TEXT?"
  http://www.siao2.com/2008/03/18/8306597.aspx
• "Myth busting in the console"
  http://www.siao2.com/2010/10/07/10072032.aspx
• "Cunningly conquering communicated console caveats. Comprende, mon Capitán?"
  http://www.siao2.com/2010/05/07/10008232.aspx

See also fwide() function.

Good luck...

[vadmium]

This seems to be discussing the same sort of stuff that ended up with the bpo-1602 implementation.