PEP 540: Add a new UTF-8 mode

[vstinner]

BPO	29240
Nosy	@vstinner, @ezio-melotti, @methane, @eryksun
PRs	bpo-29240: PEP 540: Add a new UTF-8 mode #855 bpo-29240, bpo-32030: pymain_set_sys_argv() now copies argv #4838 bpo-29240: Don't define decode_locale() on macOS #4895 bpo-29240, bpo-32030: Py_Main() re-reads config if encoding changes #4899 bpo-29240: Skip test_readline.test_nonascii() #4968 bpo-29240: readline now ignores the UTF-8 Mode #5145 bpo-29240: Ignore UTF-8 Mode in time module #5148 bpo-29240: Fix locale encodings in UTF-8 Mode #5170 bpo-31900: Fix localeconv() encoding for LC_NUMERIC #4174 bpo-29240: Skip test_readline.test_nonascii() on C locale #5203 [3.6] bpo-29240: Skip test_readline.test_nonascii() on C locale (GH-5203) #5204 bpo-29240: PyUnicode_DecodeLocale() uses UTF-8 on Android #5272
Files	test_all_locales.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-01-17.14:28:41.827>
created_at = <Date 2017-01-11.11:19:52.086>
labels = ['interpreter-core', 'type-feature', 'library', 'expert-unicode', '3.7']
title = 'PEP 540: Add a new UTF-8 mode'
updated_at = <Date 2022-02-08.11:52:04.506>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2022-02-08.11:52:04.506>
actor = 'yan12125'
assignee = 'none'
closed = True
closed_date = <Date 2018-01-17.14:28:41.827>
closer = 'vstinner'
components = ['Interpreter Core', 'Library (Lib)', 'Unicode']
creation = <Date 2017-01-11.11:19:52.086>
creator = 'vstinner'
dependencies = []
files = ['47385']
hgrepos = []
issue_num = 29240
keywords = ['patch']
message_count = 37.0
messages = ['285214', '285215', '285216', '285275', '285276', '285277', '285278', '285280', '285296', '285298', '285325', '285332', '285357', '285407', '285482', '307694', '307695', '308182', '308183', '308198', '308213', '308217', '308430', '308448', '308915', '308916', '309782', '309798', '309958', '309959', '310029', '310092', '310097', '310177', '310443', '310444', '412665']
nosy_count = 4.0
nosy_names = ['vstinner', 'ezio.melotti', 'methane', 'eryksun']
pr_nums = ['855', '4838', '4895', '4899', '4968', '5145', '5148', '5170', '4174', '5203', '5204', '5272']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue29240'
versions = ['Python 3.7']

[vstinner]

This issue tracks the implementation of the PEP-540.

Attached pep540_cli.py script can be used to play with it.

[vstinner]

PEP-540.patch: first draft

Changes:

• Add sys.flags.utf8mode
• Add -X utf8 command line option
• Add PYTHONUTF8 environment variable
• sys.stdin, sys.stdout and sys.stderr encoding and errors are modified in UTF-8 mode
• open() default encoding and errors is modified in the UTF-8 mode
• Add Lib/test/test_utf8mode.py
• Skip a few tests relying on the locale encoding if the UTF-8 mode is enabled
• Document changes

Allowed options:

• Disable UTF-8 mode: -X utf8=0 or PYTHONUTF8=0
• Enable UTF-8 mode: -X utf8=1 or PYTHONUTF8=1
• Enable UTf-8 Strict mode: -X utf8=strict or PYTHONUTF8=strict
• Other -X utf8 and PYTHONUTF8 values cause a fatal error

Prioririties (highest to lowest):

• open() encoding and errors arguments
• PYTHONIOENCODING
• UTF-8 mode
• os.device_encoding()
• locale encoding

TODO:

• re-encode sys.argv from the local encoding to UTF-8 in Py_Main() when the UTF-8 mode is enabled
• support strict mode in Py_DecodeLocale() and Py_EncodeLocale()

[vstinner]

Examples with pep540_cli.py.

Python 3.5:

$ python3 pep540_cli.py 
sys.argv: ['pep540_cli.py']
stdin: UTF-8/strict
stdout: UTF-8/strict
stderr: UTF-8/backslashreplace
open(): UTF-8/strict

$ LC_ALL=C python3 pep540_cli.py 
sys.argv: ['pep540_cli.py']
stdin: ANSI_X3.4-1968/surrogateescape
stdout: ANSI_X3.4-1968/surrogateescape
stderr: ANSI_X3.4-1968/backslashreplace
open(): ANSI_X3.4-1968/strict

Patched Python 3.7:

$ ./python pep540_cli.py 
UTF-8 mode: 0
sys.argv: ['pep540_cli.py']
stdin: UTF-8/strict
stdout: UTF-8/strict
stderr: UTF-8/backslashreplace
open(): UTF-8/strict

$ LC_ALL=C ./python pep540_cli.py 
UTF-8 mode: 1
sys.argv: ['pep540_cli.py']
stdin: utf-8/surrogateescape
stdout: utf-8/surrogateescape
stderr: utf-8/backslashreplace
open(): utf-8/surrogateescape

$ ./python -X utf8 pep540_cli.py 
UTF-8 mode: 1
sys.argv: ['pep540_cli.py']
stdin: utf-8/surrogateescape
stdout: utf-8/surrogateescape
stderr: utf-8/backslashreplace
open(): utf-8/surrogateescape

$ ./python -X utf8=strict pep540_cli.py 
UTF-8 mode: 2
sys.argv: ['pep540_cli.py']
stdin: utf-8/strict
stdout: utf-8/strict
stderr: utf-8/backslashreplace
open(): utf-8/strict

[vstinner]

PEP-540-2.patch: Patch version 2, updated to the latest version of the PEP-540. It has no more FIXME/TODO and has more unit tests. The main change is that the strict mode doesn't use strict anymore for OS data, but keeps surrogateescape. See the PEP for the rationale (especially the "Use the strict error handler for operating system data" alternative).

[vstinner]

Oops, I introduced an obvious bug in my latest refactoring. It's now fixed in the patch version 3: PEP-540-3.patch.

[vstinner]

Hum, PEP-540-3.patch doesn't work if the locale encoding is different than ASCII and UTF-8. argv must be reencoded:

$ LC_ALL=fr_FR ./python -X utf8 -c 'import sys; print(ascii(sys.argv))' $(echo -ne "\xff")
['-c', '\xff']

The result should not depend on the locale, it should be the same than:

$ LC_ALL=fr_FR.utf8 ./python -X utf8 -c 'import sys; print(ascii(sys.argv))' $(echo -ne "\xff")
['-c', '\udcff']

$ LC_ALL=C ./python -X utf8 -c 'import sys; print(ascii(sys.argv))' $(echo -ne "\xff")
['-c', '\udcff']

[vstinner]

I only tested the the PEP-540 implementation on Linux.

The PEP and its implementation should adjusted for Windows, especially Windows-only env vars like PYTHONLEGACYWINDOWSFSENCODING.

Changes are maybe also needed for Mac OS X and Android, which always use UTF-8. Currently, the locale encoding is still used on these platforms (ex: by open()). Is it possible to a locale encoding different than UTF-8 on Android for example?

[methane]

Hum, PEP-540-3.patch doesn't work if the locale encoding is different than ASCII and UTF-8. argv must be reencoded:

I want to skip reencoding.
On UTF-8 mode, arbitrary bytes in cmdline (e.g. broken filename passed by xarg) should be able to roundtrip by UTF-8/surrogateescape.

I don't trust wcstombs/mbstowcs. It may not guarantee round tripping of arbitrary bytes.

Can -X utf8 option be processed before Py_Main()?

[vstinner]

Can -X utf8 option be processed before Py_Main()?

I'm trying to implement that, but it's hard to factorize the code. I will probably have to duplicate the code handling -E, -X utf8, PYTHONMALLOC and PYTHONUTF8 for wchar_t* (UCS4 or UTF-16) and char* (bytes).

[vstinner]

Hum, test_utf8mode lacks an unit test on the -E command line option:
PYTHONUTF8 should be ignored if -E is used.