Py_Initialize() and Py_Main() should not enable C locale coercion

[vstinner]

BPO	34589
Nosy	@ncoghlan, @vstinner, @ned-deily, @koobs
PRs	bpo-34589: C locale coercion off by default #9073 WIP bpo-34589: Move locale coercion back to the CLI app #9257 bpo-34589: Make _PyCoreConfig.coerce_c_locale private #9371 bpo-34589: Add -X coerce_c_locale command line option #9378 [3.7] bpo-34589: Add -X coerce_c_locale option; C locale coercion off by default #9379 bpo-34589: Add test_PYTHONCOERCECLOCALE_ignored() #9382 [3.7] Revert "[3.7] bpo-34589: Add -X coerce_c_locale option; C locale coercion off by default (GH-9379)" #9416 Revert "bpo-34589: Add -X coerce_c_locale command line option (GH-9378)" #9430

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-12-19.10:48:18.727>
created_at = <Date 2018-09-05.19:36:35.051>
labels = ['interpreter-core', '3.8', 'type-bug', '3.7']
title = 'Py_Initialize() and Py_Main() should not enable C locale coercion'
updated_at = <Date 2019-04-07.11:58:09.393>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2019-04-07.11:58:09.393>
actor = 'ncoghlan'
assignee = 'none'
closed = True
closed_date = <Date 2018-12-19.10:48:18.727>
closer = 'ncoghlan'
components = ['Interpreter Core']
creation = <Date 2018-09-05.19:36:35.051>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 34589
keywords = ['patch']
message_count = 38.0
messages = ['324649', '325008', '325009', '325010', '325093', '325097', '325098', '325242', '325246', '325256', '325282', '325493', '325554', '325588', '325597', '325600', '325605', '325606', '325608', '325709', '325710', '325712', '325713', '325714', '325723', '325748', '325783', '325806', '325865', '325866', '325948', '325984', '329351', '330378', '330380', '332123', '339476', '339574']
nosy_count = 4.0
nosy_names = ['ncoghlan', 'vstinner', 'ned.deily', 'koobs']
pr_nums = ['9073', '9257', '9371', '9378', '9379', '9382', '9416', '9430']
priority = 'normal'
resolution = 'postponed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue34589'
versions = ['Python 3.7', 'Python 3.8']

[vstinner]

According to Nick Coghlan, author of the PEP-538, the C locale coercion should not be enabled by Py_Initialize() and Py_Main(), only by the python3 program.

Attached PR fix this issue.

[ncoghlan]

The entire change affecting the PEP-538 implementation in 9454060#diff-8c018c3ada66d06c8e101e47a313c2c7 needs to be reverted: the locale should be coerced before *ANY* calls are made to Py_DecodeLocale, as the whole point of the architecture of the PEP implementation was to ensure that *nothing ever gets decoded incorrectly in the first place*.

I just initially missed that in Victor's enthusiasm for fixing the incorrect decodings that can arise in the absence of locale coercion he'd *introduced* incorrect decoding into the locale coercion case.

[ncoghlan]

(The one exception to "nothing gets decoded incorrectly" is that PYTHONCOERCECLOCALE itself is always interpreted as an ASCII field: the values that it checks for are actually ASCII byte sequences, not Unicode code points.

The documentation could definitely be much clearer on that point though, as even in the PEP it's only implied by the final paragraph in https://www.python.org/dev/peps/pep-0538/#legacy-c-locale-coercion-in-the-standalone-python-interpreter-binary which is mostly talking about the fact that this particular environment variable is still checked, even if you pass the -I or -E command line options.

[ncoghlan]

FWIW, this is also why I don't want PEP-432 to expose any wstr fields in the configuration settings: it means embedding applications have to figure out which encoding Python is expecting to be used for those fields.

Everything should be much cleaner if the public API requires those settings to be provided as Py_Unicode objects, and the core configuration step takes care of getting the C API to the point where the embedding application can create Py_Unicode objects for passing to later steps.

The fact that CPython itself currently uses wstr to configure these settings should be considered a legacy implementation detail that we do our best to hide from the public API.

[vstinner]

(...) the locale should be coerced before *ANY* calls are made to Py_DecodeLocale, as the whole point of the architecture of the PEP implementation was to ensure that *nothing ever gets decoded incorrectly in the first place*.

See my comment:
#9073 (comment)

[vstinner]

(...) The documentation could definitely be much clearer on that point though, as even in the PEP it's only implied by the final paragraph in https://www.python.org/dev/peps/pep-0538/#legacy-c-locale-coercion-in-the-standalone-python-interpreter-binary which is mostly talking about the fact that this particular environment variable is still checked, even if you pass the -I or -E command line options.

That's a separated issue: I created bpo-34639.

[vstinner]

Example of C locale coercion with Python 3.7:

$ env -i PYTHONCOERCECLOCALE=0 ./python -X utf8=0 -c 'import sys; print(ascii(sys.argv[1]))' 'hé'
'h\udcc3\udca9'

$ env -i PYTHONCOERCECLOCALE=1 ./python -X utf8=0 -c 'import sys; print(ascii(sys.argv[1]))' 'hé'
'h\xe9'

It works as expected no? What's wrong with the current implementation according to you, Nick?

[ncoghlan]

Reviewing the way things are now, I don't think it's worth rearranging this yet again for 3.7 - instead, I'll just put it back the way I intended it to be (and the way the PEP describes) for 3.8.

The issue is that Victor had good reasons for moving the handling of the "PYTHONCOERCECLOCALE=warn" case to be after the low level C stdio streams were properly initialised, and retaining that aspect of the changes while still moving the C locale coercion back to the beginning of the program means changing the signature of _Py_CoerceLegacyLocale to return a stateless static string to be printed to stderr later.

[ncoghlan]

The actual functional error is that the following will currently give different outputs on Fedora and CentOS 7, whereas in the original PEP-538 implementation it would always print "C", even if locale coercion would otherwise normally work on your current platform:

    LC_CTYPE=C PYTHONCOERCECLOCALE=0 python3 -E -c 'import os; print(os.env["LC_CTYPE"])'

As per the comment that was deleted in 9454060#diff-8c018c3ada66d06c8e101e47a313c2c7, that was an intentional behaviour to allow developers testing scripts that are invoked using -E and -I to pretend that their local C.UTF-8 locale didn't exist when trying to reproducing behaviour otherwise seen only when locale coercion fails due to the lack of a suitable coercion target.

I'm working on an alternate PR that restores that aspect of the original behaviour, while preserving all the other improvements you made (such as ensuring that emitting the warning gets delayed until after the C level stderr is properly initialised)