PYTHONCOERCECLOCALE=0 ignored

[lazka]

BPO	38667
Nosy	@vstinner, @lazka

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-11-04.07:18:40.410>
created_at = <Date 2019-11-02.13:14:24.447>
labels = ['invalid', '3.7', '3.8']
title = 'PYTHONCOERCECLOCALE=0 ignored'
updated_at = <Date 2019-11-04.07:18:40.407>
user = 'https://github.com/lazka'

bugs.python.org fields:

activity = <Date 2019-11-04.07:18:40.407>
actor = 'lazka'
assignee = 'none'
closed = True
closed_date = <Date 2019-11-04.07:18:40.410>
closer = 'lazka'
components = []
creation = <Date 2019-11-02.13:14:24.447>
creator = 'lazka'
dependencies = []
files = []
hgrepos = []
issue_num = 38667
keywords = []
message_count = 3.0
messages = ['355868', '355919', '355927']
nosy_count = 2.0
nosy_names = ['vstinner', 'lazka']
pr_nums = []
priority = 'normal'
resolution = 'not a bug'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue38667'
versions = ['Python 3.7', 'Python 3.8']

[lazka]

Python 3.7.5rc1 and 3.8.0 on Ubuntu 19.10

$ LC_CTYPE=C PYTHONCOERCECLOCALE=warn python3 -c "import sys; print(sys.getfilesystemencoding())"
Python detected LC_CTYPE=C: LC_CTYPE coerced to C.UTF-8 (set another locale or PYTHONCOERCECLOCALE=0 to disable this locale coercion behavior).
utf-8

$ LC_CTYPE=C PYTHONCOERCECLOCALE=0 python3 -c "import sys; print(sys.getfilesystemencoding())"   
utf-8

The warning states that passing PYTHONCOERCECLOCALE=0 disables the coercion, but it doesn't change anything in 3.7 and 3.8.

What am I missing?

[vstinner]

You should not use sys.getfilesystemencoding() nor locale.getpreferredencofing() to check if the C locale has been coerced, but read the LC_CTYPE locale instead:

# C locale coerced
$ LC_CTYPE=C python3 -c "import locale; print(locale.setlocale(locale.LC_CTYPE, ''))"
C.UTF-8

# C locale not coerced: remains "C"
$ LC_CTYPE=C PYTHONCOERCECLOCALE=0 python3 -c "import locale; print(locale.setlocale(locale.LC_CTYPE, ''))"
C

--

$ LC_CTYPE=C PYTHONCOERCECLOCALE=0 python3 -c "import sys; print(sys.getfilesystemencoding())"   
utf-8

In this example, the UTF-8 encoding is used because of the UTF-8 Mode:

$ LC_CTYPE=C PYTHONCOERCECLOCALE=0 python3 -X utf8=1 -c "import sys; print(sys.getfilesystemencoding(), sys.flags.utf8_mode)"
utf-8 1

$ LC_CTYPE=C PYTHONCOERCECLOCALE=0 python3 -X utf8=0 -c "import sys; print(sys.getfilesystemencoding(), sys.flags.utf8_mode)"
ascii 0

The relationship between the PEP-538 and the PEP-540 is non obvious:
https://www.python.org/dev/peps/pep-0540/#relationship-with-the-locale-coercion-pep-538

I guess that you would like: PYTHONUTF8=0 env var or -X utf8=0 command line option.

--

I don't think that this issue is a bug, and so I suggest to close it as "not a bug".

[lazka]

Thanks! I didn't quite see the relation between PEP-538 and PEP-540 after only reading the former.

Anyone else finding this: See "Relationship with the locale coercion" in https://www.python.org/dev/peps/pep-0540/ for the details.