_PyCoreConfig: add stdio_encoding and stdio_errors

[vstinner]

BPO	34485
Nosy	@ncoghlan, @vstinner
PRs	bpo-34485: Add _PyCoreConfig.stdio_encoding #8881 bpo-34485: Enhance init_sys_streams() #8978 bpo-34485: Fix _Py_InitializeCore() for C locale coercion #8979 [3.7] bpo-34485: Fix _Py_InitializeCore() for C locale coercion (GH-8979) #8981 bpo-34485: Add _Py_ClearStandardStreamEncoding() #8982 bpo-34485: stdout uses surrogateescape on POSIX locale #8986 [3.7] bpo-34485: stdout uses surrogateescape on POSIX locale (GH-8986) #8987 bpo-34485, Windows: LC_CTYPE set to user preference #8988 bpo-34485: Emit C locale coercion warning later #9002 bpo-34485: Fix pymain_read_conf() LC_ALL locale #9041 bpo-34544: pymain_read_conf() don't change LC_ALL #9045

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-08-29.09:50:04.427>
created_at = <Date 2018-08-23.23:44:09.419>
labels = ['3.8']
title = '_PyCoreConfig: add stdio_encoding and stdio_errors'
updated_at = <Date 2018-09-03.15:32:34.723>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2018-09-03.15:32:34.723>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2018-08-29.09:50:04.427>
closer = 'vstinner'
components = []
creation = <Date 2018-08-23.23:44:09.419>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 34485
keywords = ['patch']
message_count = 15.0
messages = ['323980', '324290', '324294', '324297', '324298', '324306', '324309', '324311', '324315', '324316', '324349', '324382', '324384', '324511', '324526']
nosy_count = 2.0
nosy_names = ['ncoghlan', 'vstinner']
pr_nums = ['8881', '8978', '8979', '8981', '8982', '8986', '8987', '8988', '9002', '9041', '9045']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue34485'
versions = ['Python 3.8']

[vstinner]

Currently, the code to select the encoding and error handler of sys.stdin, sys.stdout and sys.stderr is run later in Py_Initialize(). I propose to move most of this code into _PyCoreConfig_Read() and add stdio_encoding and stdio_errors to _PyCoreConfig to easily let Python embedders to select the encoding.

Attached PR implements this idea. During Py_Initialize(), the code still reads the LC_CTYPE locale to choose the error handler if stdio_errors is NULL. This check cannot be done earlier, since the LC_CTYPE locale is only set to the user locale in Py_Initialize(): after _PyCoreConfig_Read().

The Py_SetStandardStreamEncoding() function has been added by Nick Coghlan in bpo-16129.

This new pre-initialization API allows embedding
applications like Blender to force a particular
encoding and error handler for the standard IO streams.

[vstinner]

New changeset 9e4994d by Victor Stinner in branch 'master':
bpo-34485: Enhance init_sys_streams() (GH-8978)
9e4994d

[vstinner]

New changeset 2c8ddcf by Victor Stinner in branch 'master':
bpo-34485: Fix _Py_InitializeCore() for C locale coercion (GH-8979)
2c8ddcf

[vstinner]

New changeset 98c49c6 by Victor Stinner in branch '3.7':
bpo-34485: Fix _Py_InitializeCore() for C locale coercion (GH-8979) (GH-8981)
98c49c6

[vstinner]

New changeset 124b9eb by Victor Stinner in branch 'master':
bpo-34485: Add _Py_ClearStandardStreamEncoding() (GH-8982)
124b9eb

[vstinner]

New changeset 315877d by Victor Stinner in branch 'master':
bpo-34485: stdout uses surrogateescape on POSIX locale (GH-8986)
315877d

[vstinner]

New changeset 0b9ea4b by Victor Stinner in branch '3.7':
[3.7] bpo-34485: stdout uses surrogateescape on POSIX locale (GH-8986) (GH-8987)
0b9ea4b

[vstinner]

New changeset 177d921 by Victor Stinner in branch 'master':
bpo-34485, Windows: LC_CTYPE set to user preference (GH-8988)
177d921

[vstinner]

New changeset dfe0dc7 by Victor Stinner in branch 'master':
bpo-34485: Add _PyCoreConfig.stdio_encoding (GH-8881)
dfe0dc7

[vstinner]

Oh wow, I didn't expect that it would be so complicated to "just" get the encoding of sys.stdout... I had to fix many bugs to be able to do that!

[vstinner]

New changeset cf21504 by Victor Stinner in branch 'master':
bpo-34485: Emit C locale coercion warning later (GH-9002)
cf21504

[ncoghlan]

Yeah, there were some good reasons I went with the relatively brute force option of provideding Blender with a new config API call back in bpo-16129 - as we've seen, actually fixing it properly has been a multi-year multi-person effort :)

[vstinner]

Yeah, there were some good reasons I went with the relatively brute force option of provideding Blender with a new config API call back in bpo-16129 - as we've seen, actually fixing it properly has been a multi-year multi-person effort :)

Yeah, I see ;-) Thanks for Py_SetStandardStreamEncoding(), it fixed Blender use case!

[vstinner]

New changeset f01b2a1 by Victor Stinner in branch 'master':
bpo-34544: Fix setlocale() in pymain_read_conf() (GH-9041)
f01b2a1

[vstinner]

New changeset 73b00be by Victor Stinner in branch 'master':
bpo-34544: pymain_read_conf() don't change LC_ALL (GH-9045)
73b00be