Py_Initialize affects the console

[twoone-3]

BPO	41941
Nosy	@pfmoore, @vstinner, @tjguk, @ezio-melotti, @zware, @eryksun, @zooba, @twoone-3
Files	Screenshots.zip: Test result

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-10-05.13:03:41.063>
created_at = <Date 2020-10-05.08:51:42.232>
labels = ['type-bug', '3.8', 'OS-windows', 'expert-IO', 'expert-C-API', 'invalid', 'expert-unicode']
title = 'Py_Initialize affects the console'
updated_at = <Date 2020-10-05.13:26:53.997>
user = 'https://github.com/twoone-3'

bugs.python.org fields:

activity = <Date 2020-10-05.13:26:53.997>
actor = 'eryksun'
assignee = 'none'
closed = True
closed_date = <Date 2020-10-05.13:03:41.063>
closer = 'vstinner'
components = ['Unicode', 'Windows', 'IO', 'C API']
creation = <Date 2020-10-05.08:51:42.232>
creator = 'twoone3'
dependencies = []
files = ['49495']
hgrepos = []
issue_num = 41941
keywords = []
message_count = 11.0
messages = ['378004', '378012', '378014', '378016', '378021', '378023', '378024', '378026', '378027', '378028', '378029']
nosy_count = 8.0
nosy_names = ['paul.moore', 'vstinner', 'tim.golden', 'ezio.melotti', 'zach.ware', 'eryksun', 'steve.dower', 'twoone3']
pr_nums = []
priority = 'normal'
resolution = 'not a bug'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue41941'
versions = ['Python 3.8']

[twoone-3]

When I set Py_LegacyWindowsStdioFlag to 0, after Py_Initialize, the console will forcibly change what encoding is displayed. At this time, my standard stream output is utf-8, py's print is utf-8, chcp 65001 is useless, and still garbled. When I set Py_LegacyWindowsStdioFlag to 1, the console encoding is mbcs, the standard stream output is UTF-8 and not garbled, but the print of py is gbk encoding, and my system is Windows server 2016. I hope that Py_Initialize will not affect the console in subsequent versions. Do any behavior, I suggest you test the coding problem in Chinese

[eryksun]

Python supports Unicode in a Windows console session by using the console API's wide-character functions (i.e. ReadConsoleW and WriteConsoleW) with UTF-16 encoded text. This is implemented in the io stack via io._WindowsConsoleIO, and for PyOS_Readline (e.g. builtin input) via _PyOS_WindowsConsoleReadline. The UTF-16 aspect is an internal detail that's presented externally as UTF-8 by automatically converting between the two.

Notwithstanding the behavior of third-party packages, CPython intentionally makes no changes to a console session's global settings, including:

* input & output codepages
* input & output modes (except for msvcrt.getwch, etc)
* cursor size and visibility
* screen-buffer size, font, colors, & attributes
* window size
* window title

[twoone-3]

So how does this explain

[vstinner]

I suggest you to use https://docs.python.org/dev/c-api/init_config.html#c.PyConfig.PyConfig_InitIsolatedConfig API instead of the legacy Py_Initialize() API.

[twoone-3]

This problem has bothered me for a month, thank you for helping me solve this problem, you can close it

[vstinner]

This problem has bothered me for a month, thank you for helping me solve this problem, you can close it

I close the issue, but I'm not sure how you fixed it. Don't hesitate commenting to explain how you solved it ;-)

[eryksun]

Having looked at the screenshots, it seems that your issue is in part due to non-legacy mode not setting the standard I/O files to binary mode (_O_BINARY) from their default ANSI text mode (_O_TEXT). This is not a problem on its own.

The real issue is that you're not using an isolated configuration, as Victor suggested, so the LC_CTYPE locale gets set to the default user locale instead of "C". When writing to the console in ANSI text mode with a configured locale other than the default "C" locale, the C runtime _write() function does a double translation from the locale encoding to the console codepage encoding via the internal function write_double_translated_ansi_nolock().

[twoone-3]

I used the link you gave me https://docs.python.org/dev/c-api/init_config.html#c.PyConfig.PyConfig_InitIsolatedConfig Click to open and turn up to see the box, I follow that box The example inside is done, and the coding problem of the console is solved, but I hope that cpython can provide a more concise API, remove useless functions or variables, so that it is convenient for you to maintain and easy for users to read.

[twoone-3]

That's it

[vstinner]

PyConfig_InitIsolatedConfig() reduces side effects on the process. For example, it doesn't set the LC_CTYPE locale and it doesn't change the standard streams (stdio).

config_init_stdio() is not called in an isolated configuration:

cpython/Python/initconfig.c

Line 1855 in dcc5421

if (config->configure_c_stdio) {

cpython/Python/initconfig.c

Lines 1809 to 1817 in dcc5421

	static void
	config_init_stdio(const PyConfig *config)
	{
	#if defined(MS_WINDOWS) || defined(__CYGWIN__)
	/* don't translate newlines (\r\n <=> \n) */
	_setmode(fileno(stdin), O_BINARY);
	_setmode(fileno(stdout), O_BINARY);
	_setmode(fileno(stderr), O_BINARY);
	#endif

[eryksun]

config_init_stdio() is not called in an isolated configuration:

config_init_stdio wasn't being called anyway since Py_Initialize uses _PyConfig_InitCompatConfig. The issue was primarily due to the LC_CTYPE locale being set to the default user locale, as I discussed in msg378024.

Note that for legacy mode, i.e. Py_LegacyWindowsStdioFlag = 1, there's no simple way to not modify the standard I/O files. The io stack needs binary mode. You'd have to use and modify duped file descriptors.