Py_Initialize(): computing path configuration must not have side effect (PEP 432)

[vstinner]

BPO	34170
Nosy	@vstinner
PRs	bpo-34170: Rework _PyCoreConfig_Read() to avoid side effect #8353 bpo-34170: _PyCoreConfig_Read() leaves Py_IsolatedFlag unchanged #8361 bpo-34170: Add _PyCoreConfig.isolated #8417 bpo-34170: Add _PyCoreConfig.bytes_warning #8447 bpo-34170: Add _Py_InitializeFromConfig() #8454 bpo-34170: Cleanup pymain_run_python() #8455 bpo-34170: Enhance _PyCoreConfig_Read() #8468 bpo-34170: Cleanup pymain_read_conf() #8476 bpo-34170, test_embed: write Py_Initialize() tests #8484 bpo-34247: actually use environment variables in Py_Initialize #8521 bpo-34170: Add _PyCoreConfig._frozen parameter #8591 bpo-34170: Py_Main() updates config on setting Py_InspectFlag #8593 bpo-34170: Rename _PyCoreConfig.unbuffered_stdio #8594 bpo-34170: _PyCoreConfig_Read() defaults to argc=0 #8595 bpo-34170: Add Python/coreconfig.c for _PyCoreConfig #8607 bpo-31650: Remove _Py_CheckHashBasedPycsMode global config var #8608 bpo-34170: Cleanup pymain_run_filename() #8631 bpo-34170: _PyCoreConfig_Read() don't replace coerce_c_locale #8658 bpo-34170: Fix pymain_run_file() #8660 bpo-34207: Fix pymain_init_cmdline_argv() #8867 bpo-34207: Fix pymain_read_conf() for UTF-8 Mode #8868 [3.7] bpo-34207: Fix pymain_read_conf() for UTF-8 Mode (GH-8868) #8870

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-08-23.10:44:58.712>
created_at = <Date 2018-07-20.15:22:57.855>
labels = ['interpreter-core', '3.8']
title = 'Py_Initialize(): computing path configuration must not have side effect (PEP 432)'
updated_at = <Date 2018-08-23.10:44:58.711>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2018-08-23.10:44:58.711>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2018-08-23.10:44:58.712>
closer = 'vstinner'
components = ['Interpreter Core']
creation = <Date 2018-07-20.15:22:57.855>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 34170
keywords = ['patch']
message_count = 27.0
messages = ['322015', '322018', '322020', '322040', '322060', '322290', '322297', '322336', '322337', '322340', '322366', '322371', '322393', '322427', '322440', '322828', '322829', '322830', '322831', '322868', '322870', '322871', '322982', '323073', '323086', '323936', '323938']
nosy_count = 1.0
nosy_names = ['vstinner']
pr_nums = ['8353', '8361', '8417', '8447', '8454', '8455', '8468', '8476', '8484', '8521', '8591', '8593', '8594', '8595', '8607', '8608', '8631', '8658', '8660', '8867', '8868', '8870']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue34170'
versions = ['Python 3.8']

[vstinner]

Attached PR reworks _PyCoreConfig_Read() to leave _Py_path_config unchanged: *reading* the core configuration must not *modify* the path configuration.

The PR adds _PyCoreConfig.dll_path field.

The change is related to the PEP-432.

[vstinner]

See also bpo-34008: "Do we support calling Py_Main() after Py_Initialize()?".

[vstinner]

PR 8353 is the last piece of my long work on Py_Main() to separate code *reading* configuration and code *applying* a new configuration. My work on the new _PyPathConfig structure, and the PR should finish that work.

Extract of the commit message:

"A new _PyCoreConfig_SetPathConfig() function now recreates the path configuration from the core configuration. This function is now called very late in _Py_InitializeCore(), just before calling initimport()."

[vstinner]

New changeset b1147e4 by Victor Stinner in branch 'master':
bpo-34170: Rework _PyCoreConfig_Read() to avoid side effect (GH-8353)
b1147e4

[vstinner]

New changeset f2626ce by Victor Stinner in branch 'master':
bpo-34170: _PyCoreConfig_Read() leaves Py_IsolatedFlag unchanged (GH-8361)
f2626ce

[vstinner]

New changeset d19d8d5 by Victor Stinner in branch 'master':
bpo-34170: Add _PyCoreConfig.isolated (GH-8417)
d19d8d5

[vstinner]

Reminder for myself: I have to write unit tests :-)
#8417 (review)

[vstinner]

New changeset 1dc6e39 by Victor Stinner in branch 'master':
bpo-34170: Add _Py_InitializeFromConfig() (GH-8454)
1dc6e39

[vstinner]

I added many fields to _PyCoreConfig which duplicate global configuration varibles: _PyConfigCore.isolated duplicates Py_IsolatedFlag.

I started to modify Python to read the core configuration rather than global configuration flags. The problem is that sometimes, global configuration flags are updated, but not their duplicated core configuration fields.

Example from Modules/main.c:

static void
pymain_repl(_PyMain *pymain, _PyCoreConfig *config, PyCompilerFlags *cf)
{
    /* Check this environment variable at the end, to give programs the
       opportunity to set it from Python. */
    if (!Py_InspectFlag && config_get_env_var(config, "PYTHONINSPECT")) {
        Py_InspectFlag = 1;
    }
    ...
}

Only Py_InspectFlag is not, not core_config.inspect.

[vstinner]

New changeset d3b1919 by Victor Stinner in branch 'master':
bpo-34170: Cleanup pymain_run_python() (GH-8455)
d3b1919

[vstinner]

TODO list, open questions:

• Should isolated=1 always imply config->ignore_environment=1 and config->user_site_directory = 0?
• Should we rename debug to parser_debug? "debug" is a very generic term, whereas this flag is really specific to the parser
• Remove _Py_CheckHashBasedPycsMode: replace it with config._check_hash_pycs_mode?

[vstinner]

I created bpo-34228 "Allow PYTHONTRACEMALLOC=0 and -X tracemalloc=0 to disable explicitly tracemalloc" that I need to write unit tests on Python initiallization.

[vstinner]

New changeset ecf411c by Victor Stinner in branch 'master':
bpo-34170: Enhance _PyCoreConfig_Read() (GH-8468)
ecf411c

[vstinner]

New changeset d145775 by Victor Stinner in branch 'master':
bpo-34170: Cleanup pymain_read_conf() (GH-8476)
d145775

[vstinner]

New changeset 56b29b6 by Victor Stinner in branch 'master':
bpo-34170, test_embed: write Py_Initialize() tests (GH-8484)
56b29b6

[vstinner]

New changeset b75d7e2 by Victor Stinner in branch 'master':
bpo-34170: Add _PyCoreConfig._frozen parameter (GH-8591)
b75d7e2

[vstinner]

New changeset a4d20b2 by Victor Stinner in branch 'master':
bpo-34170: Py_Main() updates config when setting Py_InspectFlag (GH-8593)
a4d20b2

[vstinner]

New changeset 9851227 by Victor Stinner in branch 'master':
bpo-34170: Rename _PyCoreConfig.unbuffered_stdip (GH-8594)
9851227

[vstinner]

New changeset ea68d83 by Victor Stinner in branch 'master':
bpo-34170: _PyCoreConfig_Read() defaults to argc=0 (GH-8595)
ea68d83

[vstinner]

New changeset 6c785c0 by Victor Stinner in branch 'master':
bpo-34170: Add Python/coreconfig.c for _PyCoreConfig (GH-8607)
6c785c0

[vstinner]

New changeset 80b762f by Victor Stinner in branch 'master':
bpo-31650: Remove _Py_CheckHashBasedPycsMode global config var (GH-8608)
80b762f

[vstinner]

"""
TODO list, open questions:

• Should isolated=1 always imply config->ignore_environment=1 and config->user_site_directory = 0?
• Should we rename debug to parser_debug? "debug" is a very generic term, whereas this flag is really specific to the parser
• Remove _Py_CheckHashBasedPycsMode: replace it with config._check_hash_pycs_mode?
  """

=> DONE!

[vstinner]

New changeset 72ec319 by Victor Stinner in branch 'master':
bpo-34170: Cleanup pymain_run_filename() (GH-8631)
72ec319

[vstinner]

New changeset 5a953fd by Victor Stinner in branch 'master':
bpo-34170: _PyCoreConfig_Read() don't replace coerce_c_locale (GH-8658)
5a953fd

[vstinner]

New changeset d807862 by Victor Stinner in branch 'master':
bpo-34170: Fix pymain_run_file() (GH-8660)
d807862

[vstinner]

New changeset 89487f5 by Victor Stinner in branch 'master':
bpo-34207: Fix pymain_read_conf() for UTF-8 Mode (GH-8868)
89487f5

[vstinner]

New changeset 80a0eba by Victor Stinner in branch '3.7':
bpo-34207: Fix pymain_read_conf() for UTF-8 Mode (GH-8868) (GH-8870)
80a0eba