PYTHONOPTIMIZE ignored in 3.7.0 when using custom launcher

[ronaldoussoren]

BPO	34247
Nosy	@brettcannon, @ronaldoussoren, @ncoghlan, @vstinner, @ned-deily, @ericsnowcurrently, @miss-islington
PRs	bpo-34247: actually use environment variables in Py_Initialize #8521 [3.7] bpo-34247: Fix Python 3.7 initialization #8659 bpo-34247: add porting note to 3.7 What's New #9223 [3.7] bpo-34247: add porting note to 3.7 What's New (GH-9223) #9275 bpo-36142: PYTHONMALLOC overrides PYTHONDEV #12191
Files	t.c foo.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/vstinner'
closed_at = <Date 2018-09-20.12:32:35.054>
created_at = <Date 2018-07-27.12:07:29.981>
labels = ['interpreter-core', 'type-bug', '3.7']
title = 'PYTHONOPTIMIZE ignored in 3.7.0 when using custom launcher'
updated_at = <Date 2019-03-06.11:51:55.385>
user = 'https://github.com/ronaldoussoren'

bugs.python.org fields:

activity = <Date 2019-03-06.11:51:55.385>
actor = 'vstinner'
assignee = 'vstinner'
closed = True
closed_date = <Date 2018-09-20.12:32:35.054>
closer = 'ncoghlan'
components = ['Interpreter Core']
creation = <Date 2018-07-27.12:07:29.981>
creator = 'ronaldoussoren'
dependencies = []
files = ['47716', '47717']
hgrepos = []
issue_num = 34247
keywords = ['patch', '3.7regression']
message_count = 27.0
messages = ['322487', '322519', '322520', '322521', '322522', '322523', '322524', '322529', '322532', '322533', '322550', '322562', '323068', '323070', '323135', '323144', '323145', '323156', '325172', '325281', '325283', '325288', '325289', '325862', '325874', '325877', '337294']
nosy_count = 7.0
nosy_names = ['brett.cannon', 'ronaldoussoren', 'ncoghlan', 'vstinner', 'ned.deily', 'eric.snow', 'miss-islington']
pr_nums = ['8521', '8659', '9223', '9275', '12191']
priority = None
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue34247'
versions = ['Python 3.7']

[ronaldoussoren]

Attached is a very basic launcher for python scripts. This script works as expected in Python 3.6, but not in 3.7.0.

In particular, the launcher sets the environment variable PYTHONOPTIMIZE before calling Py_Initialize and that setting is ignored.

I haven't tried testing with the tip of the tree yet.

[ncoghlan]

This feels like a duplicate of https://bugs.python.org/issue31845 (see d7ac061 )

Is this definitely with the 3.7.0 release, or is it potentially with one of the earlier alphas before that fix was applied?

[ncoghlan]

Ah, wait - I see a relevant difference: the test case for bpo-31845 uses Py_Main, *not* Py_Initialize (since it's a command line test, not an embedding test).

That means it's currently possible for the sequence of operations in

cpython/Python/pylifecycle.c

Line 913 in 7cbde0e

_Py_InitializeEx_Private(int install_sigs, int install_importlib)

to be wrong (or missing steps) without causing the test suite to fail.

[ncoghlan]

Comparing to Python 3.6, we can see that the code to update the C global flags from the environment variables used to live directly in _PyInitializeEx_Private:

cpython/Python/pylifecycle.c

Line 305 in 9d85856

_Py_InitializeEx_Private(int install_sigs, int install_importlib)

In Python 3.7 that section (https://github.com/python/cpython/blob/3.7/Python/pylifecycle.c#L913 ) currently calls _PyCoreConfig_Read instead, which *only* updates the config struct - it doesn't update the C level global variables.

By contrast, Py_Main eventually calls

cpython/Modules/main.c

Line 1394 in 7cbde0e

pymain_set_global_config(_PyMain *pymain, _Py_CommandLineDetails *cmdline)

which handles writing the state from the config struct back to the global variables.

[ncoghlan]

Next steps:

• add a test case that runs the same sys.flags checks as in d7ac061, but as a test_embed embedding test using Py_Initialize rather than Py_Main
• create a shared internal API that both Py_Main and Py_Initialize can use to write the environment variable settings back to the global variables

[ncoghlan]

Ned, I think we should consider this a release blocker for 3.7.1

[ncoghlan]

This seems closely related to the work Victor is already pursuing to resolve bpo-34170 for Python 3.8 (e.g. 56b29b6 ).

The bpo-34170 changes are more invasive than we'd like for a maintenance release, but it would be preferable to keep at least any new Python 3.7 test cases somewhat aligned with their Python 3.8 counterparts.

[ronaldoussoren]

I'm at the EuroPython sprints and currently working on a testcase for this. Should be a nice way to get back into actively contributing to CPython :-)

[ronaldoussoren]

FYI: I've filed bpo-34255 while working on this, test_embed assumes that you're building in the source directory (which I usually don't do).

[ronaldoussoren]

Interesting... pylifecycle.c uses code in main.c, I hadn't expected that.

If I've read the code correctly Py_Initialize won't look at PYTHONOPTIMZE at all (and other environment variables that have a command-line equivalent). Those are read in main.c:cmdline_get_env_flags, which is only called (indirectly) from Py_Main and _Py_UnixMain.

I'm note sure what's the correct solution for this.

1. Reading PYTHONOPTIMZE (and related variables) could be done
   in _PyCoreConfig_Read, but that would then need to grow a flag to ensure
   that this won't replace values calculated earlier in the implementation
   of Py_Main

2. Writing back the global variables is probably best done using a new
   function (lets _PyCoreConfig_SetGlobals), mirroring
   pymain_set_global_config but using a _PyCoreConfig config structure
   instead of the command-line structure used by the latter function.

[ronaldoussoren]

I've created a pull request that seems to work properly, although I am not entirely sure that this is the right way to fix this.

As I mentioned in the pull request I created it against the 3.7 branch because of bpo-34170, but would happily rebase it for master.

[ncoghlan]

I believe Victor is at the EuroPython sprints as well, so if I'm right about that, I think the best option would be for the two of you to work out a resolution for 3.7.1 in person that at least keeps the test suites reasonably consistent, even if the implementations have diverged (the new tests already added for bpo-34170 were the main reason I stopped working on my initial patch for this).

And agreed on fixing the current dependency inversion between pylifecycle and the py_main code - the functions needed by both Py_Main and Py_Initialize should be migrated down the stack into pylifecycle as part of the bpo-34170 work.

[vstinner]

I first proposed to backport all code from master:
https://mail.python.org/pipermail/python-dev/2018-July/154882.html

But I'm not sure that it's a good idea, since I made *many* changes in the master branch since 3.7... I even added more files, and the _PyCoreConfig API is going to change anyway...

So I wrote a simpler PR 8659 which backports unit tests, adapt them for Python 3.7, and fix this issue: PYTHONOPTIMIZE and other environment variables and now read by Py_Initialize(), not only by Py_Main().

[vstinner]

I tested manually attached t.c with PR 8659: I get optimize=2 as expected. Full output:

sys.flags(debug=0, inspect=0, interactive=0, optimize=2, dont_write_bytecode=0, no_user_site=0, no_site=0, ignore_environment=0, verbose=0, bytes_warning=0, quiet=0, hash_randomization=1, isolated=0, dev_mode=False, utf8_mode=1)

[ronaldoussoren]

The PR works for me as well, and looks ok to me.

[vstinner]

New changeset 0c90d6f by Victor Stinner in branch '3.7':
[3.7] bpo-34247: Fix Python 3.7 initialization (bpo-8659)
0c90d6f

[vstinner]

On my PR, Nick Coghlan wrote:

"We may want to tweak the porting note in What's New, as I believe this may read more environment variables at init time than the interpreter did previously, so embedding applications that want more complete control may now need to set Py_IgnoreEnvironmentFlag when they could previously get by without it."

#8659 (review)

I'm not sure about this, since it's really a bug of Python 3.7.0 and it's now fixed.

I let you decide if the issue should be closed or not, I leave it open.

[ncoghlan]

My comment wasn't about the 3.7.0 -> 3.7.1 fix, but rather about the fact that I suspect that 3.7.1 is now going to respect some environment variables in this case that 3.6.x ignored. I don't know for sure though, since we didn't have a test case for this until you wrote one.

So I was thinking we could include a porting note along the lines of:

===============

• starting in 3.7.1, `Py_Initialize` now consistently reads and respects all of the same environment settings as `Py_Main` (in earlier Python versions, it respected an ill-defined subset of those environment variables, while in Python 3.7.0 it didn't read any of them due to :issue:`34247`). If this behaviour is unwanted, set `Py_IgnoreEnvironmentFlag = 1` before calling `Py_Initialize`.
  ===============

That note then serves two purposes:

• it lets embedders know that Py_IgnoreEnvironmentFlag = 0 just plain doesn't work properly in 3.7.0 (without their needing to find this bug for themselves)
• it lets embedders know that they may want to set Py_IgnoreEnvironmentFlag = 1 if their embedded interpreter is now being affected by environmental settings that it ignored in previous versions

[ned-deily]

PR 9223 adds Nick's proposed wording to the 3.7 What's News.

[ned-deily]

New changeset 66755cb by Ned Deily in branch 'master':
bpo-34247: add porting note to 3.7 What's New (GH-9223)
66755cb

[vstinner]

(in earlier Python versions, it respected an ill-defined subset of those environment variables, ...)

I'm not aware of the Python 3.6 issue ("ill-defined"): which env vars were not properly handled?

[miss-islington]

New changeset 3050564 by Miss Islington (bot) in branch '3.7':
bpo-34247: add porting note to 3.7 What's New (GH-9223)
3050564

[ned-deily]

I've merged Nick's suggested porting note so I'm going to remove the "release blocker" status. Can we also close this now?

[ncoghlan]

The "ill-defined" in Python 3.6 relates to the fact that we never actually defined or tested which environment variables were read by Py_Main and which ones were read by Py_Initialize, since the majority of our tests only covered Py_Main (by launching a full Python subprocess).

Thus the only way to find out which environment variables fell into which category would be to read the Python 3.6 source code.

That's technically still the case for Python 3.7, but Victor's done some excellent refactoring work so it's much clearer in the code which vars may be updated if Py_Initialize is run a second time.

[vstinner]

The "ill-defined" in Python 3.6 relates to the fact that we never actually defined or tested which environment variables were read by Py_Main and which ones were read by Py_Initialize, since the majority of our tests only covered Py_Main (by launching a full Python subprocess).

Thus the only way to find out which environment variables fell into which category would be to read the Python 3.6 source code.

Oh ok, I see. Thanks for the explanation.

That's technically still the case for Python 3.7, but Victor's done some excellent refactoring work so it's much clearer in the code which vars may be updated if Py_Initialize is run a second time.

Well, for me the most important part is not the code, but tests. test_embed currently tests the following environment variables:

static void test_init_env_putenvs(void)
{
    putenv("PYTHONHASHSEED=42");
    putenv("PYTHONMALLOC=malloc_debug");
    putenv("PYTHONTRACEMALLOC=2");
    putenv("PYTHONPROFILEIMPORTTIME=1");
    putenv("PYTHONMALLOCSTATS=1");
    putenv("PYTHONUTF8=1");
    putenv("PYTHONVERBOSE=1");
    putenv("PYTHONINSPECT=1");
    putenv("PYTHONOPTIMIZE=2");
    putenv("PYTHONDONTWRITEBYTECODE=1");
    putenv("PYTHONUNBUFFERED=1");
    putenv("PYTHONNOUSERSITE=1");
    putenv("PYTHONFAULTHANDLER=1");
    putenv("PYTHONDEVMODE=1");
    /* FIXME: test PYTHONWARNINGS */
    /* FIXME: test PYTHONEXECUTABLE */
    /* FIXME: test PYTHONHOME */
    /* FIXME: test PYTHONDEBUG */
    /* FIXME: test PYTHONDUMPREFS */
    /* FIXME: test PYTHONCOERCECLOCALE */
    /* FIXME: test PYTHONPATH */
}

As you can see, the test suite is not complete yet. But since the tests are there, it shouldn't be hard to extend the test suite.

test_embed has InitConfigTests in 3.7 and master branches. I'm not interested to backport these tests to 3.6. I modified Python initialization deeply in 3.7, and the status of the code in 3.6 is "undefined". I don't think that it's worth it to backport these tests to 2.7 or 3.6. I success to focus on the master branch where we want to finish the implementation of the PEP-432 which should provide a better *public* API for that.

[ncoghlan]

Yep, exactly - things are much improved already, thanks primarily to your work, and it seems likely they'll be even further improved by the time 3.8.0 comes around :)

[vstinner]

New changeset 25d13f3 by Victor Stinner in branch 'master':
bpo-36142: PYTHONMALLOC overrides PYTHONDEV (GH-12191)
25d13f3