Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PYTHONOPTIMIZE ignored in 3.7.0 when using custom launcher #78428

Closed
ronaldoussoren opened this issue Jul 27, 2018 · 27 comments
Closed

PYTHONOPTIMIZE ignored in 3.7.0 when using custom launcher #78428

ronaldoussoren opened this issue Jul 27, 2018 · 27 comments
Assignees
Labels
3.7 interpreter-core type-bug

Comments

@ronaldoussoren
Copy link
Contributor

@ronaldoussoren ronaldoussoren commented Jul 27, 2018

BPO 34247
Nosy @brettcannon, @ronaldoussoren, @ncoghlan, @vstinner, @ned-deily, @ericsnowcurrently, @miss-islington
PRs
  • #8521
  • #8659
  • #9223
  • #9275
  • #12191
  • Files
  • t.c
  • foo.py
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/vstinner'
    closed_at = <Date 2018-09-20.12:32:35.054>
    created_at = <Date 2018-07-27.12:07:29.981>
    labels = ['interpreter-core', 'type-bug', '3.7']
    title = 'PYTHONOPTIMIZE ignored in 3.7.0 when using custom launcher'
    updated_at = <Date 2019-03-06.11:51:55.385>
    user = 'https://github.com/ronaldoussoren'

    bugs.python.org fields:

    activity = <Date 2019-03-06.11:51:55.385>
    actor = 'vstinner'
    assignee = 'vstinner'
    closed = True
    closed_date = <Date 2018-09-20.12:32:35.054>
    closer = 'ncoghlan'
    components = ['Interpreter Core']
    creation = <Date 2018-07-27.12:07:29.981>
    creator = 'ronaldoussoren'
    dependencies = []
    files = ['47716', '47717']
    hgrepos = []
    issue_num = 34247
    keywords = ['patch', '3.7regression']
    message_count = 27.0
    messages = ['322487', '322519', '322520', '322521', '322522', '322523', '322524', '322529', '322532', '322533', '322550', '322562', '323068', '323070', '323135', '323144', '323145', '323156', '325172', '325281', '325283', '325288', '325289', '325862', '325874', '325877', '337294']
    nosy_count = 7.0
    nosy_names = ['brett.cannon', 'ronaldoussoren', 'ncoghlan', 'vstinner', 'ned.deily', 'eric.snow', 'miss-islington']
    pr_nums = ['8521', '8659', '9223', '9275', '12191']
    priority = None
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'behavior'
    url = 'https://bugs.python.org/issue34247'
    versions = ['Python 3.7']

    @ronaldoussoren
    Copy link
    Contributor Author

    @ronaldoussoren ronaldoussoren commented Jul 27, 2018

    Attached is a very basic launcher for python scripts. This script works as expected in Python 3.6, but not in 3.7.0.

    In particular, the launcher sets the environment variable PYTHONOPTIMIZE before calling Py_Initialize and that setting is ignored.

    I haven't tried testing with the tip of the tree yet.

    @ronaldoussoren ronaldoussoren added 3.7 interpreter-core type-bug labels Jul 27, 2018
    @ncoghlan
    Copy link
    Contributor

    @ncoghlan ncoghlan commented Jul 28, 2018

    This feels like a duplicate of https://bugs.python.org/issue31845 (see d7ac061 )

    Is this definitely with the 3.7.0 release, or is it potentially with one of the earlier alphas before that fix was applied?

    @ncoghlan
    Copy link
    Contributor

    @ncoghlan ncoghlan commented Jul 28, 2018

    Ah, wait - I see a relevant difference: the test case for bpo-31845 uses Py_Main, *not* Py_Initialize (since it's a command line test, not an embedding test).

    That means it's currently possible for the sequence of operations in

    _Py_InitializeEx_Private(int install_sigs, int install_importlib)
    to be wrong (or missing steps) without causing the test suite to fail.

    @ncoghlan
    Copy link
    Contributor

    @ncoghlan ncoghlan commented Jul 28, 2018

    Comparing to Python 3.6, we can see that the code to update the C global flags from the environment variables used to live directly in _PyInitializeEx_Private:

    _Py_InitializeEx_Private(int install_sigs, int install_importlib)

    In Python 3.7 that section (https://github.com/python/cpython/blob/3.7/Python/pylifecycle.c#L913 ) currently calls _PyCoreConfig_Read instead, which *only* updates the config struct - it doesn't update the C level global variables.

    By contrast, Py_Main eventually calls

    pymain_set_global_config(_PyMain *pymain, _Py_CommandLineDetails *cmdline)
    which handles writing the state from the config struct back to the global variables.

    @ncoghlan
    Copy link
    Contributor

    @ncoghlan ncoghlan commented Jul 28, 2018

    Next steps:

    • add a test case that runs the same sys.flags checks as in d7ac061, but as a test_embed embedding test using Py_Initialize rather than Py_Main
    • create a shared internal API that both Py_Main and Py_Initialize can use to write the environment variable settings back to the global variables

    @ncoghlan
    Copy link
    Contributor

    @ncoghlan ncoghlan commented Jul 28, 2018

    Ned, I think we should consider this a release blocker for 3.7.1

    @ncoghlan
    Copy link
    Contributor

    @ncoghlan ncoghlan commented Jul 28, 2018

    This seems closely related to the work Victor is already pursuing to resolve bpo-34170 for Python 3.8 (e.g. 56b29b6 ).

    The bpo-34170 changes are more invasive than we'd like for a maintenance release, but it would be preferable to keep at least any new Python 3.7 test cases somewhat aligned with their Python 3.8 counterparts.

    @ronaldoussoren
    Copy link
    Contributor Author

    @ronaldoussoren ronaldoussoren commented Jul 28, 2018

    I'm at the EuroPython sprints and currently working on a testcase for this. Should be a nice way to get back into actively contributing to CPython :-)

    @ronaldoussoren
    Copy link
    Contributor Author

    @ronaldoussoren ronaldoussoren commented Jul 28, 2018

    FYI: I've filed bpo-34255 while working on this, test_embed assumes that you're building in the source directory (which I usually don't do).

    @ronaldoussoren
    Copy link
    Contributor Author

    @ronaldoussoren ronaldoussoren commented Jul 28, 2018

    Interesting... pylifecycle.c uses code in main.c, I hadn't expected that.

    If I've read the code correctly Py_Initialize won't look at PYTHONOPTIMZE at all (and other environment variables that have a command-line equivalent). Those are read in main.c:cmdline_get_env_flags, which is only called (indirectly) from Py_Main and _Py_UnixMain.

    I'm note sure what's the correct solution for this.

    1. Reading PYTHONOPTIMZE (and related variables) could be done
      in _PyCoreConfig_Read, but that would then need to grow a flag to ensure
      that this won't replace values calculated earlier in the implementation
      of Py_Main

    2. Writing back the global variables is probably best done using a new
      function (lets _PyCoreConfig_SetGlobals), mirroring
      pymain_set_global_config but using a _PyCoreConfig config structure
      instead of the command-line structure used by the latter function.

    @ronaldoussoren
    Copy link
    Contributor Author

    @ronaldoussoren ronaldoussoren commented Jul 28, 2018

    I've created a pull request that seems to work properly, although I am not entirely sure that this is the right way to fix this.

    As I mentioned in the pull request I created it against the 3.7 branch because of bpo-34170, but would happily rebase it for master.

    @ncoghlan
    Copy link
    Contributor

    @ncoghlan ncoghlan commented Jul 28, 2018

    I believe Victor is at the EuroPython sprints as well, so if I'm right about that, I think the best option would be for the two of you to work out a resolution for 3.7.1 in person that at least keeps the test suites reasonably consistent, even if the implementations have diverged (the new tests already added for bpo-34170 were the main reason I stopped working on my initial patch for this).

    And agreed on fixing the current dependency inversion between pylifecycle and the py_main code - the functions needed by both Py_Main and Py_Initialize should be migrated down the stack into pylifecycle as part of the bpo-34170 work.

    @vstinner
    Copy link
    Member

    @vstinner vstinner commented Aug 3, 2018

    I first proposed to backport all code from master:
    https://mail.python.org/pipermail/python-dev/2018-July/154882.html

    But I'm not sure that it's a good idea, since I made *many* changes in the master branch since 3.7... I even added more files, and the _PyCoreConfig API is going to change anyway...

    So I wrote a simpler PR 8659 which backports unit tests, adapt them for Python 3.7, and fix this issue: PYTHONOPTIMIZE and other environment variables and now read by Py_Initialize(), not only by Py_Main().

    @vstinner
    Copy link
    Member

    @vstinner vstinner commented Aug 3, 2018

    I tested manually attached t.c with PR 8659: I get optimize=2 as expected. Full output:

    sys.flags(debug=0, inspect=0, interactive=0, optimize=2, dont_write_bytecode=0, no_user_site=0, no_site=0, ignore_environment=0, verbose=0, bytes_warning=0, quiet=0, hash_randomization=1, isolated=0, dev_mode=False, utf8_mode=1)

    @ronaldoussoren
    Copy link
    Contributor Author

    @ronaldoussoren ronaldoussoren commented Aug 5, 2018

    The PR works for me as well, and looks ok to me.

    @vstinner
    Copy link
    Member

    @vstinner vstinner commented Aug 5, 2018

    New changeset 0c90d6f by Victor Stinner in branch '3.7':
    [3.7] bpo-34247: Fix Python 3.7 initialization (bpo-8659)
    0c90d6f

    @vstinner
    Copy link
    Member

    @vstinner vstinner commented Aug 5, 2018

    On my PR, Nick Coghlan wrote:

    "We may want to tweak the porting note in What's New, as I believe this may read more environment variables at init time than the interpreter did previously, so embedding applications that want more complete control may now need to set Py_IgnoreEnvironmentFlag when they could previously get by without it."

    #8659 (review)

    I'm not sure about this, since it's really a bug of Python 3.7.0 and it's now fixed.

    I let you decide if the issue should be closed or not, I leave it open.

    @serhiy-storchaka serhiy-storchaka changed the title PYTHONOPTIMZE ignored in 3.7.0 when using custom launcher PYTHONOPTIMIZE ignored in 3.7.0 when using custom launcher Aug 5, 2018
    @ncoghlan
    Copy link
    Contributor

    @ncoghlan ncoghlan commented Aug 5, 2018

    My comment wasn't about the 3.7.0 -> 3.7.1 fix, but rather about the fact that I suspect that 3.7.1 is now going to respect some environment variables in this case that 3.6.x ignored. I don't know for sure though, since we didn't have a test case for this until you wrote one.

    So I was thinking we could include a porting note along the lines of:

    ===============

    • starting in 3.7.1, `Py_Initialize` now consistently reads and respects all of the same environment settings as `Py_Main` (in earlier Python versions, it respected an ill-defined subset of those environment variables, while in Python 3.7.0 it didn't read any of them due to :issue:`34247`). If this behaviour is unwanted, set `Py_IgnoreEnvironmentFlag = 1` before calling `Py_Initialize`.
      ===============

    That note then serves two purposes:

    • it lets embedders know that Py_IgnoreEnvironmentFlag = 0 just plain doesn't work properly in 3.7.0 (without their needing to find this bug for themselves)
    • it lets embedders know that they may want to set Py_IgnoreEnvironmentFlag = 1 if their embedded interpreter is now being affected by environmental settings that it ignored in previous versions

    @ned-deily
    Copy link
    Member

    @ned-deily ned-deily commented Sep 12, 2018

    PR 9223 adds Nick's proposed wording to the 3.7 What's News.

    @ned-deily
    Copy link
    Member

    @ned-deily ned-deily commented Sep 13, 2018

    New changeset 66755cb by Ned Deily in branch 'master':
    bpo-34247: add porting note to 3.7 What's New (GH-9223)
    66755cb

    @vstinner
    Copy link
    Member

    @vstinner vstinner commented Sep 13, 2018

    (in earlier Python versions, it respected an ill-defined subset of those environment variables, ...)

    I'm not aware of the Python 3.6 issue ("ill-defined"): which env vars were not properly handled?

    @miss-islington
    Copy link
    Contributor

    @miss-islington miss-islington commented Sep 13, 2018

    New changeset 3050564 by Miss Islington (bot) in branch '3.7':
    bpo-34247: add porting note to 3.7 What's New (GH-9223)
    3050564

    @ned-deily
    Copy link
    Member

    @ned-deily ned-deily commented Sep 13, 2018

    I've merged Nick's suggested porting note so I'm going to remove the "release blocker" status. Can we also close this now?

    @ncoghlan
    Copy link
    Contributor

    @ncoghlan ncoghlan commented Sep 20, 2018

    The "ill-defined" in Python 3.6 relates to the fact that we never actually defined or tested which environment variables were read by Py_Main and which ones were read by Py_Initialize, since the majority of our tests only covered Py_Main (by launching a full Python subprocess).

    Thus the only way to find out which environment variables fell into which category would be to read the Python 3.6 source code.

    That's technically still the case for Python 3.7, but Victor's done some excellent refactoring work so it's much clearer in the code which vars may be updated if Py_Initialize is run a second time.

    @vstinner
    Copy link
    Member

    @vstinner vstinner commented Sep 20, 2018

    The "ill-defined" in Python 3.6 relates to the fact that we never actually defined or tested which environment variables were read by Py_Main and which ones were read by Py_Initialize, since the majority of our tests only covered Py_Main (by launching a full Python subprocess).

    Thus the only way to find out which environment variables fell into which category would be to read the Python 3.6 source code.

    Oh ok, I see. Thanks for the explanation.

    That's technically still the case for Python 3.7, but Victor's done some excellent refactoring work so it's much clearer in the code which vars may be updated if Py_Initialize is run a second time.

    Well, for me the most important part is not the code, but tests. test_embed currently tests the following environment variables:

    static void test_init_env_putenvs(void)
    {
        putenv("PYTHONHASHSEED=42");
        putenv("PYTHONMALLOC=malloc_debug");
        putenv("PYTHONTRACEMALLOC=2");
        putenv("PYTHONPROFILEIMPORTTIME=1");
        putenv("PYTHONMALLOCSTATS=1");
        putenv("PYTHONUTF8=1");
        putenv("PYTHONVERBOSE=1");
        putenv("PYTHONINSPECT=1");
        putenv("PYTHONOPTIMIZE=2");
        putenv("PYTHONDONTWRITEBYTECODE=1");
        putenv("PYTHONUNBUFFERED=1");
        putenv("PYTHONNOUSERSITE=1");
        putenv("PYTHONFAULTHANDLER=1");
        putenv("PYTHONDEVMODE=1");
        /* FIXME: test PYTHONWARNINGS */
        /* FIXME: test PYTHONEXECUTABLE */
        /* FIXME: test PYTHONHOME */
        /* FIXME: test PYTHONDEBUG */
        /* FIXME: test PYTHONDUMPREFS */
        /* FIXME: test PYTHONCOERCECLOCALE */
        /* FIXME: test PYTHONPATH */
    }

    As you can see, the test suite is not complete yet. But since the tests are there, it shouldn't be hard to extend the test suite.

    test_embed has InitConfigTests in 3.7 and master branches. I'm not interested to backport these tests to 3.6. I modified Python initialization deeply in 3.7, and the status of the code in 3.6 is "undefined". I don't think that it's worth it to backport these tests to 2.7 or 3.6. I success to focus on the master branch where we want to finish the implementation of the PEP-432 which should provide a better *public* API for that.

    @ncoghlan
    Copy link
    Contributor

    @ncoghlan ncoghlan commented Sep 20, 2018

    Yep, exactly - things are much improved already, thanks primarily to your work, and it seems likely they'll be even further improved by the time 3.8.0 comes around :)

    @vstinner
    Copy link
    Member

    @vstinner vstinner commented Mar 6, 2019

    New changeset 25d13f3 by Victor Stinner in branch 'master':
    bpo-36142: PYTHONMALLOC overrides PYTHONDEV (GH-12191)
    25d13f3

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.7 interpreter-core type-bug
    Projects
    None yet
    Development

    No branches or pull requests

    5 participants