PYTHONDONTWRITEBYTECODE and PYTHONOPTIMIZE have no effect

[Perlence]

BPO	31845
Nosy	@ncoghlan, @ericsnowcurrently, @Perlence
PRs	bpo-31845: Fix reading flags from environment #4105 bpo-22257: Mention startup refactoring in What's New #4286

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ncoghlan'
closed_at = <Date 2017-10-25.02:14:21.810>
created_at = <Date 2017-10-23.11:25:30.491>
labels = ['interpreter-core', 'type-bug', '3.7']
title = 'PYTHONDONTWRITEBYTECODE and PYTHONOPTIMIZE have no effect'
updated_at = <Date 2017-11-05.04:58:48.549>
user = 'https://github.com/Perlence'

bugs.python.org fields:

activity = <Date 2017-11-05.04:58:48.549>
actor = 'ncoghlan'
assignee = 'ncoghlan'
closed = True
closed_date = <Date 2017-10-25.02:14:21.810>
closer = 'ncoghlan'
components = ['Interpreter Core']
creation = <Date 2017-10-23.11:25:30.491>
creator = 'Perlence'
dependencies = []
files = []
hgrepos = []
issue_num = 31845
keywords = ['patch']
message_count = 8.0
messages = ['304794', '304796', '304798', '304899', '304955', '304956', '305582', '305584']
nosy_count = 3.0
nosy_names = ['ncoghlan', 'eric.snow', 'Perlence']
pr_nums = ['4105', '4286']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue31845'
versions = ['Python 3.7']

[Perlence]

Setting PYTHONDONTWRITEBYTECODE doesn't seem to have an effect in Python 3.7.0a2:

    $ python -V
    Python 3.7.0a2
    $ env PYTHONDONTWRITEBYTECODE=1 python -c 'import sys; print(sys.dont_write_bytecode)'
    False

[Perlence]

Looks like PYTHONOPTIMIZE has no effect either:

    $ python -V
    Python 3.7.0a2
    $ env PYTHONOPTIMIZE=1 python -c 'import sys; print(sys.flags.optimize)'
    0

[Perlence]

It seems 1abcf67[1] is the first bad commit.

[1]1abcf67

[ncoghlan]

Huh, it appears the tests for these features are relying solely on the command line options, and not checking that the environment variables are also setting the flags properly. We should be able to account for that omission with a single new test in test_cmd_line that sets every relevant environment variable and checks for the expected sys.flags contents.

As far as what's actually going wrong goes, it's this sequence of events in Py_Main:

_Py_InitializeCore(&core_config);
... other code ...
apply_command_line_and_environment(&cmdline);

_Py_InitializeCore is setting the internal flags appropriately based on the runtime environment, but then Py_Main is stomping over those environmental settings with the settings from the command line.

Historically, these operations happened the other way around, so it was solely up to the code reading the environment variables to ensure they played nice with each other. Now the command line processing logic needs to be updated to also ensure that it only ever increases these values and never reduces them.

[ncoghlan]

New changeset d7ac061 by Nick Coghlan in branch 'master':
bpo-31845: Fix reading flags from environment (GH-4105)
d7ac061

[ncoghlan]

We could still use some more comprehensive test cases for the env var handling and the way that interacts with the command line settings, but the merged PR includes at least a rudimentary check for the four that directly affect sys.flags without any tricky side effects (PYTHONDEBUG, PYTHONVERBOSE, PYTHONOPTIMIZE, PYTHONDONTWRITEBYTECODE).

[ncoghlan]

Cross-linking to the work-in-progress RFE that introduced the error: https://bugs.python.org/issue22257

[ncoghlan]

New changeset 1b46131 by Nick Coghlan in branch 'master':
bpo-22257: Mention startup refactoring in What's New (GH-4286)
1b46131