bpo-2506: Add -X noopt command line option #13600

vstinner · 2019-05-27T22:48:09Z

Add -X noopt command line option to disable compiler optimizations.

Add sys.flags.noopt flag.
importlib uses .noopt.pyc suffix for .pyc files
if sys.flags.noopt is true.
Add PyConfig.optimize parameter.
compiler: rename c_optimize to c_optimization_level,
add c_optimize.

Co-Authored-By: Yury Selivanov yury@magic.io

https://bugs.python.org/issue2506

vstinner · 2019-05-27T22:49:20Z

@nedbat: Would you mind to test this change, maybe also review it?

Doc/library/sys.rst

merwok · 2019-05-28T12:52:31Z

Doc/library/sys.rst

@@ -440,6 +441,9 @@ always available.
      Added ``dev_mode`` attribute for the new :option:`-X` ``dev`` flag
      and ``utf8_mode`` attribute for the new  :option:`-X` ``utf8`` flag.


If you want to change these two (and line 424-425) for consistent style :)

I chose to not change these on purpose, it would be an unrelated change. Someone can work on a PR to update these once this PR is merged ;-)

vstinner · 2019-05-28T16:16:39Z

I ran the Python test suite with -X noopt and I got multiple failures. I fixed all of them. I had to make additional changes:

importlib.util.cache_from_source() gets a new noopt keyword-only parameter.
Update subprocess._optim_args_from_interpreter_flags().
Add support.requires_compiler_optimizations()
Update unit tests which rely on compiler optimizations for noopt: add @support.requires_compiler_optimizations.

Doc/library/importlib.rst

vstinner · 2019-05-28T22:17:20Z

Azure Pipelines PR: test_venv failed with a timeout (20 min) in the win32 job, but passed when run again. It's unrelated.

vstinner · 2019-05-29T00:55:31Z

Oh wow. When I ran the test suite using -X noopt, I found more and more issues, and I had to modify more ane more code. I didn't expect that I would have to go up to compile() to add a new parameter. It is need indirectly by distutils which uses py_compile.

IMHO distutils must simply ignore noopt and always enable compiler optimizations. I don't see much value in distributing non optimized .pyc files on purpose. The common case is to expect optimized .pyc files.

I will not have time to finish this PR and review it carefully before Friday (Python 3.8 feature freeze), so I prefer to tag the PR as WIP.

brettcannon · 2019-05-30T22:21:55Z

I agree with @vstinner that worrying about compile() is unnecessary. If you view this as an analysis tool then the vast majority of it is going to be coming from .pyc files only.

merwok · 2019-05-30T22:43:38Z

It is unfortunate that in the current state of the code, you can’t control the compilation of pyc files in distutils build commands with options: the result depends on the -B/-O options passed to the python command itself.

I had changed it in distutils2 but thought it could not be changed in distutils.
https://hg.python.org/distutils2/rev/7c0a88497b5c

serhiy-storchaka

This is a very large change. Are all of them necessary? Could we encode this option as special optimization level or compiler flag? This might keep more code unchanged.

serhiy-storchaka · 2019-06-01T18:38:33Z

Doc/library/py_compile.rst

@@ -27,7 +27,7 @@ byte-code cache files in the directory containing the source code.
   Exception raised when an error occurs while attempting to compile the file.


-.. function:: compile(file, cfile=None, dfile=None, doraise=False, optimize=-1, invalidation_mode=PycInvalidationMode.TIMESTAMP)
+.. function:: compile(file, cfile=None, dfile=None, doraise=False, optimize=-1, invalidation_mode=PycInvalidationMode.TIMESTAMP, *, noopt=False)


optimize and noopt. It looks slightly weird.

I know but I failed to find better names.

Doc/c-api/init_config.rst

serhiy-storchaka · 2019-06-01T18:42:00Z

Include/compile.h

+    PyCompilerFlags *flags,
+    int optimization_level,
+    PyArena *arena,
+    int noopt);


Should not it be controlled by flags or optimization_level?

optimization_level=0 still means having optimizations enabled. I cannot change to keep the backward compatibility.

Do you think that it would be better to add a new PyCF_xxx flag to avoid adding this new function?

Lib/test/support/__init__.py

serhiy-storchaka · 2019-06-01T19:02:31Z

Python/compile.c

@@ -158,7 +158,8 @@ struct compiler {
    PyFutureFeatures *c_future; /* pointer to module's __future__ */
    PyCompilerFlags *c_flags;

-    int c_optimize;              /* optimization level */
+    int c_optimize;              /* optimize bytecode? */
+    int c_optimization_level;    /* optimization level */


Maybe use -1 for noopt?

-1 has a special meaning for compile functions. I prefer to have a separated field to avoid confusion.

vstinner · 2019-06-02T22:11:17Z

@serhiy-storchaka: "This is a very large change. Are all of them necessary? Could we encode this option as special optimization level or compiler flag? This might keep more code unchanged."

In GCC, gcc -O0 really means "no optimization". In Python, sadly, for historical reasons, optimization_level=0 still means "optimize but only with a few optimizations". It's not "don't optimize". And we cannot change the meaning of optimization_level=0: it would just break too much code. That's why I chose to add a new option.

I dislike optimization_level=-1. It sounds really surprising that an optimization level can be negative.

Add -X noopt command line option to disable compiler optimizations. distutils ignores noopt: always enable compiler optimizations. * Add sys.flags.noopt flag. * Add 'noopt' keyword-only parameter to builtin compile(), importlib.util.cache_from_source() and py_compile.compile(). * importlib: SourceLoader gets an additional keyword-only '_noopt' parameter. It is used by py_compile.compile(). * importlib uses ``.noopt.pyc`` suffix for .pyc files if sys.flags.noopt is true. * Add PyConfig.optimize parameter. * compiler: rename c_optimize to c_optimization_level, add c_optimize. * Update subprocess._optim_args_from_interpreter_flags(). * Add support.requires_compiler_optimizations() * Update unit tests which rely on compiler optimizations for noopt: add @support.requires_compiler_optimizations. * Add _PyAST_Compile() and _Py_CompileString() private functions: they have an additional 'noopt' parameter. * Set _Py_CONFIG_VERSION to 2 Co-Authored-By: Yury Selivanov <yury@magic.io>

vstinner · 2019-08-20T11:57:30Z

I rebased my PR:

The PR now targets Python 3.9: update the documentation, changes are now documented in What's New in Python 3.9
Document changes
Update compile.c for the new BEGIN_DO_NOT_EMIT_BYTECODE

Latest commit message:

commit fa659c13f8d000f7e8b229e5b701df99f778cdc6
Author: Victor Stinner <vstinner@redhat.com>
Date:   Mon May 27 23:58:28 2019 +0200

    [bpo-2506](https://bugs.python.org/issue2506): Add -X noopt command line option
    
    Add -X noopt command line option to disable compiler optimizations.
    
    distutils ignores noopt: always enable compiler optimizations.
    
    * Add sys.flags.noopt flag.
    * Add 'noopt' keyword-only parameter to
      builtin compile(), importlib.util.cache_from_source() and
      py_compile.compile().
    * importlib: SourceLoader gets an additional keyword-only '_noopt'
      parameter. It is used by py_compile.compile().
    * importlib uses ``.noopt.pyc`` suffix for .pyc files
      if sys.flags.noopt is true.
    * Add PyConfig.optimize parameter.
    * compiler: rename c_optimize to c_optimization_level,
      add c_optimize.
    * Update subprocess._optim_args_from_interpreter_flags().
    * Add support.requires_compiler_optimizations()
    * Update unit tests which rely on compiler optimizations for noopt:
      add @support.requires_compiler_optimizations.
    * Add _PyAST_Compile() and _Py_CompileString() private functions:
      they have an additional 'noopt' parameter.
    * Set _Py_CONFIG_VERSION to 2
    
    Co-Authored-By: Yury Selivanov <yury@magic.io>

vstinner · 2019-08-20T12:05:03Z

Open questions:

Do we really want this feature? See https://bugs.python.org/issue2506 for the rationale.
"optimize", "optimization_level", "noopt": naming is hard, I failed to find better names... The main blocker issue is that compile(optimize=0) does not disable optimization and that cannot be changed to keep the backward compatibility.
Do we need to modify compile(), importlib.util.cache_from_source() and py_compile.compile()? I began with a minimum change but I got many test failures. So slowly, I had to modify more and more code. IMO we need to modify these 3 functions.
How to handle PyConfig structure update? I changed _Py_CONFIG_VERSION from 1 to 2, but the codes doesn't implement ABI compatibility. This may be fixed in a separated PR: before or after this one.
compile.c: I added c_optimization_level field to the compiler structure, but I understood that @serhiy-storchaka would prefer to set compiler.c_optimize to -1 to disable optimizations. I chose to add a new field because -1 has a special meaning for function arguments. I prefer a separated field to avoid confusion.
compile.c: should we add a new PyCF_xxx flag rather than adding new _Py_CompileString() and _PyAST_Compile() functions?

This PR is quite large, but I'm not sure how to split it into smaller changes. I'm not sure that it can be splitted into smaller atomic changes, since all changes are related.

pablogsal · 2019-08-23T10:03:30Z

I chose to add a new field because -1 has a special meaning for function arguments. I prefer a separated field to avoid confusion.

I understand why you prefered that option, but I find it very confusing. I agree with Serhiy in that it would be ideal if we can use compiler.c_optimize to signal the no-opt state.

vstinner · 2019-09-28T02:56:38Z

Open questions:
How to handle PyConfig structure update? I changed _Py_CONFIG_VERSION from 1 to 2, but the codes doesn't implement ABI compatibility. This may be fixed in a separated PR: before or after this one.

So I tested and ... it didn't work at all :-( The PEP 587 had a flaw which prevented that! I just fixed the implementation to add a "struct_size" field to PyConfig. So it becomes possible to add new fields to PyConfig in Python 3.9, and be ABI compatible with Python 3.8.

The fix: https://bugs.python.org/issue38304

vstinner · 2019-10-01T08:46:51Z

Open questions: How to handle PyConfig structure update? I changed _Py_CONFIG_VERSION from 1 to 2, but the codes doesn't implement ABI compatibility. This may be fixed in a separated PR: before or after this one.

New update: the whole idea of a stable ABI for PyConfig has been abandoned. We can freely add new fields in Python 3.9 without having to care about backward compatibility.

vstinner · 2020-03-10T17:51:45Z

I'm no longer interesting to push this feature. The implementation is quite intrusive and @brettcannon didn't like that I passed the "no optimization" flag to so many places. I don't need this feature myself, so I'm not really motivated to fight each individual issue.

If someone wants to complete this change, feel free to copy/fork this PR. You can start from https://github.com/python/cpython/pull/13600.patch for example.

By the way, this PR was created from an old patch by Yury Selivanov.

vstinner requested review from methane, 1st1 and serhiy-storchaka May 27, 2019 22:48

the-knights-who-say-ni added the CLA signed label May 27, 2019

bedevere-bot added the awaiting core review label May 27, 2019

vstinner mentioned this pull request May 27, 2019

bpo-2506: Experiment with adding a "-X noopt" flag #9693

Closed

8 tasks

merwok reviewed May 28, 2019

View reviewed changes

Doc/library/sys.rst Outdated Show resolved Hide resolved

merwok reviewed May 28, 2019

View reviewed changes

vstinner requested a review from gpshead as a code owner May 28, 2019 16:15

tirkarthi reviewed May 28, 2019

View reviewed changes

Doc/library/importlib.rst Outdated Show resolved Hide resolved

asvetlov approved these changes May 28, 2019

View reviewed changes

bedevere-bot added awaiting merge and removed awaiting core review labels May 28, 2019

vstinner added the DO-NOT-MERGE label May 29, 2019

vstinner changed the title ~~bpo-2506: Add -X noopt command line option~~ [WIP] bpo-2506: Add -X noopt command line option May 29, 2019

serhiy-storchaka reviewed Jun 1, 2019

View reviewed changes

nedbat mentioned this pull request Jun 14, 2019

bpo-37271: Optimize bytecode multiple times until it cannot be optimized further #14068

Closed

vstinner removed the DO-NOT-MERGE label Aug 20, 2019

vstinner changed the title ~~[WIP] bpo-2506: Add -X noopt command line option~~ bpo-2506: Add -X noopt command line option Aug 20, 2019

vstinner and others added 2 commits August 20, 2019 13:46

Refactor requires_compiler_optimizations()

d3bcde8

vstinner closed this Mar 10, 2020

vstinner deleted the noopt branch March 10, 2020 17:51

vstinner mentioned this pull request Apr 22, 2020

bpo-39562: Prevent collision of future and compiler flags #19230

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bpo-2506: Add -X noopt command line option #13600

bpo-2506: Add -X noopt command line option #13600

vstinner commented May 27, 2019 •

edited by bedevere-bot

Loading

vstinner commented May 27, 2019

merwok May 28, 2019

vstinner May 28, 2019

vstinner commented May 28, 2019

vstinner commented May 28, 2019

vstinner commented May 29, 2019

brettcannon commented May 30, 2019

merwok commented May 30, 2019

serhiy-storchaka left a comment

serhiy-storchaka Jun 1, 2019

vstinner Aug 20, 2019

serhiy-storchaka Jun 1, 2019

vstinner Aug 20, 2019

serhiy-storchaka Jun 1, 2019

vstinner Aug 20, 2019

vstinner commented Jun 2, 2019 •

edited by brettcannon

Loading

vstinner commented Aug 20, 2019 •

edited by bedevere-bot

Loading

vstinner commented Aug 20, 2019

pablogsal commented Aug 23, 2019

vstinner commented Sep 28, 2019

vstinner commented Oct 1, 2019

vstinner commented Mar 10, 2020 •

edited by brettcannon

Loading

		@@ -440,6 +441,9 @@ always available.
		Added ``dev_mode`` attribute for the new :option:`-X` ``dev`` flag
		and ``utf8_mode`` attribute for the new :option:`-X` ``utf8`` flag.

bpo-2506: Add -X noopt command line option #13600

bpo-2506: Add -X noopt command line option #13600

Conversation

vstinner commented May 27, 2019 • edited by bedevere-bot Loading

vstinner commented May 27, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vstinner commented May 28, 2019

vstinner commented May 28, 2019

vstinner commented May 29, 2019

brettcannon commented May 30, 2019

merwok commented May 30, 2019

serhiy-storchaka left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vstinner commented Jun 2, 2019 • edited by brettcannon Loading

vstinner commented Aug 20, 2019 • edited by bedevere-bot Loading

vstinner commented Aug 20, 2019

pablogsal commented Aug 23, 2019

vstinner commented Sep 28, 2019

vstinner commented Oct 1, 2019

vstinner commented Mar 10, 2020 • edited by brettcannon Loading

vstinner commented May 27, 2019 •

edited by bedevere-bot

Loading

vstinner commented Jun 2, 2019 •

edited by brettcannon

Loading

vstinner commented Aug 20, 2019 •

edited by bedevere-bot

Loading

vstinner commented Mar 10, 2020 •

edited by brettcannon

Loading