Move constant folding to AST level

[serhiy-storchaka]

BPO	26297
Nosy	@brettcannon, @birkenfeld, @ncoghlan, @vstinner, @benjaminp, @serhiy-storchaka, @1st1
Superseder	bpo-1346238: A constant folding optimization pass for the AST

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2016-02-07.18:10:55.489>
created_at = <Date 2016-02-05.17:34:07.306>
labels = ['interpreter-core', 'type-feature']
title = 'Move constant folding to AST level'
updated_at = <Date 2016-02-07.18:10:55.488>
user = 'https://github.com/serhiy-storchaka'

bugs.python.org fields:

activity = <Date 2016-02-07.18:10:55.488>
actor = 'serhiy.storchaka'
assignee = 'none'
closed = True
closed_date = <Date 2016-02-07.18:10:55.489>
closer = 'serhiy.storchaka'
components = ['Interpreter Core']
creation = <Date 2016-02-05.17:34:07.306>
creator = 'serhiy.storchaka'
dependencies = []
files = []
hgrepos = []
issue_num = 26297
keywords = []
message_count = 6.0
messages = ['259680', '259692', '259694', '259696', '259699', '259795']
nosy_count = 8.0
nosy_names = ['brett.cannon', 'georg.brandl', 'ncoghlan', 'vstinner', 'benjamin.peterson', 'serhiy.storchaka', 'yselivanov', 'abarnert']
pr_nums = []
priority = 'normal'
resolution = 'duplicate'
stage = 'resolved'
status = 'closed'
superseder = '1346238'
type = 'enhancement'
url = 'https://bugs.python.org/issue26297'
versions = ['Python 3.6']

[serhiy-storchaka]

For using more efficient bytecode (either specialized 8-bit opcodes or 16-bit opcodes) we need to move some optimizations from bytecode level to AST level, since LOAD_CONST variants could have variable size. Now with the Constant node this should be easy.

[vstinner]

*If* the PEP-511 is accepted, it will be super easy to start with an
implementation in pure Python.

But I think that we should benchmark the overhead of the Python API of
PEP-511, because we need to convert all AST internal objects to Python
objects, run the AST optimizer, and then again convert Python objects
to internal AST objects.

For a long running process, the time to compile a .py file doesn't
matter. For a short script, it matters. At least, we need to compile
the script itself.

By the way, would it be insane to *not* optimize the script when
running "python script.py"?

Now with the Constant node this should be easy.

ast.Constant is *not* emited by the compiler to not break backward
compatibility. I *know* that there is no stable API on AST, but I
noticed some issues when working on my AST project. For example, pip
doesn't work because an internal library uses AST and the code doesn't
handle ast.Constant (it's probably super easy to fix it).

I'm open to change the compiler to emit ast.Constant directly, but
maybe only in "optimized mode" (ex: python -O?). ast.Constant for
tuples and frozensets has a limitation: it doesn't store the location
of items (line number, column offset).

[1st1]

ast.Constant is *not* emited by the compiler to not break backward
compatibility. I *know* that there is no stable API on AST, but I
noticed some issues when working on my AST project. For example, pip
doesn't work because an internal library uses AST and the code doesn't
handle ast.Constant (it's probably super easy to fix it).

Yeah, it's probably very easy to fix. And if pip stops working on 3.6-alpha it will be fixed. We don't give any guarantees on AST backwards compatibility. In fact, we even tell that the AST might change in every release in the docs "The abstract syntax itself might change with each Python release".

I'm open to change the compiler to emit ast.Constant directly, but
maybe only in "optimized mode" (ex: python -O?).

I'm -1 on using -O flag since some people would prefer to have optimizations *and* have their assert statements.

In fact I'm -1 on using any kind of flag to trigger optimizations on or off. Those flags will only drive confusion. If the optimizations are any good, people will simply start using the flag all the time.

[vstinner]

Yury Selivanov added the comment:

In fact I'm -1 on using any kind of flag to trigger optimizations on or off. Those flags will only drive confusion. If the optimizations are any good, people will simply start using the flag all the time.

Yeah, Guido calls this a missed optimization opportunity.

My only concern is:

ast.Constant for tuples and frozensets has a limitation: it doesn't store the location of items (line number, column offset).

Is it a problem in practice? I didn't have time to investigate this.

Since the PEP-511 is still under discussion, I prefer to focus on the
PEP itself than this specific implementation detail ;-)

[vstinner]

we need to move some optimizations from bytecode level to AST level

You may be aware that there is a long list of similar (old) issues proposing something similar:
https://www.python.org/dev/peps/pep-0511/#usage-1-ast-optimizer

My PEP-511 tries to propose a generic API to support all proposed optimizations.

--

I didn't try yet to do something like importlib to compile an AST optimizer written in Python into a frozen module and then use it in Python. It would allow to implement an AST optimizer in pure Python to ease its maintenance *and* have it builtin into CPython, rather than having to play with AST in C (less fun).

My plan for the PEP-511 is more to start to experiment multiple AST optimizers and bytecode optimizers outside CPython, wait until the code becomes more stable, and later discuss if one optimizer deserves to become part of Python stdlib.

"Writing an optimizer or a preprocessor is out of the scope of this PEP."

--

But maybe you would prefer to have a single and simple AST optimizer builtin Python without new additional API?

I don't want to block your effort on experimentation :-) I just see a nice opportunity to sell you the PEP-511 ;-)

[serhiy-storchaka]

bpo-1346238 has a patch.