updates for the compiler package

[sxanth]

BPO	1206077
Nosy	@mwhudson, @terryjreedy
Files	compiler-1.patch: patch-testfile-doc

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2005-05-29.12:11:25.000>
created_at = <Date 2005-05-21.09:23:48.000>
labels = ['library']
title = 'updates for the compiler package'
updated_at = <Date 2005-05-29.12:11:25.000>
user = 'https://bugs.python.org/sxanth'

bugs.python.org fields:

activity = <Date 2005-05-29.12:11:25.000>
actor = 'sxanth'
assignee = 'none'
closed = True
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2005-05-21.09:23:48.000>
creator = 'sxanth'
dependencies = []
files = ['6668']
hgrepos = []
issue_num = 1206077
keywords = ['patch']
message_count = 8.0
messages = ['48372', '48373', '48374', '48375', '48376', '48377', '48378', '48379']
nosy_count = 3.0
nosy_names = ['mwh', 'terry.reedy', 'sxanth']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue1206077'
versions = ['Python 2.4']

[sxanth]

The compiler package produces lower quality bytecode
since it does not do
any of the peephole optimizations. I tried to fix that
to make the package
produce bytecode which is as good as the internal
compiler. Yeah, probably
wasted time but because python is so great it was much
easier than I thought
and now the package produces rather superior bytecode!

The patch implements all the optimizations mentioned in
compile.c and
additionally:

• constant folding (or however this is called). For
  example, it turns
  x = -(1<<2) to the constant -4. Also nested tuples
  of constants are
  converted to one big tuple constant.

• The optimization which checks JUMPs that lead to
  other JUMPs is extended
  to re-run the optimization when something is
  modified. For example,
  the builtin optimizer can't do:
  if (a and b) and c
  where there is a JUMP_IF_FALSE which leads to a
  JUMP_IF_FALSE which leads
  to a third JUMP_IF_FALSE.

While I was at it, I discovered several "bugs" which I
couldn't resist
attempting to fix.

• Implemented LIST_APPEND for list comprehensions

• The docstrings of functions where added in the
  co_consts of the module

• All the names that appeared in a function where added
  to co_varnames.
  Not sure about the fix

• Made co_filename and co_name interns

• In generator expressions the outmost-iterable's names
  where considered
  freevars and therefore in:
  def bar(y):
  return foo (x for x in y)
  the generator expression was considered a closure

I've used the new compiler for my own benchmark suite
and all went well.
I've also managed to just *compile* the standard
library and with the
new compiler it turns out 40167 bytes smaller (!). But
really, somebody
should run the regression tests with it (I can't due to
inefficient
hardware). IFF it passes the tests if *could* be used
to re-compile
the standard library for more efficient code.

The patch is vs 2.4.1.
However, the patch it's not 100% ready to be applied to
mainline; but it's easier to
review. The maintainer should tell me how to proceed :-)

Further Questions:

• generator expressions do not emit
  SETUP_LOOP/POP_BLOCK. Should they?

• What is the status of the package? I discovered all
  those bugs in a couple
  of hours. Does this mean that there are many many
  others? How many new ones
  has my patch added? Is the package not suitable for
  serious development?

[mwhudson]

Logged In: YES
user_id=6656

Why do you say this patch isn't ready to apply? Oh, I've
read it now :)

What is the status of the package?

Not as well looked after as it should be :-/

I'm not sure most of the changes you describe are really
bugs, more inefficiencies (e.g. compiler generated list
comps still work even if they don't emit LIST_APPEND).

Running "./python Lib/test/regrtest.py test_compiler -uall"
will compile the entire stdlib with Lib/compiler (this is
probably an overnight job) then running just plain "./python
Lib/test/regrtest.py" will run the test suite using the
compiled copy.

It probably makes sense to submit a number of patches,
particularly for the optimization versus straight fix divide.

[sxanth]

Logged In: YES
user_id=667962

Thanks for the instructions to regtest the compiler.

I was hoping for a quick review by someone who knows well
the compiler package to confirm that I'm doing the right
thing in the places with XXX. It'll save me a lot of time.

Hmmm. Since the patch is not ready to be applied, you can
close this one if you want and we'll keep the link as a
reference for future work.

[terryjreedy]

Logged In: YES
user_id=593130

If you think you might split and resubmit within a few months, you
could keep this open for further comments and then use it for the
optimization upgrades (updates). Put bug fixes in Bug Fixers ....

[terryjreedy]

Logged In: YES
user_id=593130

If you think you might split and resubmit within a few months, you
could keep this open for further comments and then use it for the
optimization upgrades (updates). Put bug fixes in Bug Fixers ....

[sxanth]

Logged In: YES
user_id=667962

I'm closing this one. Discovered many more things that can
be improved and the patch will eventually get bigger than
the entire package. I think it may be the best solution to
submit an complete reimplementation of the package.

[sxanth]

Logged In: YES
user_id=667962

Some comments after regtesting:

• The fix for generator expressions is *wrong* (grep the
  patch for the string 'genexp-1'). The right way to do this
  is to make the SymbolVisitor aware of the outmost iterable.

• The FoldConst optimizer must be prepared to handle
  ZeroDivisionError.

• It appears that the optimizer exposes some weird unicode
  bug. I can't find what's wrong so I suppose it's a python
  bug. Try to compile test_grammar to see it.

[sxanth]

Logged In: YES
user_id=667962

Correction: The bug is exposed by test_codecs, not test_grammar.