Benchmarks: 2to3 failures on the py3 side

[francismb]

BPO	13990
Nosy	@brettcannon, @pitrou, @benjaminp, @ezio-melotti
Files	FailuresErrors.txt: Failures and errors summary testsResult.txt testsResult2.txt

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/brettcannon'
closed_at = <Date 2012-09-12.21:08:52.325>
created_at = <Date 2012-02-10.17:32:33.848>
labels = ['expert-2to3', 'performance']
title = 'Benchmarks: 2to3 failures on the py3 side'
updated_at = <Date 2012-09-13.16:23:40.639>
user = 'https://bugs.python.org/francismb'

bugs.python.org fields:

activity = <Date 2012-09-13.16:23:40.639>
actor = 'francismb'
assignee = 'brett.cannon'
closed = True
closed_date = <Date 2012-09-12.21:08:52.325>
closer = 'brett.cannon'
components = ['2to3 (2.x to 3.x conversion tool)', 'Benchmarks']
creation = <Date 2012-02-10.17:32:33.848>
creator = 'francismb'
dependencies = []
files = ['24477', '27183', '27184']
hgrepos = []
issue_num = 13990
keywords = []
message_count = 11.0
messages = ['153069', '153073', '153135', '170374', '170420', '170431', '170432', '170434', '170440', '170443', '170445']
nosy_count = 6.0
nosy_names = ['brett.cannon', 'collinwinter', 'pitrou', 'benjamin.peterson', 'ezio.melotti', 'francismb']
pr_nums = []
priority = 'normal'
resolution = 'out of date'
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue13990'
versions = ['Python 3.2']

[francismb]

Steps to reproduce:
after following the usage instructions from make_perf3.sh go to the the py3 benchmarks directory and run the lib/2to3 tests (python3.2 test.py). The result are 3 failures and 17 errors. The equivalent procedure in the py2 benchmarks directory succeeds (python2.7 test.py).

A summary is attached (the whole paste at http://pastebin.com/PAAhGsdR)

Cheers,
francis

[brettcannon]

It looks like 2to3 isn't being converted properly. Benjamin, is lib2to3 supposed to work on itself and result in a sane output? If not then the benchmark should probably drop its internal copy of lib2to3.

Antoine (since you last looked at the benchmark suite for Python 3), do you know anything about this?

[pitrou]

Antoine (since you last looked at the benchmark suite for Python 3), do
you know anything about this?

I don't know. It works here. It may depend on the version of 2to3 used for the conversion, try passing PYTHON=some_newer_python when running make_perf.sh.

[brettcannon]

It's translating properly for me now as well (but on a different machine). I will give it another try when I get home to see if I can reproduce success on the machine that was failing for me previously.

[brettcannon]

I can't reproduce the problem anymore.

[francismb]

Just info:
I still can reproduce that. (It's maybe just something wrong with my setup?)

[francismb]

The actual tip for me is:

ci@random:~/Prog/cpython/benchmarks/py2$ hg tip
changeset: 164:61768f86170c
tag: tip
user: Brett Cannon <brett@python.org>
date: Fri Aug 31 18:58:24 2012 -0400
summary: Allow for the specification of a base directory for either interpreter

[brettcannon]

Can you copy-and-paste the exact commands you are using to build and run the benchmarks (along with version info for the interpreter used to do the building) as I'm on the same revision and can't reproduce anymore?

$ mkdir benchmarks_py3k
$ cd benchmarks_py3k/
$ $ /usr/bin/python2.6 -V
Python 2.6.5
$ PYTHON=/usr/bin/python2.6 ../benchmarks/make_perf3.sh ../benchmarks
# ... 2to3 output
$ python3 -V
Python 3.2.3
$ python3 perf.py -f -b 2to3 `which python3` `which python3`
Running 2to3...
INFO:root:Running /usr/bin/python3 lib/2to3/2to3 -f all lib/2to3/lib2to3/refactor.py
INFO:root:Running `['/usr/bin/python3', 'lib/2to3/2to3', '-f', 'all', 'lib/2to3/lib2to3/refactor.py']` 1 time
INFO:root:Running /usr/bin/python3 lib/2to3/2to3 -f all lib/2to3/lib2to3/refactor.py
INFO:root:Running `['/usr/bin/python3', 'lib/2to3/2to3', '-f', 'all', 'lib/2to3/lib2to3/refactor.py']` 1 time

Report on Linux importthis.wat.corp.google.com 3.2.5-gg944 #1 SMP Thu Aug 2 02:35:13 PDT 2012 x86_64 x86_64
Total CPU cores: 12

### 2to3 ###
0.540000 -> 0.560000: 1.04x slower

[francismb]

The box is a debian (wheezy):
ci@random:~/Prog/cpython/benchmarks/py2$ uname -a
Linux random 3.2.0-3-amd64 #1 SMP Mon Jul 23 02:45:17 UTC 2012 x86_64 GNU/Linux

The source repository dir:
ci@random:~/Prog/cpython/benchmarks/py2$ dir
lib LICENSE.txt make_perf3.sh performance perf.py perf.pyc README.txt test_perf.py

Then:

ci@random:/Prog/cpython/benchmarks/py2$ cd ..
ci@random:/Prog/cpython/benchmarks$ mkdir benchmarks_py3k
ci@random:/Prog/cpython/benchmarks$ dir
benchmarks_py3k py2
ci@random:/Prog/cpython/benchmarks$ cd benchmarks_py3k/
ci@random:/Prog/cpython/benchmarks/benchmarks_py3k$
ci@random:/Prog/cpython/benchmarks/benchmarks_py3k$ /usr/bin/python2.6 -V
Python 2.6.8
ci@random:/Prog/cpython/benchmarks/benchmarks_py3k$ PYTHON=/usr/bin/python2.6 ../py2/make_perf3.sh ../py2
RefactoringTool: Skipping implicit fixer: buffer
RefactoringTool: Skipping implicit fixer: idioms
...
RefactoringTool: Warnings/messages while refactoring:
RefactoringTool: ### In file lib/2to3/example.py ###
RefactoringTool: Line 371: You should use a for loop here
RefactoringTool: Line 372: You should use a for loop here
RefactoringTool: Line 373: You should use a for loop here
RefactoringTool: Line 374: You should use a for loop here
RefactoringTool: Line 375: You should use a for loop here
RefactoringTool: ### In file lib/2to3/example.py ###
RefactoringTool: Line 393: You should use 'hasattr(foo, 'call')' here.
RefactoringTool: Line 394: You should use 'operator.contains(foo, bar)' here.
RefactoringTool: Skipping implicit fixer: buffer
...
RefactoringTool: lib/mako/test/templates/subdir/foo/modtest.html.py
ci@random:/Prog/cpython/benchmarks/benchmarks_py3k$ python3 -V
Python 3.2.3
python3 perf.py -f -b 2to3 which python3 which python3
Running 2to3...
INFO:root:Running /usr/bin/python3 lib/2to3/2to3 -f all lib/2to3/lib2to3/refactor.py
INFO:root:Running ['/usr/bin/python3', 'lib/2to3/2to3', '-f', 'all', 'lib/2to3/lib2to3/refactor.py'] 1 time
INFO:root:Running /usr/bin/python3 lib/2to3/2to3 -f all lib/2to3/lib2to3/refactor.py
INFO:root:Running ['/usr/bin/python3', 'lib/2to3/2to3', '-f', 'all', 'lib/2to3/lib2to3/refactor.py'] 1 time

Report on Linux random 3.2.0-3-amd64 #1 SMP Mon Jul 23 02:45:17 UTC 2012 x86_64
Total CPU cores: 4

### 2to3 ###
0.616038 -> 0.624039: 1.01x slower

And further:
ci@random:/Prog/cpython/benchmarks/benchmarks_py3k$ dir
lib performance perf.py
cd lib/2to3
ci@random:/Prog/cpython/benchmarks/benchmarks_py3k/lib/2to3$ dir
2to3 example.py HACKING lib2to3 README scripts test.py
ci@random:~/Prog/cpython/benchmarks/benchmarks_py3k/lib/2to3$ python3 test.py 2> testsResult2.txt

[brettcannon]

So the tests failed but the benchmarks finished without issue? I'm not too worried about the tests since they are almost all str/bytes interaction issues (although I don't get the .decode() failure). If the benchmarks run without errors then I'm satisfied, else we figure out what this fails ever or we have a already translated copy kept in a lib3 directory to prevent problems.

[francismb]

So the tests failed but the benchmarks finished without issue?
Yes. But the apparent diff is that I've used 2.6.8

If the benchmarks run without errors then I'm satisfied, else we figure
out what this fails ever
or we have a already translated copy kept in a lib3 directory to prevent
problems.

The actual trunk revision for 2to3 from:
https://svn.python.org/projects/sandbox/trunk/2to3/ is
Revision 88981: /sandbox/trunk/2to3

the README.txt says:
2to3: r72994 from http://svn.python.org/projects/sandbox/trunk/2to3

I've just updated to that version but nothing changed: same errors (and
a new test test_formfeed (lib2to3.tests.test_parser.TestDriver) that was ok)

cheers,
francis