test_lib2to3.test_parser.test_all_project_files() fails

[vstinner]

BPO	30117
Nosy	@warsaw, @vstinner, @benjaminp, @appeltel
PRs	bpo-30117: fix lib2to3 ParserIdempotency test #1242 [3.6] closes bpo-30117: fix lib2to3 ParserIdempotency test (GH-1242) #5443

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-01-30.18:22:28.051>
created_at = <Date 2017-04-20.12:45:21.197>
labels = ['easy', '3.7', 'tests']
title = 'test_lib2to3.test_parser.test_all_project_files() fails'
updated_at = <Date 2018-01-30.18:23:21.009>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2018-01-30.18:23:21.009>
actor = 'benjamin.peterson'
assignee = 'none'
closed = True
closed_date = <Date 2018-01-30.18:22:28.051>
closer = 'benjamin.peterson'
components = ['Tests']
creation = <Date 2017-04-20.12:45:21.197>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 30117
keywords = ['easy']
message_count = 8.0
messages = ['291970', '292059', '311246', '311262', '311264', '311266', '311287', '311288']
nosy_count = 4.0
nosy_names = ['barry', 'vstinner', 'benjamin.peterson', 'Eric Appelt']
pr_nums = ['1242', '5443']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue30117'
versions = ['Python 3.7']

[vstinner]

The test_lib2to3.test_parser.test_all_project_files() test fails and produces an annoying output. See the issue bpo-13125

Example of a buildbot:

0:11:09 [398/404] test_lib2to3 passed (39 sec)
--- /root/buildarea/3.x.angelico-debian-amd64/build/Lib/lib2to3/tests/data/bom.py	2017-02-11 12:20:43.532000000 +1100
+++ @	2017-04-20 22:05:37.157911808 +1000
@@ -1,2 +1,2 @@
-# coding: utf-8
+# coding: utf-8
 print "BOM BOOM!"

Example by running: ./python -m test -v test_lib2to3
---

test_all_project_files (lib2to3.tests.test_parser.TestParserIdempotency) ... /home/haypo/prog/python/master/Lib/lib2to3/tests/test_parser.py:393: UserWarning: ParseError on file /home/haypo/prog/python/master/Lib/lib2to3/main.py (bad input: type=22, value='=', context=('', (130, 38)))
  warnings.warn('ParseError on file %s (%s)' % (filepath, err))
/home/haypo/prog/python/master/Lib/lib2to3/tests/test_parser.py:393: UserWarning: ParseError on file /home/haypo/prog/python/master/Lib/lib2to3/tests/pytree_idempotency.py (bad input: type=22, value='=', context=('', (49, 33)))
  warnings.warn('ParseError on file %s (%s)' % (filepath, err))
--- /home/haypo/prog/python/master/Lib/lib2to3/tests/data/bom.py	2017-02-10 23:10:03.392778645 +0100
+++ @	2017-04-20 14:32:49.921613096 +0200
@@ -1,2 +1,2 @@
-# coding: utf-8
+# coding: utf-8
 print "BOM BOOM!"
expected failure

---

The test fails to parse the following code:
---

from __future__ import print_function
import sys
print("WARNING", file=sys.stderr)

---

whereas lib2to3 is able to parse the code. It seems like the code uses a lib2to3.tests.support.driver object. Maybe this object lacks the fixer which parses the __future__ imports?

The minimum fix would be to make the test quiet: remove the warning, don't log anything.

The best would be fix the test.

[appeltel]

I added a PR to fix these two (in my opinion) spurious failure conditions in the lib2to3.tests.test_parser.TestParserIdempotency test_parser test with the following changes to the test:

1. Use the same encoding found in the initial file to write a temp file for a diff. This retains the BOM if the encoding was initially utf-8-sig.

2. If the file cannot be parsed using the normal grammar, try again with no print statement which should succeed for valid files using future print_function

For case (1), the driver was correctly handling a BOM in a utf-8 file, but then the test was not writing a comparison file using 'utf-8-sig' to diff against, so the BOM got removed. I don't think that is the fault of the parser, and lib2to3 will retain the BOM.

For case (2), lib2to3 pre-detects the use of from __future__ import print_function or allows the user to force this interpretation with a -p flag, and then selects a different grammar with the print statement removed. That makes the test cases unfair to this test as the driver itself doesn't know which grammar to use. As a minimal fix, the test will try using a grammar with the print statement, and if that fails fall back on a grammar without it. A more thorough handling of the idempotency test would to be to parse all files using both grammars and ignore if one of the two failed but otherwise check both. I didn't think this was necessary but can change.

[benjaminp]

New changeset 14e976e by Benjamin Peterson (Eric Appelt) in branch 'master':
closes bpo-30117: fix lib2to3 ParserIdempotency test (GH-1242)
14e976e

[vstinner]

Failure on AMD64 Windows8.1 Non-Debug 3.x:

http://buildbot.python.org/all/#/builders/12/builds/585

======================================================================
ERROR: test_all_project_files (lib2to3.tests.test_parser.TestParserIdempotency)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release\build\lib\lib2to3\tests\test_parser.py", line 432, in test_all_project_files
    x = diff(filepath, new, encoding=encoding)
  File "D:\buildarea\3.x.ware-win81-release\build\lib\lib2to3\tests\test_parser.py", line 488, in diff
    return subprocess.call(['diff', '-u', fn, '@'], stdout=(subprocess.DEVNULL if verbose < 1 else None))
  File "D:\buildarea\3.x.ware-win81-release\build\lib\subprocess.py", line 304, in call
    with Popen(*popenargs, **kwargs) as p:
  File "D:\buildarea\3.x.ware-win81-release\build\lib\subprocess.py", line 756, in __init__
    restore_signals, start_new_session)
  File "D:\buildarea\3.x.ware-win81-release\build\lib\subprocess.py", line 1160, in _execute_child
    startupinfo)
FileNotFoundError: [WinError 2] The system cannot find the file specified

[vstinner]

Different failure on AMD64 Windows10 3.x:

======================================================================
FAIL: test_all_project_files (lib2to3.tests.test_parser.TestParserIdempotency)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "D:\buildarea\3.x.bolen-windows10\build\lib\lib2to3\tests\test_parser.py", line 434, in test_all_project_files
    self.fail("Idempotency failed: %s" % filepath)
AssertionError: Idempotency failed: D:\buildarea\3.x.bolen-windows10\build\lib\lib2to3\btm_matcher.py

[vstinner]

x86 Windows7 3.x:

http://buildbot.python.org/all/#/builders/58/builds/563

======================================================================
FAIL: test_nonexisting_with_pipes (test.test_subprocess.ProcessTestCase)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\test\test_subprocess.py", line 1194, in test_nonexisting_with_pipes
    self.assertEqual(stderr, "")
AssertionError: 'Traceback (most recent call last):\n  Fil[1129 chars]le\n' != ''
Diff is 1153 characters long. Set self.maxDiff to None to see it.

[benjaminp]

1e17d4a. I don't think you can pin test_subprocess on this change, though.

[benjaminp]

New changeset a23a2c5 by Benjamin Peterson in branch '3.6':
[3.6] closes bpo-30117: fix lib2to3 ParserIdempotency test (GH-1242) (GH-5443)
a23a2c5