Add a test which uses the ast module to compile the stdlib

[brettcannon]

BPO	24981
Nosy	@brettcannon, @rhettinger, @terryjreedy, @1st1, @iritkatriel
Files	temast.py tem_ast2.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2015-09-01.20:49:15.728>
labels = ['tests']
title = 'Add a test which uses the ast module to compile the stdlib'
updated_at = <Date 2021-08-31.20:12:25.571>
user = 'https://github.com/brettcannon'

bugs.python.org fields:

activity = <Date 2021-08-31.20:12:25.571>
actor = 'iritkatriel'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Tests']
creation = <Date 2015-09-01.20:49:15.728>
creator = 'brett.cannon'
dependencies = []
files = ['40359', '40360']
hgrepos = []
issue_num = 24981
keywords = []
message_count = 8.0
messages = ['249516', '249526', '249817', '249818', '249819', '249826', '249839', '400775']
nosy_count = 5.0
nosy_names = ['brett.cannon', 'rhettinger', 'terry.reedy', 'yselivanov', 'iritkatriel']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'test needed'
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue24981'
versions = ['Python 3.6']

[brettcannon]

Issue bpo-24975 shows that we can easily end up with holes in support from the ast module. We should probably make sure we have a test that goes through the entire stdlib, Does an ast.parse(), and then passes the result to compile() to verify no exceptions are raised. We should either avoid the test/ directory or blacklist specific files which will fail because they are designed to be syntactically incorrect. The test can also be behind a -u flag if it turns out to be an expensive test.

[rhettinger]

+1 This is a great idea.

[terryjreedy]

The attached recurses one level deep in Lib. Catching UnicodeDecodeError on file read is needed at least for "test/test_source_encoding.py 'utf-8' codec can't decode byte 0xf0 in position 267: invalid continuation byte". Catching "badxxx.py" is needed for several test/ modules.

This test may have found another hole: It prints

"C:/programs/Python35/Lib\distutils _msvccompiler.py None disallowed in expression list"

whereas

>>> compile(open('C:/programs/Python35/Lib\distutils\_msvccompiler.py').read(), '', 'exec')
<code object <module> at 0x00000000034A3F60, file "", line 8>

[terryjreedy]

Forgot to mention: runtime 4 seconds.

[terryjreedy]

Ran with installed 3.5.0rc1.

[brettcannon]

Two things about the UTF-8 decode issue. One is you don't need to decode the source for it to be compiled; ast.parse()/compile() will work from bytes objects and thus handle the decoded for you. Two, if you want to decode source files into text, use importlib.util.decode_source().

Since Terry's script shows a complete compilation of the stdlib is very fast, we should make this a proper test. Any resulting test should probably should go into test_compile(). We can have it read all files and those that raise a a SyntaxError or some such exception can simply be skipped as the test is about AST -> code and not source -> AST (although we could theoretically have the test validate the failure with a source -> code compilation as well and verify the same exception is raised). We can also double-check that any AST -> code failure passes under source -> code and then report a failure (basically I'm trying to avoid blacklisting files that are test cases for the compiler overall and are known to fail).

[terryjreedy]

The distutils error is a ValueError, which was printed as such before I added 'Exception' to the except statement.

I did not initially catch SyntaxError separately because some are caught during the compile phase. Revised file corrects 'pass' to 'continue' and continues with parse errors also.

Many test/badsyntax files have compile-time syntax errors.

SyntaxError 'await' outside function (badsyntax_async1.py, line 1)
SyntaxError 'yield' inside async function (badsyntax_async6.py, line 2)
SyntaxError from __future__ imports must occur at the beginning of the file (badsyntax_future10.py, line 3)
SyntaxError future feature rested_snopes is not defined (badsyntax_future3.py, line 3)
SyntaxError not a chance (badsyntax_future9.py, line 3)

Attached is a fully recursive version that only report compile errors for non-'badsyntax' files. It takes about 2 seconds more. Output:

C:/programs/Python35/Lib\distutils _msvccompiler.py ValueError None disallowed in expression list
1776 (files compiled)

I excluded site-packages after testing on 3.4 because pylint has bad syntax examples giving a page of error messages.

[iritkatriel]

I think there is something like this here:

cpython/Lib/test/test_ast.py

Line 1498 in 22fe0eb

def test_stdlib_validates(self):

[iritkatriel]

Closing as out of date because test_stdlib_validates does this.