modulefinder fails if module contains syntax error

[jgosmann]

BPO	17396
Nosy	@brettcannon, @ncoghlan, @merwok, @bitdancer, @florentx, @ericsnowcurrently, @berkerpeksag
PRs	bpo-35936: Updates to modulefinder #11787 bpo-35936: Updates to modulefinder #11787 bpo-35936: Updates to modulefinder #11787 bpo-35936: Updates to modulefinder #11787
Files	fix-handling-of-syntax-errors.diff: Reraise SyntaxError as ImportError if imported module code cannot be compiled fix-handling-of-syntax-errors.diff: Reraise SyntaxError as ImportError if imported module code cannot be compiled

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-04-07.08:06:01.330>
created_at = <Date 2013-03-11.12:03:45.792>
labels = ['3.8', 'type-bug', 'library']
title = 'modulefinder fails if module contains syntax error'
updated_at = <Date 2019-04-07.08:06:01.323>
user = 'https://bugs.python.org/jgosmann'

bugs.python.org fields:

activity = <Date 2019-04-07.08:06:01.323>
actor = 'ncoghlan'
assignee = 'none'
closed = True
closed_date = <Date 2019-04-07.08:06:01.330>
closer = 'ncoghlan'
components = ['Library (Lib)']
creation = <Date 2013-03-11.12:03:45.792>
creator = 'jgosmann'
dependencies = []
files = ['29376', '30237']
hgrepos = []
issue_num = 17396
keywords = ['patch']
message_count = 8.0
messages = ['183953', '187938', '187941', '187960', '187961', '189059', '189060', '339562']
nosy_count = 8.0
nosy_names = ['brett.cannon', 'ncoghlan', 'eric.araujo', 'r.david.murray', 'flox', 'eric.snow', 'berker.peksag', 'jgosmann']
pr_nums = ['11787', '11787', '11787', '11787']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue17396'
versions = ['Python 3.8']

[jgosmann]

Running modulefinder.ModuleFinder.run_script on a script which has an import statement with a module containing a syntax error will raise a SyntaxError exception. I think, modulefinder should instead continue its analysis and add the module to badmodules. Especially, as there are valid reasons for importing modules with syntax errors like in the following code snippet

if not python3:
    from .exec_py2 import exec_
else:
    from .exec_py3 import exec_

I attached a patch which changes the code to catch potential SyntaxError exceptions and change them to an ImportError.

[merwok]

Thanks for the report and patch. Could you add a test?

I’m not sure where this sits on the bug vs. feature line.

[jgosmann]

Could you point me to some documentation on how to add a test? I have not been involved in Python development so far.

[merwok]

Certainly: http://docs.python.org/devguide contains information to get the source code, describes the files layout and explains how to generate a patch. You’ll find existing unit tests in Lib/test/test_modulefinder.py

Feel free to ask any question you might have on this bug, or if you prefer on the friendly core-mentorship mailing list (where I am).

[merwok]

I would recommend that you base your patch on the default branch, i.e. what will become Python 3.4. If we judge that this is not a new feature but actually a bug fix, the core developer who commits the patch will take care of backporting it to 2.7 and 3.3.

[jgosmann]

Here is an updated patch, also containing a test.

[jgosmann]

It's based on the default branch becoming 3.4.

[ncoghlan]

Fixed for Python 3.8 via the patch for bpo-35936. We won't be backporting that patch as it also migrates modulefinder from the deprecated imp API to support importlib one, and is hence considered overly intrusive for a bug fix release.