Bugs in PyImport_ImportModuleLevelObject

[serhiy-storchaka]

BPO	27419
Nosy	@brettcannon, @ncoghlan, @ericsnowcurrently, @berkerpeksag, @serhiy-storchaka
Files	fix-import.patch import_bug_1.py import_bug_2.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-08-03.05:16:26.853>
created_at = <Date 2016-06-30.08:28:21.453>
labels = ['interpreter-core', 'type-crash']
title = 'Bugs in PyImport_ImportModuleLevelObject'
updated_at = <Date 2018-08-03.05:16:26.851>
user = 'https://github.com/serhiy-storchaka'

bugs.python.org fields:

activity = <Date 2018-08-03.05:16:26.851>
actor = 'serhiy.storchaka'
assignee = 'none'
closed = True
closed_date = <Date 2018-08-03.05:16:26.853>
closer = 'serhiy.storchaka'
components = ['Interpreter Core']
creation = <Date 2016-06-30.08:28:21.453>
creator = 'serhiy.storchaka'
dependencies = []
files = ['43585', '43586', '43621']
hgrepos = []
issue_num = 27419
keywords = ['patch']
message_count = 10.0
messages = ['269564', '269565', '269805', '269813', '270624', '270630', '270631', '270632', '322993', '323021']
nosy_count = 6.0
nosy_names = ['brett.cannon', 'ncoghlan', 'python-dev', 'eric.snow', 'berker.peksag', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue27419'
versions = ['Python 3.5', 'Python 3.6']

[serhiy-storchaka]

1. If __package__ is None, and __spec__ is None or not set, and emitting ImportWarning raises an error, package is a borrowed reference to None, and it is decrefed. As result, the reference count of None is decremented.

2. __import__ is looked in globals or builtins and passed to _find_and_load(), _handle_fromlist() or called with highest package name. This doesn't match Python implementation. Python implementation uses _gcd_import which implements the original __import__, and isn't affected by overriding __import__ in globals or builtins.

[serhiy-storchaka]

Here is a reproducer for the first bug.

In Python 3.6:

$ ./python import_bug_1.py
1738
738
Fatal Python error: deallocating None

Current thread 0xb7552700 (most recent call first):
Aborted (core dumped)

In Python 3.5:

$ ./python import_bug_1.py
1709
Traceback (most recent call last):
  File "import_bug_1.py", line 10, in <module>
    from .foo import bar
SystemError: Parent module '' not loaded, cannot perform relative import

[serhiy-storchaka]

Here is a reproducer for the second bug.

In Python 2.7 and Python 3.2 all works.

In Python 3.3+:

$ ./python import_bug_2.py 
Traceback (most recent call last):
  File "import_bug_2.py", line 6, in <module>
    import os.path
  File "import_bug_2.py", line 4, in __import__
    raise AssertionError
AssertionError

[brettcannon]

The semantics for looking up __import__ is to only grab it from builtins and to ignore globals() and locals() (see issue bpo-25500), so any fix/change should follow those semantics.

[python-dev]

New changeset c88ec1bb67d0 by Serhiy Storchaka in branch '3.5':
Issue bpo-27419: Standard __import__() no longer look up "__import__" in globals
https://hg.python.org/cpython/rev/c88ec1bb67d0

New changeset d87f99e297d5 by Serhiy Storchaka in branch 'default':
Issue bpo-27419: Standard __import__() no longer look up "__import__" in globals
https://hg.python.org/cpython/rev/d87f99e297d5

[berkerpeksag]

I'm getting a segfault after d87f99e297d5 (SubinterpreterTest.test_subinterps).

$ ./python -m test test_capi
Run tests sequentially
0:00:00 [1/1] test_capi
Fatal Python error: Segmentation fault

Current thread 0x00007fd5e401d700 (most recent call first):
File "/home/berker/projects/cpython/default/Lib/test/support/init.py", line 2440 in run_in_subinterp
File "/home/berker/projects/cpython/default/Lib/test/test_capi.py", line 337 in test_subinterps
File "/home/berker/projects/cpython/default/Lib/unittest/case.py", line 600 in run
File "/home/berker/projects/cpython/default/Lib/unittest/case.py", line 648 in __call__
File "/home/berker/projects/cpython/default/Lib/unittest/suite.py", line 122 in run
File "/home/berker/projects/cpython/default/Lib/unittest/suite.py", line 84 in __call__
File "/home/berker/projects/cpython/default/Lib/unittest/suite.py", line 122 in run
File "/home/berker/projects/cpython/default/Lib/unittest/suite.py", line 84 in __call__
File "/home/berker/projects/cpython/default/Lib/unittest/suite.py", line 122 in run
File "/home/berker/projects/cpython/default/Lib/unittest/suite.py", line 84 in __call__
File "/home/berker/projects/cpython/default/Lib/test/support/init.py", line 1704 in run
File "/home/berker/projects/cpython/default/Lib/test/support/init.py", line 1805 in _run_suite
File "/home/berker/projects/cpython/default/Lib/test/support/init.py", line 1839 in run_unittest
File "/home/berker/projects/cpython/default/Lib/test/libregrtest/runtest.py", line 179 in test_runner
File "/home/berker/projects/cpython/default/Lib/test/libregrtest/runtest.py", line 180 in runtest_inner
File "/home/berker/projects/cpython/default/Lib/test/libregrtest/runtest.py", line 144 in runtest
File "/home/berker/projects/cpython/default/Lib/test/libregrtest/main.py", line 334 in run_tests_sequential
File "/home/berker/projects/cpython/default/Lib/test/libregrtest/main.py", line 406 in run_tests
File "/home/berker/projects/cpython/default/Lib/test/libregrtest/main.py", line 466 in _main
File "/home/berker/projects/cpython/default/Lib/test/libregrtest/main.py", line 446 in main
File "/home/berker/projects/cpython/default/Lib/test/libregrtest/main.py", line 508 in main
File "/home/berker/projects/cpython/default/Lib/test/main.py", line 2 in <module>
File "/home/berker/projects/cpython/default/Lib/runpy.py", line 85 in _run_code
File "/home/berker/projects/cpython/default/Lib/runpy.py", line 184 in _run_module_as_main
Segmentation fault (core dumped)

See also http://buildbot.python.org/all/builders/s390x%20RHEL%203.x/builds/1387/steps/test/logs/stdio for the same failure on buildbots.

[python-dev]

New changeset 4c07faa33915 by Serhiy Storchaka in branch '3.5':
Issue bpo-27419: Added temporary workaround for subinterpreters.
https://hg.python.org/cpython/rev/4c07faa33915

New changeset 5540234ca517 by Serhiy Storchaka in branch 'default':
Issue bpo-27419: Added temporary workaround for subinterpreters.
https://hg.python.org/cpython/rev/5540234ca517

[serhiy-storchaka]

Added temporary workaround to make buildbots working. The bug is partially reappeared in __import__ in subinterpreter. I think this is better than reverting all patches.

[berkerpeksag]

The workaround was removed in berkerpeksag@133138a Can this issue be closed now?

[serhiy-storchaka]

I think it can. Thanks Berker for reminder.