Replace __import__ w/ importlib.__import__

[brettcannon]

BPO	2377
Nosy	@brettcannon, @ncoghlan, @pitrou, @vstinner, @ericvsmith, @benjaminp, @serwy, @merwok, @alex, @Trundle, @ericsnowcurrently
Dependencies	bpo-14605: Make import machinery explicit bpo-1559549: ImportError needs attributes for module and file name
Files	bench_startup.py c011ff345a78.diff 131d1d107f26.diff 0fdf350657e3.diff update_idle.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/brettcannon'
closed_at = <Date 2012-05-04.17:44:03.273>
created_at = <Date 2008-03-18.02:40:12.363>
labels = ['interpreter-core', 'type-bug', 'release-blocker']
title = 'Replace __import__ w/ importlib.__import__'
updated_at = <Date 2012-05-04.17:44:03.272>
user = 'https://github.com/brettcannon'

bugs.python.org fields:

activity = <Date 2012-05-04.17:44:03.272>
actor = 'brett.cannon'
assignee = 'brett.cannon'
closed = True
closed_date = <Date 2012-05-04.17:44:03.273>
closer = 'brett.cannon'
components = ['Interpreter Core']
creation = <Date 2008-03-18.02:40:12.363>
creator = 'brett.cannon'
dependencies = ['14605', '1559549']
files = ['24447', '24632', '24700', '25104', '25218']
hgrepos = ['34']
issue_num = 2377
keywords = ['patch']
message_count = 58.0
messages = ['63851', '139233', '147625', '149518', '152631', '152785', '152788', '152814', '152822', '153510', '153511', '153957', '153999', '154022', '154070', '154072', '154103', '154106', '154143', '154145', '154149', '154152', '154157', '154179', '154716', '154717', '154912', '154915', '154917', '155108', '155268', '155269', '155481', '155485', '155852', '155895', '155963', '156049', '156099', '157392', '157675', '157683', '157742', '158001', '158267', '158268', '158270', '158271', '158297', '158298', '158300', '158301', '158302', '158304', '158305', '158307', '158573', '158967']
nosy_count = 13.0
nosy_names = ['brett.cannon', 'ncoghlan', 'pitrou', 'vstinner', 'eric.smith', 'benjamin.peterson', 'roger.serwy', 'eric.araujo', 'Arfrever', 'alex', 'Trundle', 'python-dev', 'eric.snow']
pr_nums = []
priority = 'release blocker'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue2377'
versions = ['Python 3.3']

[brettcannon]

Python/import.c should be replaced by the implementation under
development contained in the py3k-importlib branch.

[brettcannon]

OK, so first step is to simply replace __import__ w/ importlib.__import__ using builtins.__import__ in order to make sure that all tests pass as expected. Can probably do this by doing the switch in Py_Initialize() somewhere.

Next step after that will be seeing if _io can be used by importlib w/ the import of os postponed until after importlib is bootstrapped.

[brettcannon]

Just some notes on the branch:

The FAILING file contains tests known to fail. There is a comment explaining the failure and possible fixes. If something makes it here and stays for a while it probably means upstream changes are needed (eg. test_pydoc needs ImportError to grow an attribute as to what import failed).

Python/pythonrun.c contains XXX markers of what is left to be done (although currently the zipimport comment is out-of-date and there needs to be one for exposing some APIs created in importlib.__init__). If you are looking for something to do, just search for XXX and try to tackle one of the comments.

[ericsnowcurrently]

AFAICT, bpo-1559549 is the ImportError attribute ticket.

[brettcannon]

OK, I'm down to a single bug to be solved to call this work a "success" (there are other test failures, but they are not overtly difficult to solve).

At this point the bootstrapping is failing in the face of sub-interpreters. Specifically, when I try to load the frozen importlib code in Py_NewInterpreter() it leads to an assertion failure in the GC code when handling references in a GC generation. I have zero experience with sub-interpreters and my GC experience is rusty, so if anyone can have a look at the code I would appreciate it to see if they can figure out why loading a frozen module (while in marshal) is leading to a GC assertion error.

[brettcannon]

Thanks to Benjamin, the test_capi assert failure is fixed. At this point the only failures are listed in the FAILING file and are (probably) minor.

[vstinner]

Is there a benchmark for import? How slow is importlib? :)

[brettcannon]

On Mon, Feb 6, 2012 at 17:06, STINNER Victor <report@bugs.python.org> wrote:

STINNER Victor <victor.stinner@haypocalc.com> added the comment:

Is there a benchmark for import? How slow is importlib? :)

importlib.test.benchmark

[vstinner]

bench_startup.py: short script to compute the best startup time (I wrote the original script for the hash collision issue, bpo-13703). Result on my PC:

• original: 22.2 ms
• importlib: 27.9 ms

So importlib adds an overhead of 25.7% in startup time.

[brettcannon]

Just a quick update. I have refactored importlib in the cpython repo to allow for implementing bits of importlib.__import__() and importlib._gcd_import() in C. This means that the built-in __import__() is now calling importlib underneath the covers. Eventually what is in importlib.__import__() and _gcd_import() directly will be translated into the equivalent C code. This will speed up accessing sys.modules. After that whatever is deemed on the critical path and worth rewriting in C can be done function by function. That, though, should wait for Python-level profiling and optimization before wasting one's time.

[brettcannon]

Add Nick since the refactoring of importlib.__import__() into functions was his idea.