Make import machinery explicit

[brettcannon]

BPO	14605
Nosy	@malemburg, @brettcannon, @ncoghlan, @ericvsmith, @merwok, @bitdancer, @ericsnowcurrently
Files	explicit.diff explicit_path_hooks.diff explicit_path_hooks.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/brettcannon'
closed_at = <Date 2012-04-27.20:03:12.259>
created_at = <Date 2012-04-17.15:58:04.465>
labels = ['interpreter-core', 'type-bug', 'release-blocker']
title = 'Make import machinery explicit'
updated_at = <Date 2012-04-27.20:03:12.259>
user = 'https://github.com/brettcannon'

bugs.python.org fields:

activity = <Date 2012-04-27.20:03:12.259>
actor = 'brett.cannon'
assignee = 'brett.cannon'
closed = True
closed_date = <Date 2012-04-27.20:03:12.259>
closer = 'brett.cannon'
components = ['Interpreter Core']
creation = <Date 2012-04-17.15:58:04.465>
creator = 'brett.cannon'
dependencies = []
files = ['25307', '25334', '25359']
hgrepos = []
issue_num = 14605
keywords = ['patch']
message_count = 33.0
messages = ['158552', '158996', '158997', '159107', '159114', '159115', '159145', '159149', '159157', '159177', '159195', '159234', '159236', '159237', '159240', '159242', '159244', '159245', '159246', '159248', '159253', '159262', '159263', '159327', '159334', '159337', '159343', '159346', '159347', '159482', '159486', '159487', '159488']
nosy_count = 9.0
nosy_names = ['lemburg', 'brett.cannon', 'ncoghlan', 'eric.smith', 'eric.araujo', 'Arfrever', 'r.david.murray', 'python-dev', 'eric.snow']
pr_nums = []
priority = 'release blocker'
resolution = None
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue14605'
versions = ['Python 3.3']

[brettcannon]

There should no longer be any implicit part of import when there doesn't have to be. To make import fully explicit, some things need to happen (in context to importlib):

• Expose FileLoader
• Expose SourceFileLoader
• Expose PathFinder
• Expose the extension loader
• Expose the default path hook
• Explicitly set sys.meta_path with PathFinder
• Explicitly set sys.path_hooks with the default path hook and zipimport
• Make sys.path_importer_cache view None as no finder discovered
• Insert None into sys.path_importer_cache instead of NullImporter

I am not exposing SourcelessFileLoader because importlib publicly tries to discourage the shipping of .pyc files w/o their corresponding source files. Otherwise all objects as used by importlib for performing imports will become public.

[brettcannon]

Attached is my (failing) attempt at making import explicit. Unfortunately I have four failing tests, 3 of which revolve around __main__ (which is why I added Nick to see if he had any ideas):

test_cmd_line_script test_runpy test_threaded_import test_zipimport_support

[python-dev]

New changeset 1da623513b26 by Brett Cannon in branch 'default':
Issue bpo-14605: Expose importlib.abc.FileLoader and
http://hg.python.org/cpython/rev/1da623513b26

[brettcannon]

Updated patch for an explicit sys.path_hooks that simply tweaks the test_cmd_line_script tests to stop requiring an absolute path on the files and instead verifies that the relative paths are legit. It also adds warnings when None is found in sys.path_importer_cache and then subsequently just tries sys.path_hooks again. It also warnings when sys.path_hooks is empty.

[ncoghlan]

I put those explicit path checks in there deliberately.

I really don't want to go back to having any __file__ attributes that are sensitive to sys.path changes.

[ncoghlan]

Oops: s/sensitive to sys.path changes/sensitive to current working directory changes/

[brettcannon]

Of course you did because you just like making my life a living hell when it comes to the __file__ attribute. =)

OK, I will have another look when I get home, but last time I dealt with this the issue is some tests expect a relative path while others expect an absolute one. And allowing an empty string for the current directory is just evil.

[malemburg]

Brett Cannon wrote:

I am not exposing SourcelessFileLoader because importlib publicly tries to discourage the shipping of .pyc files w/o their corresponding source files. Otherwise all objects as used by importlib for performing imports will become public.

What's the reasoning behind this idea ? Is Python 3.3 no longer meant to
be used for closed source applications ?

[brettcannon]

That initial comment is out-of-date. If you look that the commit I made I documented importlib.machinery._SourcelessFileLoader. I am continuing the discouragement of using bytecode files as an obfuscation technique (because it's a bad one), but I decided to at least document the class so people can use it at their own peril and know about it if they happen to come across the object during execution.

[malemburg]

Brett Cannon wrote:

That initial comment is out-of-date. If you look that the commit I made I documented importlib.machinery._SourcelessFileLoader. I am continuing the discouragement of using bytecode files as an obfuscation technique (because it's a bad one), but I decided to at least document the class so people can use it at their own peril and know about it if they happen to come across the object during execution.

It's not a perfect obfuscation technique, but a pretty simple and
(legally) effective one to use.

FWIW, I don't think the comment in the check-in is appropriate:

"""
1.127 + It is strongly suggested you do not rely on this loader (hence the
1.128 + leading underscore of the class). Direct use of bytecode files (and thus not
1.129 + source code files) inhibits your modules from being usable by all Python
1.130 + implementations. It also runs the risk of your bytecode files not being
1.131 + usable by new versions of Python which change the bytecode format. This
1.132 + class is only documented as it is directly used by import and thus can
1.133 + potentially have instances show up as a module's __loader__ attribute.
"""

The "risks" you mention there are really up to the application developers
to decide how to handle, not the Python developers. Python has a long
tradition of being friendly to commercial applications and I don't see
any reason why we should stop that.

If you do want this to change, please write a PEP. This may appear
to be a small change in direction, but it does in fact have quite
some impact on the usefulness of CPython in commercial settings.

I also think that the SourcelessFileLoader loader should be first class
citizen without the leading underscore if the importlib is to completely
replace the current import mechanism. Why force developers to write their
own loader instead of using the standard one just because of the leading
underscore, when it's only 20 lines of code ?

Thanks,

Marc-Andre Lemburg
eGenix.com

---

2012-04-28: PythonCamp 2012, Cologne, Germany 4 days to go

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::

eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611
http://www.egenix.com/company/contact/

[brettcannon]

I documented it explicitly so people can use it if they so choose (e.g. look at sys._getframe()). If you want to change this that's fine, but I am personally not going to put the effort in to rename the class, update the tests, and change the docs for this (we almost stopped allowing the importation of bytecode directly not so long ago but got push-back so we backed off).