Deprecate __loader__, __package__, and __cached__ on modules

[brettcannon]

BPO	33277
Nosy	@warsaw, @brettcannon, @rhettinger, @ncoghlan, @pitrou, @pmp-p, @ericsnowcurrently, @FFY00, @iritkatriel
PRs	bpo-33277: Hack to proxy module attributes through their spec #29078

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2018-04-13.21:48:22.091>
labels = ['library', '3.11']
title = 'Deprecate __loader__, __package__, and __cached__ on modules'
updated_at = <Date 2021-10-21.16:58:47.610>
user = 'https://github.com/brettcannon'

bugs.python.org fields:

activity = <Date 2021-10-21.16:58:47.610>
actor = 'eric.snow'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2018-04-13.21:48:22.091>
creator = 'brett.cannon'
dependencies = []
files = []
hgrepos = []
issue_num = 33277
keywords = ['patch']
message_count = 17.0
messages = ['315269', '315276', '315521', '315534', '315550', '315585', '315620', '315635', '398926', '398938', '404473', '404475', '404510', '404554', '404555', '404556', '404623']
nosy_count = 9.0
nosy_names = ['barry', 'brett.cannon', 'rhettinger', 'ncoghlan', 'pitrou', 'pmpp', 'eric.snow', 'FFY00', 'iritkatriel']
pr_nums = ['29078']
priority = 'low'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue33277'
versions = ['Python 3.11']

[brettcannon]

If you look at https://docs.python.org/3/reference/import.html#import-related-module-attributes you will notice there are a lot of attributes on modules. But since the introduction of module specs (https://docs.python.org/3/library/importlib.html#importlib.machinery.ModuleSpec), all of those attributes became redundant.

Now some attributes can never be taken away based on common idioms in the Python community. name can't go due to if __name__ == '__main__'. path can't go due to people manipulating it with e.g. pkg_resources or pkgutil. The rest, though, don't have such strong idioms tied to them (and in the case of package, has actually been made officially redundant by the import system since Python 3.6). That means setting them is unnecessary and any time someone wants to read or potentially manipulate them they have to choose between those attributes or the spec (or update both). Either way it can lead to bugs (I know I have nearly forgotten to set both the spec and the attribute before).

I don't know if we can easily deprecate these attributes using __getattr__ on modules as added in Python 3.7, or if this is more of a documentation thing and need a long period of clearly messaging that these redundant attributes will no longer be set at some point.

[warsaw]

I agree. It should also be pointed out that we've had inconsistencies between the module attributes and the spec attributes, and even fixing those has lead to problems. There should be a single source of truth, and the module spec should be that.

+1, and I agree about not being able to remove __name__ and __path__.

[ncoghlan]

-1 from me for __package__, as setting that's the preferred way for directly executed scripts to fix their explicit relative imports (rather than trying to retroactively fix their spec). (https://docs.python.org/3/library/importlib.html also doesn't say anything about __package__ being deprecated)

I'd be OK with seeing __file__/cached/loader get deprecated, but before they went away, I'd like to see "importlib.util.get_filename()", "importlib.util.get_cache_filename()", and "importlib.util.get_loader()" helpers added.

[rhettinger]

+1 from me. The visual clutter in dir() has proven to be somewhat distracting in introduction-to-python courses.

[ncoghlan]

As a concrete proposal for 3.8, how about:

1. Add the following utility functions to importlib.util (names adjusted to match PEP-451 rather than my initial suggestions above):

    def get_location(module):
        try:
            return module.__file__
        except AttributeError:
            pass
        spec = module.__spec__
        if spec.has_location:
            return spec.origin
        return None

    def get_cached_location(module):
        try:
            return module.__cached__
        except AttributeError:
            pass
        spec = module.__spec__
        if spec.has_location:
            return spec.cached
        return None

    def get_loader(module):
        try:
            return module.__loader__
        except AttributeError:
            pass
        return module.__spec__.loader

2. Explicitly make file/cached/loader optional, and require 3rd party tools to use the relevant importlib.util.get_* APIs when they're defined rather than accessing the dunder-attributes directly.

I think those 3 are clear, but the right API for replacing direct access to __package__ is less clear, since there are two cases for querying the package name: whether you want actual packages to return their own name, or whether you want the name of their parent.

If we decide the relevant use case is "get the base package name for relative imports", then the following helper API may work:

    def get_base_package_name(module):
        """Get the package name to pass to resolve_name() for this module"""
        try:
            return module.__package__
        except AttributeError:
            pass
        modname = module.__name__
        try:
            path = module.__path__
        except AttributeError:
            pass
        else:
            return modname
        return module.__spec__.parent

[pitrou]

This sounds like gratuitous breakage to me. __file__ in particular is used quite often.

[warsaw]

Not that I've thought this through in much detail, but what if we start out by proxying those module attributes to the spec via PEP-562 hooks? We can do that in 3.8, issue (silent?) deprecation warnings, and then do a full deprecation process for those module attributes?

It's not a terrible thing to keep those attributes as long as they match what's in the spec. It's a problem when they're out of sync (see bpo-32305 and bpo-32303).

[brettcannon]

@nick: We've had this discussion before about specs being a "receipt" of import versus not and the last time we agreed on not. :) That's why we raise a warning if __package__ doesn't match __spec__.parent:

cpython/Lib/importlib/_bootstrap.py

Lines 1056 to 1059 in d5a2377

	if spec is not None and package != spec.parent:
	_warnings.warn("__package__ != __spec__.parent "
	f"({package!r} != {spec.parent!r})",
	ImportWarning, stacklevel=3)

It's documented through https://docs.python.org/3/reference/import.html#__package__ by saying __package__ has to match __spec__.parent. We didn't make it stronger since we can't get rid of any of these attributes until Python 2.7 compatibility is no longer a concern. And I believe __package__/spec.parent is the only attribute whose data is in any way directly relied on in importlib which is being proposed for removal (e.g. __file__, __cached__, and __loader__ are just recorded).

@Raymond: Thanks for the teaching info. I actually opened this after running dir() on a module and having there be just as many module-related attributes as defined attributes.

@antoine: we can keep __file__ if necessary (I was on the fence about keeping it to begin with). But as Barry pointed out, bugs have occurred because we now have two places for most of these details and people don't always know which one to update.

@barry: the tricky bit is what to do if someone sets their own __getattr__() on a module? E.g. I just published modutil which provides a __getattr__ for lazy importing, so users of this would need a way to not have either function get stomped on by the other.

[iritkatriel]

We didn't make it stronger since we can't get rid of any of these attributes until Python 2.7 compatibility is no longer a concern.

Maybe this can be revisited now.

[brettcannon]

Maybe this can be revisited now.

I've started the work already (albeit rather slowly 😄). Importlib has been updated to prefer __spec__.parent of __package__ and warns when the values are not equal. Next step will be to raise an ImportWarning when __package__ is defined but __spec__.parent is not (then DeprecationWarning after that).

For __loader__ I got Python 3.10 to fall back on to __spec__.loader. Next step is https://bugs.python.org/issue42132 to update C code to set __spec__ where appropriate. From there the check for equivalence can go in (which then leads to the warnings).

I have not started with __cached__ yet, so next step is making sure all uses fall back on __spec__.

So, to be clear, the steps I have been following are:

1. Make sure all code falls back on __spec__ when appropriate (one release)
2. Raise an ImportWarning when values differ (one release)
3. Make __spec__ take precedence over old attribute (one release)
4. Raise an ImportWarning when having to fall back to older attribute (two releases)
5. Raise a DeprecationWarning when falling back on older attribute (two releases)
6. Clean up code