Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

importlib.invalidate_caches() does not invalidate _NamespacePath's _last_parent_path-based cache #89866

Closed
hroncok mannequin opened this issue Nov 3, 2021 · 11 comments
Closed
Labels
3.11 bug and security fixes stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error

Comments

@hroncok
Copy link
Mannequin

hroncok mannequin commented Nov 3, 2021

BPO 45703
Nosy @vstinner, @encukou, @hroncok, @miss-islington
PRs
  • bpo-45703: Invalidate _NamespacePath cache on importlib.invalidate_ca… #29384
  • [3.9] bpo-45703: Invalidate _NamespacePath cache on importlib.invalidate_ca… (GH-29384) #29964
  • [3.10] bpo-45703: Invalidate _NamespacePath cache on importlib.invalidate_cache (GH-29384) #30922
  • [3.9] bpo-45703: Invalidate _NamespacePath cache on importlib.invalidate_cache (GH-29384) (GH-30922) #31076
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2022-02-02.14:12:06.305>
    created_at = <Date 2021-11-03.11:21:56.843>
    labels = ['type-bug', 'library', '3.11']
    title = "importlib.invalidate_caches() does not invalidate _NamespacePath's _last_parent_path-based cache"
    updated_at = <Date 2022-02-02.14:12:06.304>
    user = 'https://github.com/hroncok'

    bugs.python.org fields:

    activity = <Date 2022-02-02.14:12:06.304>
    actor = 'petr.viktorin'
    assignee = 'none'
    closed = True
    closed_date = <Date 2022-02-02.14:12:06.305>
    closer = 'petr.viktorin'
    components = ['Library (Lib)']
    creation = <Date 2021-11-03.11:21:56.843>
    creator = 'hroncok'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 45703
    keywords = ['patch']
    message_count = 11.0
    messages = ['405606', '405607', '405620', '406846', '406848', '406851', '407936', '411368', '411759', '411877', '412359']
    nosy_count = 4.0
    nosy_names = ['vstinner', 'petr.viktorin', 'hroncok', 'miss-islington']
    pr_nums = ['29384', '29964', '30922', '31076']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'behavior'
    url = 'https://bugs.python.org/issue45703'
    versions = ['Python 3.11']

    @hroncok
    Copy link
    Mannequin Author

    hroncok mannequin commented Nov 3, 2021

    Recently, when debugging a weird problem (see https://bugzilla.redhat.com/show_bug.cgi?id=2018551 for details if interested, but not important for this issue), I've realized that the _NamespacePath class (from importlib/_bootstrap_external.py) has a cache that (in some cases) uses tuple(sys.path) as the key. I was expecting importlib.invalidate_caches() to invalidate this cache, but it doesn't.

    Consider the following directory structure:

    .
    ├── PATH1
    │   └── namespace
    │   └── sub1
    │   └── __init__.py
    └── PATH2
    └── namespace
    └── sub2
    └── __init__.py

    Here is a helper to create it (on Linux-ish):

    $ mkdir -p PATH1/namespace/sub1
    $ mkdir -p PATH2/namespace/sub2
    $ touch PATH1/namespace/sub1/__init__.py
    $ touch PATH2/namespace/sub2/__init__.py

    Run Python with PYTHONPATH=PATH1:PATH2 (output slightly formatted for readability):

    $ PYTHONPATH=PATH1:PATH2 python3.11
    >>> import namespace
    >>> namespace.__path__
    _NamespacePath(['.../namespace_path_cache/PATH1/namespace',
                    '.../namespace_path_cache/PATH2/namespace'])
    >>> import namespace.sub1  # works
    >>> import namespace.sub2  # works
    >>> exit()

    The namespace packages seem to work as expected.

    Now move PATH2/namespace out of the way:

    $ mv PATH2/namespace PATH2/cant-import-this

    Run Python again:

    $ PYTHONPATH=PATH1:PATH2 python3.11
    >>> import namespace
    >>> namespace.__path__
    _NamespacePath(['.../namespace_path_cache/PATH1/namespace'])
    >>> ...

    While this interpreter still runs, move the PATH2/namespace module back in:

    $ mv PATH2/cant-import-this PATH2/namespace
    >>> ...
    >>> namespace.__path__
    _NamespacePath(['.../namespace_path_cache/PATH1/namespace'])
    >>> import namespace.sub2
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    ModuleNotFoundError: No module named 'namespace.sub2'
    
    >>> import importlib
    >>> importlib.invalidate_caches()  # invalidate the cache, not helpful
    >>> namespace.__path__
    _NamespacePath(['.../namespace_path_cache/PATH1/namespace'])
    >>> import namespace.sub2
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    ModuleNotFoundError: No module named 'namespace.sub2'
    
    
    >>> import sys
    >>> sys.path.remove('')  # changing sys.path solves this
    >>> namespace.__path__
    _NamespacePath(['.../namespace_path_cache/PATH1/namespace'])
    >>> import namespace.sub2
    >>> namespace.__path__
    _NamespacePath(['.../namespace_path_cache/PATH1/namespace',
                    '.../namespace_path_cache/PATH2/namespace'])

    importlib.invalidate_caches() documentation says:

    This function should be called if any modules are created/installed while your program is running to guarantee all finders will notice the new module’s existence.

    That makes me think calling importlib.invalidate_caches() should also invalidate the cache of _NamespacePaths.

    (This also affects older Pythons, but since it is a behavior change, I've only marked 3.11).

    @hroncok hroncok mannequin added 3.11 bug and security fixes type-bug An unexpected behavior, bug, or error labels Nov 3, 2021
    @hroncok
    Copy link
    Mannequin Author

    hroncok mannequin commented Nov 3, 2021

    I have a fix in mind, will submit a draft pull request without tests shortly, continue with the tests later.

    @hroncok
    Copy link
    Mannequin Author

    hroncok mannequin commented Nov 3, 2021

    The PR is now ready for review.

    @hroncok hroncok mannequin added stdlib Python modules in the Lib dir labels Nov 3, 2021
    @miss-islington
    Copy link
    Contributor

    New changeset ae1965c by Miro Hrončok in branch 'main':
    bpo-45703: Invalidate _NamespacePath cache on importlib.invalidate_ca… (GH-29384)
    ae1965c

    @encukou
    Copy link
    Member

    encukou commented Nov 23, 2021

    Now, is this a bugfix, or a new feature?
    I lean toward calling it a bugfix and backporting to earlier versions.

    @hroncok
    Copy link
    Mannequin Author

    hroncok mannequin commented Nov 23, 2021

    I consider it a bugfix. I don't expect users to rely on the previous behavior and be surprised by the new, but yet I know https://xkcd.com/1172/ very well.

    @encukou
    Copy link
    Member

    encukou commented Dec 7, 2021

    Sadly, the backport is non-trivial. I'm putting on my TODO list, but I doubt I'll get to it soon :(

    @vstinner
    Copy link
    Member

    The automated backport to Python 3.9 failed. Miro asked me to close the PR:
    #29964

    Can someone backport the fix to 3.9? (create a PR)

    @encukou
    Copy link
    Member

    encukou commented Jan 26, 2022

    Embarassingly, it seems it bust needs regen-importlib, at least for 3.10

    @miss-islington
    Copy link
    Contributor

    New changeset 5c39e47 by Petr Viktorin in branch '3.10':
    [3.10] bpo-45703: Invalidate _NamespacePath cache on importlib.invalidate_cache (GH-29384) (GH-30922)
    5c39e47

    @miss-islington
    Copy link
    Contributor

    New changeset 8d239bf by Petr Viktorin in branch '3.9':
    [3.9] bpo-45703: Invalidate _NamespacePath cache on importlib.invalidate_cache (GH-29384) (GH-30922) (GH-31076)
    8d239bf

    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.11 bug and security fixes stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Development

    No branches or pull requests

    3 participants