-
-
Notifications
You must be signed in to change notification settings - Fork 31.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
importlib.invalidate_caches() does not invalidate _NamespacePath's _last_parent_path-based cache #89866
Comments
Recently, when debugging a weird problem (see https://bugzilla.redhat.com/show_bug.cgi?id=2018551 for details if interested, but not important for this issue), I've realized that the _NamespacePath class (from importlib/_bootstrap_external.py) has a cache that (in some cases) uses tuple(sys.path) as the key. I was expecting importlib.invalidate_caches() to invalidate this cache, but it doesn't. Consider the following directory structure: . Here is a helper to create it (on Linux-ish): $ mkdir -p PATH1/namespace/sub1
$ mkdir -p PATH2/namespace/sub2
$ touch PATH1/namespace/sub1/__init__.py
$ touch PATH2/namespace/sub2/__init__.py Run Python with PYTHONPATH=PATH1:PATH2 (output slightly formatted for readability): $ PYTHONPATH=PATH1:PATH2 python3.11
>>> import namespace
>>> namespace.__path__
_NamespacePath(['.../namespace_path_cache/PATH1/namespace',
'.../namespace_path_cache/PATH2/namespace'])
>>> import namespace.sub1 # works
>>> import namespace.sub2 # works
>>> exit() The namespace packages seem to work as expected. Now move PATH2/namespace out of the way: $ mv PATH2/namespace PATH2/cant-import-this Run Python again: $ PYTHONPATH=PATH1:PATH2 python3.11
>>> import namespace
>>> namespace.__path__
_NamespacePath(['.../namespace_path_cache/PATH1/namespace'])
>>> ... While this interpreter still runs, move the PATH2/namespace module back in: $ mv PATH2/cant-import-this PATH2/namespace >>> ...
>>> namespace.__path__
_NamespacePath(['.../namespace_path_cache/PATH1/namespace'])
>>> import namespace.sub2
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'namespace.sub2'
>>> import importlib
>>> importlib.invalidate_caches() # invalidate the cache, not helpful
>>> namespace.__path__
_NamespacePath(['.../namespace_path_cache/PATH1/namespace'])
>>> import namespace.sub2
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'namespace.sub2'
>>> import sys
>>> sys.path.remove('') # changing sys.path solves this
>>> namespace.__path__
_NamespacePath(['.../namespace_path_cache/PATH1/namespace'])
>>> import namespace.sub2
>>> namespace.__path__
_NamespacePath(['.../namespace_path_cache/PATH1/namespace',
'.../namespace_path_cache/PATH2/namespace']) importlib.invalidate_caches() documentation says:
That makes me think calling importlib.invalidate_caches() should also invalidate the cache of _NamespacePaths. (This also affects older Pythons, but since it is a behavior change, I've only marked 3.11). |
I have a fix in mind, will submit a draft pull request without tests shortly, continue with the tests later. |
The PR is now ready for review. |
Now, is this a bugfix, or a new feature? |
I consider it a bugfix. I don't expect users to rely on the previous behavior and be surprised by the new, but yet I know https://xkcd.com/1172/ very well. |
Sadly, the backport is non-trivial. I'm putting on my TODO list, but I doubt I'll get to it soon :( |
The automated backport to Python 3.9 failed. Miro asked me to close the PR: Can someone backport the fix to 3.9? (create a PR) |
Embarassingly, it seems it bust needs regen-importlib, at least for 3.10 |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: