New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Regression in pkgutil: iter_modules stopped taking Path argument in python 3.8.10 and 3.9.5 #88227
Comments
The pkgutil.iter_modules crash when using Path object in the first argument. The code below works in python 3.8.9 and 3.9.4, but stopped working in python 3.8.10 and 3.9.5. Changing from Path to str works in all versions. import pkgutil
from pathlib import Path for _, modname, ispkg in pkgutil.iter_modules([Path("/home")], 'somepackage.somesubpackage'): Error message from python 3.8.10 (other path was used): Traceback (most recent call last):
File "/home/devel/Python-3.8.10/Lib/pkgutil.py", line 415, in get_importer
importer = sys.path_importer_cache[path_item]
KeyError: PosixPath('/home/devel/pharmpy/src/pharmpy/plugins')
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "pyissue.py", line 5, in <module>
for _, modname, ispkg in pkgutil.iter_modules([Path("/home/devel/pharmpy/src/pharmpy/plugins")], 'pharmpy.plugins'):
File "/home/devel/Python-3.8.10/Lib/pkgutil.py", line 129, in iter_modules
for i in importers:
File "/home/devel/Python-3.8.10/Lib/pkgutil.py", line 419, in get_importer
importer = path_hook(path_item)
File "<frozen importlib._bootstrap_external>", line 1594, in path_hook_for_FileFinder
File "<frozen importlib._bootstrap_external>", line 1469, in __init__
File "<frozen importlib._bootstrap_external>", line 177, in _path_isabs
AttributeError: 'PosixPath' object has no attribute 'startswith' |
https://bugs.python.org/issue43105 and #25121 caused this regression. |
(obviously we're missing any tests for use of Path objects in this situation) |
Looks like it was always getting lucky in the past, as sys.path requires strings, and the "path" argument here is an alternative to it. The cache was definitely not working as intended. So while it's not clearly documented anywhere (other than the related pkgutil.extend_path() method saying it'll ignore non-str paths), we do need to only be passing str into _importlib_bootstrap. I can't do the pkgutil updates right now, but I think it's just get_importer than needs an os.fsdecode() call around its argument. Maybe some others in the same module. And add tests :) |
To be clear, I'll get to this when I can, but if someone else wants to write the fix I'm happy to review and merge. Don't wait for me to do this one just because I'm assigned. |
It's mentioned that Python 3.9.5 has this regression but the code works fine on my windows machine. Is this only reproducible on POSIX? |
I can reproduce it on latest master running on Linux. steve.dower: I wrote some tests and wrapping get_importer argument with os.fsdecode() fixes it. I'm happy to open an PR, just let me know or if OP is not willing to do so either. |
Interesting. Perhaps my actual change was to cause it to raise an error Arguably we should filter them out instead, then, rather than
Feel free. I assume they'd have arrived with a PR if they were keen to |
Thanks all for looking into this!
Yes, that snippet returns modules with the previous versions of python 3.8 and 3.9. For more background to what I tested please see:
Please go ahead! |
Thanks for the patch! I don't think this meets the 3.8 bar now, but if Łukasz wants it then it should be easy. |
Issue still actual for python 3.10.4 |
I'm on Python 3.8.10 (Ubuntu 20.04) and still see this issue |
Same, seeing this issue on Python 3.8.10 with ubuntu |
Yes, you'll need to get a newer version of 3.8. If Ubuntu doesn't offer one, you'll need to speak to them - we released it already, but we can't insert it into someone else's operating system. |
We can still see this issue in Python 3.10.11 on Red Hat When searching https://docs.python.org/release/3.10.11/whatsnew/changelog.html#changelog there's no mention of bpo-44061 But it seems to be fixed in 3.11 - https://docs.python.org/release/3.11.2/whatsnew/changelog.html#changelog And also in 3.8 and 3.9, but somehow 3.10 was skipped. |
Whoops. It probably came up just as the 3.10 branch was made, and so didn't get explicitly backported to it. I suspect a backport would be okay still, though you could also ask Red Hat to do it. |
That's @pablogsal's call. is a forgotten backport worth an exception?
Red Hat doesn't ship 3.10, it'll go straight to 11 :) |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: