"bad magic number" when Python 2's pyc file exists without py file

[mitchhentges]

BPO	43969
Nosy	@terryjreedy, @mitchhentges
PRs	bpo-43969: Don't use SourcelessFileLoader by default #25731
Files	repro.sh: Script to reproduce the issue

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-11-04.20:38:51.379>
created_at = <Date 2021-04-28.20:36:52.915>
labels = ['interpreter-core', 'type-feature', '3.11']
title = '"bad magic number" when Python 2\'s pyc file exists without py file'
updated_at = <Date 2021-11-04.20:38:51.379>
user = 'https://github.com/mitchhentges'

bugs.python.org fields:

activity = <Date 2021-11-04.20:38:51.379>
actor = 'mitchhentges'
assignee = 'none'
closed = True
closed_date = <Date 2021-11-04.20:38:51.379>
closer = 'mitchhentges'
components = ['Interpreter Core']
creation = <Date 2021-04-28.20:36:52.915>
creator = 'mitchhentges'
dependencies = []
files = ['49997']
hgrepos = []
issue_num = 43969
keywords = ['patch']
message_count = 4.0
messages = ['392254', '392347', '392521', '405752']
nosy_count = 2.0
nosy_names = ['terry.reedy', 'mitchhentges']
pr_nums = ['25731']
priority = 'normal'
resolution = None
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue43969'
versions = ['Python 3.11']

[mitchhentges]

Python 3 imports may fail if a Python 2 .pyc file exists without an accompanying .py file.

-----

My project vendors code, and we recently updated "requests" to a drastically newer version.

As part of this version change, requests/packages/__init__.py was removed, and requests/packages.py was added.

This project is long-lived, and many people have imported requests with Python 2. So, requests/packages/__init__.pyc exists.

After making this update, importing requests with Python 3 fails:
ImportError: bad magic number in 'parent.child': b'\x03\xf3\r\n'

Interestingly, deleting requests/packages/__init__.pyc allows Python 3 to work again.

-----

I've attached a "reproduction" script that produces the directory structure and tweak required to cause the failure.
I'm running Python 2.7.18 and Python 3.9.1.

[mitchhentges]

Looking closer into this, it appears that Python 3 will still try to load non-pycache .pyc files. Perhaps I'm missing context, but aren't those only supposed by to written and read by Python 2?

Location where pyc loader is registered: https://github.com/python/cpython/blob/v3.9.4/Lib/importlib/_bootstrap_external.py#L1571-L1572
Location where __pycache__ loading happens is part of SourceFileLoader here: https://github.com/python/cpython/blob/v3.9.4/Lib/importlib/_bootstrap_external.py#L870

I'll attach a patch to remove this pyc-loading for convenience, but I don't have enough context to be confident that it's the correct solution.

[terryjreedy]

Unless you find documentation that says otherwise, this an enhancement proposal for 3.11.

If no core developer (other than me) jumps on this, you might use git blame to fine out who committed code in the area you proposed changing.

[mitchhentges]

I'm going to close this here - I currently don't have the cycles to push this forward. Besides, this issue only occurs in environments that are using both Python 2 and 3, and as Python 2's EOL date gets further in the past, the severity of this ticket decreases.

For future readers, if you need a workaround for this exact problem, it was solved by my team with this import hook: https://hg.mozilla.org/mozilla-central/file/50e4a4e6975a7f828277a2b9c3cc9770a6c94107/build/mach_initialize.py#l630