zipimport raises NameError: name '_boostrap_external' is not defined

[misho88]

BPO	39033
Nosy	@Yhg1s, @ncoghlan, @encukou, @zware, @serhiy-storchaka, @animalize, @miss-islington, @tirkarthi, @misho88
PRs	bpo-39033: Fix NameError in zipimport during hash validation #17588 [3.8] bpo-39033: Fix NameError in zipimport during hash validation (GH-17588) #17642

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-01-14.11:41:34.654>
created_at = <Date 2019-12-12.22:20:17.746>
labels = ['3.8', 'library', '3.9', 'type-crash']
title = "zipimport raises NameError: name '_boostrap_external' is not defined"
updated_at = <Date 2020-01-14.11:41:34.650>
user = 'https://github.com/misho88'

bugs.python.org fields:

activity = <Date 2020-01-14.11:41:34.650>
actor = 'petr.viktorin'
assignee = 'none'
closed = True
closed_date = <Date 2020-01-14.11:41:34.654>
closer = 'petr.viktorin'
components = ['Library (Lib)']
creation = <Date 2019-12-12.22:20:17.746>
creator = 'misho88'
dependencies = []
files = []
hgrepos = []
issue_num = 39033
keywords = ['patch']
message_count = 11.0
messages = ['358310', '358311', '358316', '358317', '358318', '358319', '358321', '358326', '358459', '359964', '359965']
nosy_count = 9.0
nosy_names = ['twouters', 'ncoghlan', 'petr.viktorin', 'zach.ware', 'serhiy.storchaka', 'malin', 'miss-islington', 'xtreak', 'misho88']
pr_nums = ['17588', '17642']
priority = 'normal'
resolution = None
stage = 'resolved'
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue39033'
versions = ['Python 3.8', 'Python 3.9']

[misho88]

I think there's a "t" missing:

Lib/zipimport.py
609-
610- try:
611: _boostrap_external._validate_hash_pyc(
612- data, source_hash, fullname, exc_details)
613- except ImportError:
614- return None
615- else:
616- source_mtime, source_size = \
617- _get_mtime_and_size_of_source(self, fullpath)
618-
619- if source_mtime:

[zware]

Good catch! Would you like to submit a pull request to fix it? Ideally such a PR should also include a test to exercise this code.

[animalize]

Is it possible to scan stdlib to find similar bugs?

[tirkarthi]

A possible test case to trigger this code path based on testUncheckedHashBasedPyc would be as below. check_hash_based_pycs can be patched to be "always" so that the hash is validated using _bootstrap_external._validate_hash_pyc. The difference between "state = old" for pyc and "state = new" in py would ensure the hashes are different to raise ImportError expected and the test picks up state = new as the updated source code. Without fixing the typo trying to make hash validation would throw NameError.

@unittest.mock.patch('_imp.check_hash_based_pycs', 'always')
def test_checked_hash_based_change_pyc(self):
source = b"state = 'old'"
source_hash = importlib.util.source_hash(source)
bytecode = importlib._bootstrap_external._code_to_hash_pyc(
compile(source, "???", "exec"),
source_hash,
False,
)
files = {TESTMOD + ".py": (NOW, "state = 'new'"),
TESTMOD + ".pyc": (NOW - 20, bytecode)}
def check(mod):
self.assertEqual(mod.state, 'new')
self.doTest(None, files, TESTMOD, call=check)

[tirkarthi]

Also since the fix involves modifying zipimport.py it would also require running "make regen-importlib && make -s" for changes to ensure the updated regen code is used.

[tirkarthi]

Ma Lin, Running pylint/flake8 could possibly detect it but needs to be manually checked. bpo-36948 was a similar report I filed in the past using flake8 regarding NameError.

pylint - Lib/zipimport.py:611:20: E0602: Undefined variable '_boostrap_external' (undefined-variable)
flake8 - Lib/zipimport.py:611:21: F821 undefined name '_boostrap_external'

[serhiy-storchaka]

Wince you has virtually written a patch, do you mind to create a PR Karthikeyan?

[tirkarthi]

Sure Serhiy, I will create one.

[ncoghlan]

New changeset 79f02fe by Nick Coghlan (Xtreak) in branch 'master':
bpo-39033: Fix NameError in zipimport during hash validation (GH-17588)
79f02fe

[miss-islington]

New changeset 9955f33 by Miss Islington (bot) (Karthikeyan Singaravelan) in branch '3.8':
[3.8] bpo-39033: Fix NameError in zipimport during hash validation (GH-17588) (GH-17642)
9955f33

[encukou]

Thank you, Mihail and Karthikeyan!