ctypes: test_frozentable fails when make regen-frozen

[hroncok]

BPO	43372
Nosy	@nascheme, @vstinner, @hroncok, @hrnciar
PRs	bpo-43372: Re-generate frozen code for __hello__. #24708 bpo-43372: Use BUILDPYTHON for regen-frozen. #24714 bpo-43372: Use _freeze_importlib for regen-frozen. #24759

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/nascheme'
closed_at = <Date 2021-03-06.21:34:34.686>
created_at = <Date 2021-03-02.14:40:34.110>
labels = ['ctypes', 'build', 'tests', '3.10']
title = 'ctypes: test_frozentable fails when make regen-frozen'
updated_at = <Date 2021-03-09.10:41:57.922>
user = 'https://github.com/hroncok'

bugs.python.org fields:

activity = <Date 2021-03-09.10:41:57.922>
actor = 'vstinner'
assignee = 'nascheme'
closed = True
closed_date = <Date 2021-03-06.21:34:34.686>
closer = 'nascheme'
components = ['Tests', 'ctypes']
creation = <Date 2021-03-02.14:40:34.110>
creator = 'hroncok'
dependencies = []
files = []
hgrepos = []
issue_num = 43372
keywords = ['patch']
message_count = 6.0
messages = ['387933', '387936', '387956', '388216', '388229', '388347']
nosy_count = 4.0
nosy_names = ['nascheme', 'vstinner', 'hroncok', 'hrnciar']
pr_nums = ['24708', '24714', '24759']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'compile error'
url = 'https://bugs.python.org/issue43372'
versions = ['Python 3.10']

[hroncok]

The following test failure happens on Python 3.10.0a6+ when we make regen-frozen with the same Python version we test:

======================================================================
FAIL: test_frozentable (ctypes.test.test_values.PythonValuesTestCase)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/churchyard/Dokumenty/RedHat/cpython/Lib/ctypes/test/test_values.py", line 87, in test_frozentable
    self.assertEqual(items, expected, "PyImport_FrozenModules example "
AssertionError: Lists differ: [('__hello__', 129), ('__phello__', -129), ('__phello__.spam', 129)] != [('__hello__', 125), ('__phello__', -125), ('__phello__.spam', 125)]

First differing element 0:
('__hello__', 129)
('__hello__', 125)

• [('__hello__', 129), ('__phello__', -129), ('__phello__.spam', 129)]
  ? ^ ^ ^

+ [('__hello__', 125), ('__phello__', -125), ('__phello__.spam', 125)]
? ^ ^ ^
: PyImport_FrozenModules example in Doc/library/ctypes.rst may be out of date

----------------------------------------------------------------------
Ran 494 tests in 0.466s

FAILED (failures=1, skipped=87)

Reproducer:

1. Build Python from source: $ ./configure && make -j...
2. Run ctypes tests: $ ./python -m ctypes.test
3. Regenerate frozen: $ PYTHON_FOR_REGEN=./python make regen-frozen
4. Build Python from source again: $ ./configure && make -j...
5. Run ctypes tests: $ ./python -m ctypes.test

Actual result:

Tests in (2) pass, tests in (5) fail.

The difference after (3) is:

diff --git a/Python/frozen_hello.h b/Python/frozen_hello.h
index 9c566cc81e..d58b726aa8 100644
--- a/Python/frozen_hello.h
+++ b/Python/frozen_hello.h
@@ -9,5 +9,5 @@ static unsigned char M___hello__[] = {
     100,218,5,112,114,105,110,116,169,0,114,2,0,
     0,0,114,2,0,0,0,218,4,110,111,110,101,
     218,8,60,109,111,100,117,108,101,62,1,0,0,
-    0,115,2,0,0,0,4,1,
+    0,115,6,0,0,0,4,0,12,1,255,128,
 };

Expected results:

Tests pass, no diff.

[hroncok]

When I run PYTHON_FOR_REGEN=python3.10 make regen-frozen with Python 3.10.0a5, I get the same diff and the same problem.

When I run PYTHON_FOR_REGEN=python3.9 make regen-frozen with Python 3.9.2, I get no diff and no problem (similarly with Python 3.8.8).

[nascheme]

I believe the line table format got changed but the frozen code didn't get re-generated. If you try to call co_lines() on the __hello__ code, Python crashes.

>>> import __hello__
Hello world!
>>> co = __hello__.__spec__.loader.get_code('__hello__')
>>> co.co_linetable
b'\x04\x01'
>>> list(co.co_lines())
python: ../Objects/codeobject.c:1185: PyLineTable_NextAddressRange: Assertion `!at_end(range)' failed.

My PR re-generates the code and fixes the test. Perhaps I should also add a test to exercise co_lines() on the frozen code object.

[nascheme]

New changeset 87ec26b by Neil Schemenauer in branch 'master':
bpo-43372: Use _freeze_importlib for regen-frozen. (GH-24759)
87ec26b