ERROR: test_sundry (test.test_tools.test_sundry.TestSundryScripts): SystemExit: ERROR: missing _freeze_module

[hroncok]

BPO	45402
Nosy	@vstinner, @hroncok, @pablogsal
PRs	bpo-45402: Fix test_tools.test_sundry() #28786

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-10-07.19:19:53.416>
created_at = <Date 2021-10-07.10:05:28.827>
labels = ['tests', '3.11']
title = 'ERROR: test_sundry (test.test_tools.test_sundry.TestSundryScripts): SystemExit: ERROR: missing _freeze_module'
updated_at = <Date 2021-11-22.12:36:40.928>
user = 'https://github.com/hroncok'

bugs.python.org fields:

activity = <Date 2021-11-22.12:36:40.928>
actor = 'hroncok'
assignee = 'none'
closed = True
closed_date = <Date 2021-10-07.19:19:53.416>
closer = 'vstinner'
components = ['Tests']
creation = <Date 2021-10-07.10:05:28.827>
creator = 'hroncok'
dependencies = []
files = []
hgrepos = []
issue_num = 45402
keywords = ['patch']
message_count = 7.0
messages = ['403377', '403382', '403384', '403392', '403435', '403436', '406767']
nosy_count = 3.0
nosy_names = ['vstinner', 'hroncok', 'pablogsal']
pr_nums = ['28786']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue45402'
versions = ['Python 3.11']

[hroncok]

When we build Python 3.10.0a1 (from the git tag) in Fedora, we see:

======================================================================
ERROR: test_sundry (test.test_tools.test_sundry.TestSundryScripts)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/builddir/build/BUILD/Python-3.11.0a1/Lib/test/test_tools/test_sundry.py", line 43, in test_sundry
    import_tool(name)
    ^^^^^^^^^^^^^^^^^
  File "/builddir/build/BUILD/Python-3.11.0a1/Lib/test/test_tools/__init__.py", line 35, in import_tool
    return importlib.import_module(toolname)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/builddir/build/BUILD/Python-3.11.0a1/Lib/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1072, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1044, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1015, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 689, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 894, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/builddir/build/BUILD/Python-3.11.0a1/Tools/scripts/freeze_modules.py", line 37, in <module>
    sys.exit("ERROR: missing _freeze_module")
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
SystemExit: ERROR: missing _freeze_module

---

Ran 3 tests in 0.209s

I see in the build log we use Programs/_freeze_module regularly in the build process, so no idea why this fails. I'll try to come up with a reporducer.

[hroncok]

I can reproduce this from git main branch:

$ git clean -fdx
$ mkdir -p build/debug
$ cd build/debug
$ ../../configure --with-platlibdir=lib64 --enable-shared --without-ensurepip --with-pydebug
$ make
$ cd ../..
$ LD_LIBRARY_PATH=$PWD/build/debug build/debug/python -m test -v test_tools

[hroncok]

I can also reproduce this without the additional configure flags:

[cpython]$ git clean -fdx
[cpython]$ mkdir -p build/debug
[cpython]$ cd build/debug
[debug]$ ../../configure --with-pydebug
[debug]$ make
[debug]$ cd ../..
[cpython]$ build/debug/python -m test -v test_tools
...
SystemExit: ERROR: missing _freeze_module

As well as from the build dir:

[cpython]$ cd build/debug
[debug]$ ./python -m test -v test_tools
...
SystemExit: ERROR: missing _freeze_module

But when I build from the source directory directly:

[cpython]$ git clean -fdx
[cpython]$ ./configure --with-pydebug
[cpython]$ make
[cpython]$ ./python -m test -v test_tools
...
Tests result: SUCCESS

Hence, I assume the culprit is in building from a subdirectory.

[vstinner]

My fix for bpo-45020 no longer works, I'm not sure why:

commit 41551ee
Author: Victor Stinner <vstinner@python.org>
Date: Fri Sep 17 20:20:43 2021 +0200

bpo-45020: Fix build out of source tree (GH-28410)

* Makefile.pre.in: Add $(srcdir) when needed, remove it when it was
  used by mistake.
* freeze_modules.py tool uses ./Programs/_freeze_module if the
  executable doesn't exist in the source tree.

I'm working on a new fix.

[vstinner]

New changeset ff8859d by Victor Stinner in branch 'main':
bpo-45402: Fix test_tools.test_sundry() (GH-28786)
ff8859d

[vstinner]

Thanks Miro for the bug report, it's now fixed.

[hroncok]

There seem to be a regression in this fix, see https://bugs.python.org/issue45866