Is Python 3.12 different?

[ndw]

Description

I'm running cibuildwheel in a GitHub action. This particular example is for Mac/aarch64, I haven't tried the other platforms yet. I upgraded to cibuildwheel version 2.16.2 (we used 2.11.1 last time I ran this build). I was upgrading for Python 3.12 support, mostly. The build continues to work fine for 3.8, 3.9, 3.10, and 3.11, but falls over for 3.12:

+ python -m pip wheel /Users/runner/work/BuildWheels/BuildWheels/mac/eec_pypi --wheel-dir=/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/cibw-run-qx2gpkbn/cp312-macosx_arm64/built_wheel --no-deps
  Processing /Users/runner/work/BuildWheels/BuildWheels/mac/eec_pypi
    Installing build dependencies: started
    Installing build dependencies: finished with status 'done'
    Getting requirements to build wheel: started
    Getting requirements to build wheel: finished with status 'error'
    error: subprocess-exited-with-error
    
    × Getting requirements to build wheel did not run successfully.
    │ exit code: 1
    ╰─> [33 lines of output]
        Traceback (most recent call last):
          File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/cibw-run-qx2gpkbn/cp312-macosx_arm64/build/venv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
            main()
          File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/cibw-run-qx2gpkbn/cp312-macosx_arm64/build/venv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
            json_out['return_val'] = hook(**hook_input['kwargs'])
                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
          File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/cibw-run-qx2gpkbn/cp312-macosx_arm64/build/venv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 112, in get_requires_for_build_wheel
            backend = _build_backend()
                      ^^^^^^^^^^^^^^^^
          File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/cibw-run-qx2gpkbn/cp312-macosx_arm64/build/venv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 77, in _build_backend
            obj = import_module(mod_path)
                  ^^^^^^^^^^^^^^^^^^^^^^^
          File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module
            return _bootstrap._gcd_import(name[level:], package, level)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
          File "<frozen importlib._bootstrap>", line 1381, in _gcd_import
          File "<frozen importlib._bootstrap>", line 1354, in _find_and_load
          File "<frozen importlib._bootstrap>", line 1304, in _find_and_load_unlocked
          File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
          File "<frozen importlib._bootstrap>", line 1381, in _gcd_import
          File "<frozen importlib._bootstrap>", line 1354, in _find_and_load
          File "<frozen importlib._bootstrap>", line 1325, in _find_and_load_unlocked
          File "<frozen importlib._bootstrap>", line 929, in _load_unlocked
          File "<frozen importlib._bootstrap_external>", line 994, in exec_module
          File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
          File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-build-env-zibmg4ie/overlay/lib/python3.12/site-packages/setuptools/__init__.py", line 16, in <module>
            import setuptools.version
          File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-build-env-zibmg4ie/overlay/lib/python3.12/site-packages/setuptools/version.py", line 1, in <module>
            import pkg_resources
          File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-build-env-zibmg4ie/overlay/lib/python3.12/site-packages/pkg_resources/__init__.py", line 2172, in <module>
            register_finder(pkgutil.ImpImporter, find_on_path)
                            ^^^^^^^^^^^^^^^^^^^
        AttributeError: module 'pkgutil' has no attribute 'ImpImporter'. Did you mean: 'zipimporter'?
        [end of output]

If I run the bdist_wheel target from setup.py locally, I get some warnings about how that's deprecated, but it does succeed.

There's a footnote on the cibuildwheel page that reads:

⁵ CPython 3.12 is built by default using Python RCs, starting with cibuildwheel 2.15.

And I wonder if that's relevant. But a bit of we searching didn't turn up anything obvious.

Am I doing something wrong?

Build log

No response

CI config

https://github.com/Saxonica/BuildWheels/actions/runs/6825329700/job/18562948765

[Czaki]

Python 3.12 has removed deprecated API from importlib module here: python/cpython#98040

As You do not provide source code we cannot check where is a problem. The latest setuptools is ready on python 3.12

[ndw]

Thanks. I appreciate that lack of source code makes this more difficult to debug. I updated the requirements.txt file to rely on the most recent setuptools. I don't see any references to, for example, pkgutil in our code base so I'm still quite confused.

[Czaki]

Did you have pyproject.toml file with build-system.requires section? Maybe you pin setuptools there?

[henryiii]

Also, older versions of pretty much everything might trigger this. NumPy, for example, must be 1.26.0+ or you'll see this sort of problem. If you are pinning or limiting at all, most likely in pyproject.toml or in requirements.txt if you install from there before building (and avoid build isolation). I'd recommend fully regenerating requirements.txt from requirements.in on Python 3.12 if that's the case.

[ndw]

Thank you. It was a bad version in pyproject.toml. blush.