Test failures on GHA macos runners

[dataflake]

The macos GitHub Actions runners reveal several issues. Tests using Python 3.11 on both macos-12 and macos-11 runners as well as tests using Python 3.9 and 3.10 on macos-12 runners fail during the buildout run's dependency installation step. The binary wheel for zope.interface is typically the first such package encountered, it is downloaded and then fails to install with a traceback similar to the one shown at the bottom.

These issues do not show up on any local macOS install with either a self-compiled Python or homebrew-installed Python that will by default compile to the specific machine architecture (arm64 or x86_64). Later Python versions on the GHA runners, however, are compiled as universal2, which combines both architectures in "fat" binaries, using the configure flags --enable-universalsdk --with-universal-archs=universal2.

It seems neither zc.buildout nor setuptools deal with this correctly, and how we use setuptools with setup_requires to build binary packages that depend on other binary packages appears to be an issue as well.

• zc.buildout uses a completely unsuitable simplistic way to pick a package from a list of candidates returned from the PyPI index, see Problem with Buildout on Python 3.10 and macOS >= 11; with reproduction repository buildout/buildout#609 (comment). This algorithm cannot deal with situations where more than a single binary wheel is returned from PyPI, as would be the case when a wheel exists for the specific archtitecture plus a universal2 wheel.
• setuptools appears to misinterpret the macOS version number baked into the wheel name, see Question about pkg_resources.compatible_platforms on macOS pypa/setuptools#3687

Some of this was already discussed in zopefoundation/Zope#1071. I want to take this up again soon and hopefully find some solution.

root: Reading https://pypi.org/simple/zope.interface/
Getting distribution for 'zope.interface==5.5.2'.
root: Downloading https://files.pythonhosted.org/packages/ff/a1/788d02d891a5abdf842eec244b432891f56f5c23cb30504c35f70b588b85/zope.interface-5.5.2-cp39-cp39-macosx_10_9_x86_64.whl#sha256=40f4065745e2c2fa0dff0e7ccd7c166a8ac9748974f960cd39f63d2c19f9231f
While:
  Installing.
  Getting section alltests.
  Initializing section alltests.
  Installing recipe zc.recipe.testrunner.
  Getting distribution for 'zope.interface==5.5.2'.

An internal error occurred due to a bug in either zc.buildout or in a
recipe being used:
Traceback (most recent call last):
  File "/Users/runner/work/Zope/Zope/.tox/py39/lib/python3.9/site-packages/zc/buildout/buildout.py", line 2252, in main
    getattr(buildout, command)(args)
  File "/Users/runner/work/Zope/Zope/.tox/py39/lib/python3.9/site-packages/zc/buildout/buildout.py", line 740, in install
    [self[part]['recipe'] for part in install_parts]
  File "/Users/runner/work/Zope/Zope/.tox/py39/lib/python3.9/site-packages/zc/buildout/buildout.py", line 740, in <listcomp>
    [self[part]['recipe'] for part in install_parts]
  File "/Users/runner/work/Zope/Zope/.tox/py39/lib/python3.9/site-packages/zc/buildout/buildout.py", line 1373, in __getitem__
    options._initialize()
  File "/Users/runner/work/Zope/Zope/.tox/py39/lib/python3.9/site-packages/zc/buildout/buildout.py", line 1481, in _initialize
    self.initialize()
  File "/Users/runner/work/Zope/Zope/.tox/py39/lib/python3.9/site-packages/zc/buildout/buildout.py", line 1487, in initialize
    recipe_class = _install_and_load(reqs, 'zc.buildout', entry, buildout)
  File "/Users/runner/work/Zope/Zope/.tox/py39/lib/python3.9/site-packages/zc/buildout/buildout.py", line 1430, in _install_and_load
    zc.buildout.easy_install.install(
  File "/Users/runner/work/Zope/Zope/.tox/py39/lib/python3.9/site-packages/zc/buildout/easy_install.py", line 971, in install
    return installer.install(specs, working_set)
  File "/Users/runner/work/Zope/Zope/.tox/py39/lib/python3.9/site-packages/zc/buildout/easy_install.py", line 744, in install
    for dist in self._get_dist(req, ws):
  File "/Users/runner/work/Zope/Zope/.tox/py39/lib/python3.9/site-packages/zc/buildout/easy_install.py", line 581, in _get_dist
    dists = [_move_to_eggs_dir_and_compile(dist, self._dest)]
  File "/Users/runner/work/Zope/Zope/.tox/py39/lib/python3.9/site-packages/zc/buildout/easy_install.py", line 1948, in _move_to_eggs_dir_and_compile
    assert newdist is not None  # newloc above is missing our dist?!
AssertionError
py39: exit 1 (11.14 seconds) /Users/runner/work/Zope/Zope> .tox/py39/bin/buildout -c /Users/runner/work/Zope/Zope/buildout.cfg buildout:directory=/Users/runner/work/Zope/Zope/.tox/py39 buildout:develop=/Users/runner/work/Zope/Zope install alltests pid=2506
  py39: FAIL code 1 (17.05=setup[5.90]+cmd[11.14] seconds)

[dataflake]

Apparently the upgrade to macOS Sonoma broke zc.buildout-based installs again. Installing Zope via pip works fine, which means the pre-built wheels are OK. The error happens when buildout/setuptools converts the wheels to eggs. Here's output while installing Zope as part of a tox run. The software versions involved are the latest as of today:

$ bin/tox -epy311
...
root: Reading https://pypi.org/simple/zope.interface/
Getting distribution for 'zope.interface==6.1'.
root: Downloading https://files.pythonhosted.org/packages/3e/1f/43557bb2b6e8537002a5a26af9b899171e26ddfcdf17a00ff729b00c036b/zope.interface-6.1-cp311-cp311-macosx_11_0_arm64.whl#sha256=34c15ca9248f2e095ef2e93af2d633358c5f048c49fbfddf5fdfc47d5e263736
While:
  Installing.
  Getting section test.
  Initializing section test.
  Installing recipe zc.recipe.testrunner.
  Getting distribution for 'zope.interface==6.1'.

An internal error occurred due to a bug in either zc.buildout or in a
recipe being used:
Traceback (most recent call last):
  File "/Users/jens/src/zope/Zope/.tox/py311/lib/python3.11/site-packages/zc/buildout/buildout.py", line 2252, in main
    getattr(buildout, command)(args)
  File "/Users/jens/src/zope/Zope/.tox/py311/lib/python3.11/site-packages/zc/buildout/buildout.py", line 740, in install
    [self[part]['recipe'] for part in install_parts]
  File "/Users/jens/src/zope/Zope/.tox/py311/lib/python3.11/site-packages/zc/buildout/buildout.py", line 740, in <listcomp>
    [self[part]['recipe'] for part in install_parts]
     ~~~~^^^^^^
  File "/Users/jens/src/zope/Zope/.tox/py311/lib/python3.11/site-packages/zc/buildout/buildout.py", line 1373, in __getitem__
    options._initialize()
  File "/Users/jens/src/zope/Zope/.tox/py311/lib/python3.11/site-packages/zc/buildout/buildout.py", line 1481, in _initialize
    self.initialize()
  File "/Users/jens/src/zope/Zope/.tox/py311/lib/python3.11/site-packages/zc/buildout/buildout.py", line 1487, in initialize
    recipe_class = _install_and_load(reqs, 'zc.buildout', entry, buildout)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jens/src/zope/Zope/.tox/py311/lib/python3.11/site-packages/zc/buildout/buildout.py", line 1430, in _install_and_load
    zc.buildout.easy_install.install(
  File "/Users/jens/src/zope/Zope/.tox/py311/lib/python3.11/site-packages/zc/buildout/easy_install.py", line 971, in install
    return installer.install(specs, working_set)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jens/src/zope/Zope/.tox/py311/lib/python3.11/site-packages/zc/buildout/easy_install.py", line 744, in install
    for dist in self._get_dist(req, ws):
                ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jens/src/zope/Zope/.tox/py311/lib/python3.11/site-packages/zc/buildout/easy_install.py", line 581, in _get_dist
    dists = [_move_to_eggs_dir_and_compile(dist, self._dest)]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jens/src/zope/Zope/.tox/py311/lib/python3.11/site-packages/zc/buildout/easy_install.py", line 1922, in _move_to_eggs_dir_and_compile
    os.rename(tmp_loc, newloc)
OSError: [Errno 66] Directory not empty: '/Users/jens/src/.eggs/tmpn48nz546/zope.interface-6.1-py3.11-macosx-13.3-arm64.egg' -> '/Users/jens/src/.eggs/zope.interface-6.1-py3.11-macosx-13.3-arm64.egg'
py311: exit 1 (2.26 seconds) /Users/jens/src/zope/Zope> .tox/py311/bin/buildout -c /Users/jens/src/zope/Zope/buildout.cfg buildout:directory=/Users/jens/src/zope/Zope/.tox/py311 buildout:develop=/Users/jens/src/zope/Zope buildout:root-directory=/Users/jens/src/zope/Zope pid=13139
  py311: FAIL code 1 (3.37=setup[1.12]+cmd[2.26] seconds)
  evaluation failed :( (3.40 seconds)

[dataflake]

Followup to my previous comment and probably of interest to @icemac:

The culprit is how broken buildout/setuptools/distutils/pkg_resources is when it comes to naming binary eggs. The name contains information about the system platform, like macosx-13.3-arm64. This information can be read out in real time from the OS, or it can be found with sysconfig.get_platform_name(). Problem: sysconfig.get_platform_name() uses the value that was true when the interpreter was compiled. So an interpreter compiled on Ventura will show macosx-13.3-arm64 and Sonoma macosx-14.0-arm64.

These live vs. cached lookups are not done consistently. This will cause problems when the OS is upgraded. During the conversion from wheel to egg buildout will believe there's already an egg at zope.interface-6.1-py3.11-macosx-13.3-arm64.egg, but when pkg_resources is used to find a distribution there it won't find any - because it looks for macosx-14.0-arm64 and not macosx-13.3-arm64. So now buildout re-converts the wheel again - and names it zope.interface-6.1-py3.11-macosx-13.3-arm64.egg instead of zope.interface-6.1-py3.11-macosx-14.0-arm64.egg, which will break because that egg is already there.

Workaround: Major macOS upgrades require re-building all self-compiled Python interpreters to fix the mismatch of cached versus real system information.