Problem with Buildout on Python 3.10 and macOS >= 11; with reproduction repository

[amotl]

Dear @goschtl, @mauritsvanrees, @gotcha, and the whole Buildout community,

first things first: Thanks a stack for conceiving and maintaining this excellent package.

TLDR;

There is a minimal reproduction case repository at test-buildout-python310, where crate-workbench/test-buildout-python310#1 swiftly outlines the problem.

Details

While working on crate/crate-python#428, we ran into a similar problem you have been discussing at #606. We are also installing SQLAlchemy within a Buildout environment and it croaks specifically on Python 3.10 with macOS >= 11. Both Buildout 2 and Buildout 3 show the same symptom.

First, we thought it would be a problem with GHA, but then a colleague also confirmed the flaw happens on his workstation running macOS 12. On our research about this issue, we also discovered pypa/setuptools#2514 by @idgserpro, where similar symptoms showed up.

In order to assist you with further research on this matter, we wrapped all up into a repository with a minimal reproduction case at test-buildout-python310, where the CI/GHA jobs will precisely highlight the build matrix items where the front falls off. crate-workbench/test-buildout-python310#1 has more details about its outcome.

With kind regards,
Andreas.

/cc @proddata

Screenshot

TLDR;

No eggs found in /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/easy_install-uvggdcg2/zope.interface-5.2.0/egg-dist-tmp-s7mlm8qo (setup script problem?)

  File "/Users/runner/hostedtoolcache/Python/3.10.5/x64/lib/python3.10/site-packages/zc/buildout/easy_install.py", line 1745, in _move_to_eggs_dir_and_compile
    [tmp_loc] = glob.glob(os.path.join(tmp_dest, '*'))
ValueError: not enough values to unpack (expected 1, got 0)

Getting distribution for 'greenlet!=0.4.17'.
[...]
gcc: error: sh -c '/Applications/Xcode_13.2.1.app/Contents/Developer/usr/bin/xcodebuild -sdk /Applications/Xcode_13.2.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk -find gcc 2> /dev/null' failed with exit code 256: (null) (errno=Invalid argument)
xcode-select: Failed to locate 'gcc', requesting installation of command line developer tools.
error: Setup script exited with error: command '/usr/bin/gcc' failed with exit code 72

[amotl]

For me, the observation shared above raises those questions:

1. Why does the procedure try to invoke the compiler at all and not pull binary wheels from PyPI instead? They should be available. Here: greenlet, which is a dependency of SQLAlchemy.
2. Why does the procedure not see the gcc compiler when it is falling back to it?
3. Why does it only happen with Python 3.10?

[gotcha]

Thanks for the kind words.

Are you having those problems on a M1 ? Or classic Intel CPU ?

[amotl]

Dear @gotcha,

thanks for your quick reply. CI/GHA on behalf of test-buildout-python310 is running on classic Intel. @proddata also might want to tell us about what his workstation is running on.

On my workstation, also Intel, it is not a problem because I am still on macOS 10.15.7 (Catalina), which also works on GHA. The flaw is only present on newer versions of macOS >= 11.

With kind regards,
Andreas.

[icemac]

Hi, since a week I also see failing GHA runs for Python 3.10 when C code is involved. It is only 3.10, older and newer versions are okay and only MacOS. There is not even zc.buildout involved. (Locally on Intel this cannot be reproduced.)

See zopefoundation/zope.security#87 for details even if they might have nothing to do with this possible issue in buildout and show another traceback.

[proddata]

Dear @gotcha,

thanks for your quick reply. CI/GHA on behalf of [test-buildout-python310] is running on classic Intel. @proddata also might want to tell us about what his workstation is running on.

Only tested it with an M1 on macos12

[amotl]

Hi again,

thank you for sharing your observations, @icemac. I was a bit hesitant first, but now I've also shared them with the people at GHA, see actions/runner-images#5791.

With kind regards,
Andreas.

[idgserpro]

Did you try to use the patch as mentioned in pypa/setuptools#2514 (comment) only to see if it works and maybe we can narrow the problem? Or running a minimal buildout as specified in pypa/setuptools#2514 (comment)? This may be a setuptools issue.

[amotl]

Dear @idgserpro,

running a minimal buildout

A reproduction repository containing a minimal buildout has been created at test-buildout-python310. It can by used by anyone to investigate this issue.

Did you try to use the patch as mentioned in pypa/setuptools#2514 (comment)?

I haven't had the chance to do so. I will try and report back, thank you again!

With kind regards,
Andreas.

[icemac]

@idgserpro Thank you for the link to pypa/setuptools#2514. The comment from @goschtl there helped me in another project where installing of packages containing C code suddenly stopped in my local MacOS (Python 2.7 on zc.buildout 2.13).

[icemac]

Hey, setting _PYTHON_HOST_PLATFORM="macosx-11-x86_64" also helped to fix the GHA build in zopefoundation/zope.security#87. (At least partially.) – That's interesting.

[amotl]

Hi again,

regarding my question at #609 (comment):

For me, the observation shared above raises those questions:

1. Why does the procedure try to invoke the compiler at all and not pull binary wheels from PyPI instead? They should be available. Here: greenlet, which is a dependency of SQLAlchemy.

I just discovered #544 which I believe shows the same symptom that is happening here, only that now, Buildout fails to use wheels only when running on Python 3.10 and macOS=>11.

With kind regards,
Andreas.

[mauritsvanrees]

Hey, setting _PYTHON_HOST_PLATFORM="macosx-11-x86_64" also helped to fix the GHA build in zopefoundation/zope.security#87. (At least partially.) – That's interesting.

On the Plone core development buildout since recently we also have a problem with gh-actions on Mac with Python 3.10: an error installing MarkupSafe. Setting _PYTHON_HOST_PLATFORM helps here as well.

[amotl]

Dear Maurits,

thank you for also sharing your observations on this matter. For the lifetime of the GHA run logs, and for the sake of completeness, I would like to reference the one where Python 3.10 croaked on macOS on buildout.coredev [1].

On our end, it was enough to invoke sudo xcode-select -switch /Library/Developer/CommandLineTools in the workflow definition file, as suggested by @mikhailkoliada at actions/runner-images#5791 (comment). See crate-workbench/test-buildout-python310#2. Maybe both things are fixing different things? In any case, it is good to have them referenced here as a reference for others.

On the other aspect from my previous post: May I humbly ask you if you also would have expected that a corresponding MarkupSafe binary wheel package would have been selected from the package files published on PyPI at [2]? In this case, a compiler would not be needed and invoked at all, right? As mentioned above, I am looking at #544 here and wonder if Buildout's wheels-use machinery is broken in this specific condition as well.

With kind regards,
Andreas.

[1] https://github.com/plone/buildout.coredev/runs/7099812545
[2] https://pypi.org/project/MarkupSafe/#files

[dataflake]

I am seeing issues that may at least be related.

There's a very insidious bug in how buildout selects a suitable package from a list of packages provided from the package index in the method zc.buildout.easy_install._obtain that specifically affects macOS. It took me a long time to find it.

buildout/src/zc/buildout/easy_install.py

Lines 497 to 523 in a8d182c

	# Now find the best one:
	best = []
	bestv = None
	for dist in dists:
	distv = dist.parsed_version
	if bestv is None or distv > bestv:
	best = [dist]
	bestv = distv
	elif distv == bestv:
	best.append(dist)
	if not best:
	return None
	if len(best) == 1:
	return best[0]
	if self._download_cache:
	for dist in best:
	if (realpath(os.path.dirname(dist.location))
	==
	self._download_cache
	):
	return dist
	best.sort()
	return best[-1]

The code takes a list of distributions returned from a query to the package index for a package and version number. Example: I am running Python 3.10 on macOS/arm64 and want to install BTrees==4.11.2. The index query will return the following possible matches:

[BTrees 4.11.2 (https://files.pythonhosted.org/packages/90/4f/29127b14926568848416ba3c51a58d1310161749c9ed883bc38dd63e5a45/BTrees-4.11.2.tar.gz#sha256=98be22e387b4aeebb6381bd1ceec0311c250762dd8ba936216ae55c9419024ad),
 BTrees 4.11.2 (https://files.pythonhosted.org/packages/29/6c/cfc13b4374717c13f1b8c1ba07d01a8720b4a4c2f051a176aa133a300250/BTrees-4.11.2-cp310-cp310-macosx_10_9_universal2.whl#sha256=71333d8d2cc2bbd44427097c547e73e76b7ef462273f50344bb6f1ce1434f675),
 BTrees 4.11.2 (https://files.pythonhosted.org/packages/95/79/f9ede135b35763af1dcbd603fbe36f8e8ac0991332aacc51b9a9fb5557d2/BTrees-4.11.2-cp310-cp310-macosx_11_0_arm64.whl#sha256=d4f9f441ac760900dbce8d283032eb3100306dd7c6054787148dbc629cf98c58)]

One is the source distribution, and there are two matching binary distributions, one specifically for arm64 and the other a "fat" universal2 wheel that contains binary code with both arm64 and x86_64 compatibility. I am guessing the next steps (-> sort the resulting list and then simply return the last element) is based on the assumptions that sorting will always put the source distribution first and there is only one binary distribution. So in this particular case the arm64 wheel is returned.

This simplistic algorithm fails when Python is compiled with universal2 architecture, as the later Python versions on GitHub Actions runners are. It expects to get a universal2 wheel, but gets an arm64 wheel, leading to issues correctly converting the wheel to an egg that is usable:

While:
  Installing alltests.
  Getting distribution for 'zope.container==4.8'.

An internal error occurred due to a bug in either zc.buildout or in a
recipe being used:
Traceback (most recent call last):
  File "/Users/runner/work/Zope/Zope/.tox/py311/lib/python3.11/site-packages/zc/buildout/buildout.py", line 2252, in main
    getattr(buildout, command)(args)
  File "/Users/runner/work/Zope/Zope/.tox/py311/lib/python3.11/site-packages/zc/buildout/buildout.py", line 856, in install
    installed_files = self[part]._call(recipe.install)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/runner/work/Zope/Zope/.tox/py311/lib/python3.11/site-packages/zc/buildout/buildout.py", line 1652, in _call
    return f()
           ^^^
  File "/Users/runner/work/Zope/Zope/.tox/py311/eggs/zc.recipe.testrunner-2.2-py3.11.egg/zc/recipe/testrunner/__init__.py", line 44, in install
    eggs, ws = self.egg.working_set(('zope.testrunner', ))
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/runner/work/Zope/Zope/.tox/py311/eggs/zc.recipe.egg-2.0.7-py3.11-macosx-12.6-universal2.egg/zc/recipe/egg/egg.py", line 78, in working_set
    ws = self._working_set(
         ^^^^^^^^^^^^^^^^^^
  File "/Users/runner/work/Zope/Zope/.tox/py311/eggs/zc.recipe.egg-2.0.7-py3.11-macosx-12.6-universal2.egg/zc/recipe/egg/egg.py", line 161, in _working_set
    ws = zc.buildout.easy_install.install(
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/runner/work/Zope/Zope/.tox/py311/lib/python3.11/site-packages/zc/buildout/easy_install.py", line 971, in install
    return installer.install(specs, working_set)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/runner/work/Zope/Zope/.tox/py311/lib/python3.11/site-packages/zc/buildout/easy_install.py", line 744, in install
    for dist in self._get_dist(req, ws):
                ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/runner/work/Zope/Zope/.tox/py311/lib/python3.11/site-packages/zc/buildout/easy_install.py", line 581, in _get_dist
    dists = [_move_to_eggs_dir_and_compile(dist, self._dest)]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/runner/work/Zope/Zope/.tox/py311/lib/python3.11/site-packages/zc/buildout/easy_install.py", line 1948, in _move_to_eggs_dir_and_compile
    assert newdist is not None  # newloc above is missing our dist?!
    ^^^^^^^^^^^^^^^^^^^^^^^^^^

[mauritsvanrees]

@amotl I tried your repo https://github.com/crate-workbench/test-buildout-python310
One small thing: in the setup.py I need to add packages=[] otherwise setuptools complains, but this may depend on where you add a virtualenv.

Last night I updated my Mac from 12.6 to 13.5. When I try your buildout with Python 3.10.12 now I get the same error. I installed this with pyenv a month or so ago. When I reinstall this python (pyenv install -f 3.10.12) and try again, the buildout works.
@davisagli reported this behaviour as well a few months ago.

Reinstalling Python 3.11 fixed the same problem for me in a different buildout. I will do that with all my pyenv-installed Pythons now, and remove all *macosx-12* eggs from my eggs cache.

[amotl]

Dear Maurits, thank you very much for your swift response. I think we can consider this issue as resolved then? With kind regards, Andreas.

On 7 August 2023 12:41:13 CEST, Maurits van Rees ***@***.***> wrote: @amotl I tried your repo https://github.com/crate-workbench/test-buildout-python310 One small thing: in the `setup.py` I need to add `packages=[]` otherwise setuptools complains, but this may depend on where you add a virtualenv. Last night I updated my Mac from 12.6 to 13.5. When I try your buildout with Python 3.10.12 now I get the same error. I installed this with pyenv a month or so ago. When I reinstall this python (`pyenv install -f 3.10.12`) and try again, the buildout works. @davisagli reported this behaviour as well [a few months ago](#606 (comment)). Reinstalling Python 3.11 fixed the same problem for me in a different buildout. I will do that with all my pyenv-installed Pythons now, and remove all `*macosx-12*` eggs from my eggs cache. -- Reply to this email directly or view it on GitHub: #609 (comment) You are receiving this because you were mentioned. Message ID: ***@***.***>

-- Sent from my mind. This might have been typed on a mobile device, so please excuse my brevity.

[mauritsvanrees]

I think we can consider this issue as resolved then?

At least for me your example no longer gives an error.

There may still be subtle bugs lingering, for example I now have a problem installing zope.container 4.10 on Python 3.8, but we can discuss more where needed in issue #606.