Possible regression with Python 3.10 on macOS >= 11; with reproduction repository

[amotl]

Description

Dear people of GitHub and Microsoft,

first things first: Thanks a stack for running the whole GHA machinery. It is indespensible!

We think we discovered a flaw when doing builds on macOS >= 11, when Python 3.10 and Buildout is involved, and a Python package dependency will be installed that needs to compile a C extension, like, for example, SQLAlchemy. We already reported the issue to the Buildout project at buildout/buildout#609, as we believe that is the right place where this should go to.

However, we are almost certainly sure that the same build worked without errors on Wed Jun 1 14:20:45 2022 +0200, when crate/crate-python#391 was merged. This patch, submitted 20 days ago, succeeded without errors, where at the same repository, another build with the same set of dependencies failed on behalf of crate/crate-python#428 just 4 days ago. In both cases, the designated runner image was macos-latest.

So, there might also be a regression on GHA's side, no? In any case, we wanted to report it here mainly as a reference for others to attach to. In this spirit, @iceman reported at buildout/buildout#609 (comment):

Since a week I also see failing GHA runs for Python 3.10 when C code is involved. It is only 3.10, older and newer versions are okay and only MacOS.

Apologies for the lengthy report, we hope we have been precise enough to describe our observations. Last but not least, you may be happy to hear that there is a minimal reproduction case repository at test-buildout-python310, where crate-workbench/test-buildout-python310#1 swiftly outlines the problem.

With kind regards,
Andreas.

P.S.: The macOS-11 images in question, released around and after Jun 1, 2022, but before Jun 17, 2022, are:

• 20220605.1
• 20220614.2

Platforms affected

• Azure DevOps
• GitHub Actions

Virtual environments affected

• Ubuntu 18.04
• Ubuntu 20.04
• Ubuntu 22.04
• macOS 10.15
• macOS 11
• macOS 12
• Windows Server 2019
• Windows Server 2022

Image version and build link

Failure

Image versions

• GHA: macos-11/20220614.2
• GHA: macos-12/20220615.1

Reproduction repository

A minimal reproduction case.

• https://github.com/crate-workbench/test-buildout-python310
• https://github.com/crate-workbench/test-buildout-python310/runs/6949516442

Original repository

Where the regression was discovered.

• CI: Investigate why the sandbox setup croaks with Python 3.10 on macOS >= 11, with Buildout crate/crate-python#428
• https://github.com/crate/crate-python/runs/6980393799

Is it a regression?

We believe so. The last known good configuration was:

Image version

• GHA: macos-11/20220530.2

Latest successful build

• https://github.com/crate/crate-python/actions/runs/2427279967

Expected behavior

The build should complete, as before with macos-11/20220530.2. Otherwise, it will be nice if we can explore the details about the potential change to share with the community in order to give them a chance to get a grasp what would need to be adjusted on their end.

Actual behavior

The build croaks, as outlined at crate-workbench/test-buildout-python310#1 and buildout/buildout#609.

Repro steps

There is a minimal reproduction case repository at test-buildout-python310, where crate-workbench/test-buildout-python310#1 swiftly outlines the problem.

[amotl]

Dear Mehdi,

thanks for your quick reply. I already exercised this detail and explicitly tested using all of Python 3.10.3, 3.10.4, and 3.10.5. The behavior was the same.

With kind regards,
Andreas.

[mikhailkoliada]

@amotl Hi! It does not look like a bug to me in the first place, your pipelines report xcode-select: Failed to locate 'gcc', requesting installation of command line developer tools, have you tried calling an xcode-tools installation to see if it helps?

[amotl]

Dear Mikhail,

thank you for taking the time to look into this.

Response

It does not look like a bug to me in the first place.

I also don't think it qualifies as a bug for GHA, because everything works flawlessly when doing the same things on CI/GHA without using Buildout. @idgserpro suggested at buildout/buildout#609 (comment) that it even might be an issue with the setuptools package.

Your pipelines report xcode-select: Failed to locate 'gcc', requesting installation of command line developer tools.

I saw that. However, I didn't follow up on trying to fix anything on that yet, because everything works fine without Buildout. So, I assume everything is perfectly well configured on the GHA images, and only Buildout messes it up.

Have you tried calling an xcode-tools installation to see if it helps?

I would not know how to do so. Any hints?

Further thoughts

It would need some amount of dedication to trace that issue down to the real root cause, which is most probably something within Buildout or setuptools, but only triggered on the most recent releases of the corresponding GHA runner images macos-11 and macos-12, where macos-10.15 is still fine. And, Python 3.10 only!

I think the best way forward is currently to follow @idgserpro's recommendation and try out the patch they outlined at pypa/setuptools#2514 (comment) to see if that might improve the situation. At least, it is a very good candidate to get an idea about where this issue might be originating from, like pkg_resources.compatible_platforms() and friends from the setuptools package ¹. However, chances are that Buildout mangles that what setuptools might be seeing, so vanilla setuptools is not affected?

If I had free wishes

The best thing to enable us to investigate those blurry issues in a self-service manner would be the possibility to address/request GHA images by version tag when running jobs, like macos-11:20220614.2. In that way, we would be able to switch back and forth in order to find out more details about possible regressions. However, I understand that this will probably never happen for general availability. Maybe there is a secret way to do it? ;]

With kind regards,
Andreas.

Footnotes

1. https://github.com/pypa/setuptools/blob/main/pkg_resources/__init__.py#L414-L457 ↩

[mikhailkoliada]

@amotl

I did not investigate what happens in your code under the hood, but usual dev dir selection solved the problem for me.

[amotl]

Dear Mikhail,

thank you very much for coming up with a solution so quickly. I just exercised your suggestion at crate-workbench/test-buildout-python310#2 and want to confirm it works flawlessly.

Now that I never ever typed that line you are suggesting into a GHA workflow definition, I am now learning that I will probably need to put that into my muscle memory, right? Or, otherwise, may I humbly ask you to elaborate more on this detail, for example whether something on that changed only recently?

With kind regards,
Andreas.

[amotl]

Hi again Mikhail.

Now that the issue to invoke the GCC compiler for building C extensions of Python modules on GHA/macOS>=11/Python310 has been addressed, I am returning to my post at crate-workbench/test-buildout-python310#1 (comment), where I said:

This observation raises two questions:

1. Why does the procedure try to invoke the compiler at all and not pull binary wheels from PyPI instead?
2. Why does the procedure not see the gcc compiler when it is falling back to it?

So, while I believe 2. can be recognized as resolved by fixing it through adding the line sudo xcode-select -switch /Library/Developer/CommandLineTools to the workflow definition file, as you suggested, I am still wondering why 1. also happens, beforehand already.

However, that might be a specific question to the maintainers of the greenlet project. If everything would go according to plan, when looking at the list of published wheel packages on PyPI [1], I think the expectation would be that the greenlet-1.1.2-cp310-cp310-macosx_10_14_x86_64.whl package [2] would be selected for installation.

If you still feel you can help us on shedding more light onto this detail of our observations, we would be very grateful. For example, if your outcome might be that the greenlet maintainers would have to adjust something on the packaging side, we will be happy to carry the corresponding communication back and forth. Edit: We also shared our observations at sqlalchemy/sqlalchemy#8182.

Appreciating your help very much.

With kind regards,
Andreas.

[1] https://pypi.org/project/greenlet/#files
[2] https://files.pythonhosted.org/packages/47/20/433693ac90ae70c8577bf4896951859e8d293e59df5818073033f181ee7b/greenlet-1.1.2-cp310-cp310-macosx_10_14_x86_64.whl

[mikhailkoliada]

@amotl Hi,

I did not investigate the code much, but something is clearly broken there in the tools detection part.
The thing is, we are installing xcode tools twice separately. In the first place we are installing the command-line-tools as they are pulling git and other tools as needed for homebrew installation, then we are installing xcode which always installs its own copy of toolchain and basic tools (so in real world you either do not need xcode and just install xcode command line tools or you install xcode and you do not need its command line tools separately). In the macOS runners by default we are pointing default developer path to the xcode installation, for instance, you can type xcode-select -p and it will return you the path of the currently default xcode with its developer dir. for example on macos 11 it will return /Applications/Xcode_13.2.1.app/Contents/Developer.

In the mean time you can still switch to the command-line only tools.
In your specific case something really weird happens during the build, if you take a closer look, the build tries to detect a gcc version from xcode, by calling something like:

$(xcode-select -p)/usr/bin/xcodebuild -sdk $(xcode-select -p)/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk -find gcc

if you perform this command during runtime toy'll see it returns you the correct gcc's path, while during the build it does not.

[mikhailkoliada]

@amotl as there is nothing we can help you with anymore, I am going to close the issue, but feel free to reopen it if you have more questions.

[amotl]

Dear Mikhail,

thank you very much for your support!

I also think this issue aggregated many useful details which may help others to resolve some specific issues they are facing. I also agree that, if at all, improvements or fixes would have to go into Buildout instead, because everything works flawlessly on GHA runners within vanilla Python environments.

With kind regards,
Andreas.