Build changes from MacPorts #368

danchr · 2021-06-13T17:22:37Z

The PyObjC build system is currently quite lenient and fragile. In particular, failure to build a module will just stop the build, print something, and exit with success. This has caused issues for MacPorts, where we build PyObjC for all applicable OS releases, hosted on that release itself, and preferably using the same libffi everywhere.

The just-stop-and-feign-success behaviour became particularly annoying on Big Sur and Catalina, where pyobjc-core doesn't actually build with stock libffi. So, I fixed it to use the system libffi, and made it use subprocess.check_call() for invoking sub-builds, only to find out that anything earlier than 10.11 as well as 10.13 didn't build.

This PR also contains some other fixes to use setuptools more, handle more SDKs and not pass -Werror during the build. Some of these might not be acceptable to you; if so, I can drop them.

This prevents passing e.g. -j to the build

Should one of the modules fail, the build would just print an error, but still exit with a success.

This is generally unsuitable for distribution, as it may cause failures on e.g. deprecation.

ronaldoussoren · 2021-06-14T06:38:44Z

develop.py

@@ -15,7 +15,7 @@
 import shutil
 import subprocess
 import sys
-from distutils.sysconfig import get_config_var
+from setuptools._distutils.sysconfig import get_config_var


This doesn't look correct, the new version of the code uses a private submodule of setuptools

I won't use this bit, I've changed the script to use sysconfig.get_config_var.

ronaldoussoren · 2021-06-14T06:38:53Z

develop.py

@@ -122,7 +122,7 @@ def get_sdk_level():
    assert sdkname.startswith("MacOSX")
    assert sdkname.endswith(".sdk")

-    if sdkname == "MacOSX.sdk":
+    if sdkname == "MacOSX.sdk" or "." not in sdkname[6:-4]:


Why is this needed?

Because MacOSX11.sdk exists as a valid option at /Library/Developer/CommandLineTools/SDKs. This particular one affects Homebrew as well.

I'm working on a change that will turn the logic around: use SDKSettings.plist where it is available and only use the version in the base name of the SDK when there is no plist file. Both here and in the various setup files.

Sounds good to me. We also primarily use SDKSettings in Homebrew's SDK selection and it's worked well so it should work for PyObjC too. Just be aware you may need to strip the patch version like you do to the OS version (there was a 10.15.4 SDK).

Stripping shouldn't be necessary, the code should work fine with more than 3 labels....

db2cdc8 implements the new code.

ronaldoussoren · 2021-06-14T06:39:09Z

development-support/wheel-smoke-test.py

@@ -6,7 +6,7 @@
 """

 import platform
-from distutils.version import LooseVersion
+from setuptools._distutils.version import LooseVersion


See above, private submodule of setuptools.

changeset 12daa01 removes the dependency on distutils in this file.

ronaldoussoren · 2021-06-14T06:39:39Z

install.py

@@ -15,7 +15,7 @@
 import shutil
 import subprocess
 import sys
-from distutils.sysconfig import get_config_var
+from setuptools._distutils.sysconfig import get_config_var


ronaldoussoren · 2021-06-14T06:39:48Z

install.py

@@ -122,7 +122,7 @@ def get_sdk_level():
    assert sdkname.startswith("MacOSX")
    assert sdkname.endswith(".sdk")

-    if sdkname == "MacOSX.sdk":
+    if sdkname == "MacOSX.sdk" or "." not in sdkname[6:-4]:


ronaldoussoren · 2021-06-14T06:40:46Z

install.py

@@ -221,16 +221,10 @@ def build_project(project, extra_args):
        shutil.rmtree(os.path.join(proj_dir, "build"))

    print("Installing {!r} using {!r}".format(project, sys.executable))
-    status = subprocess.call(
+    subprocess.check_call(


I don't like this change, the old code nicely prints which subproject fails to install and the new one doesn't.

ronaldoussoren · 2021-06-14T06:42:48Z

install.py

-        if not ok:
-            break
+        print(f"\nBuilding {project}...\n")
+        build_project(project, sys.argv[1:])


Not ok. The old code is intentional, the code in framework wrappers should either build correctly or be skipped entirely (that is, bindings for frameworks introduced in macOS 11 will be skipped automatically on macOS 10.15 or earlier).

ronaldoussoren · 2021-06-14T06:43:59Z

pyobjc-core/setup.py

-from distutils.errors import DistutilsError, DistutilsPlatformError, DistutilsSetupError
-from distutils.sysconfig import get_config_var as _get_config_var
-from distutils.sysconfig import get_config_vars
+from setuptools._distutils import log


once again imports from a private submodule.

ronaldoussoren · 2021-06-14T06:51:17Z

How does MacPorts build PyObjC? The "install.py" and "development.py" scripts in the root of the repository are mostly meant as easy ways to install the software during development, that they are useful for others is a nice side effect.

The correct way to install PyObjC is to use the distribution artefacts on PyPI, where the wheel archives are the preferred installation archive. Although I totally understand that a project like MacPorts prefers to install from source.

ronaldoussoren · 2021-10-26T12:59:16Z

See my previous question: How does macports package pyobjc? The correct way to package the software is IMHO to package all subprojects, and not use install.py/develop.py.

ronaldoussoren · 2021-10-26T13:07:08Z

I won't be changing "-Werror" in the setup.py files, I've caught too many errors that would have been missed without this flag.

Apparently some users get compile time errors in some situations, but so far I've not been able to reproduce these.

danchr · 2021-11-01T09:23:46Z

See my previous question: How does macports package pyobjc? The correct way to package the software is IMHO to package all subprojects, and not use install.py/develop.py.

In order to do that, I'd have to add logic to extract all the many frameworks, list each and every one as a subport and determine on which platforms each one is available. In other words, I'd have to reimplement the logic you already have in develop.py and install.py. Simply having one port containing all frameworks is much simpler and maintainable, and allows me to get the “right” list of frameworks for every single OS without further ado.

ronaldoussoren · 2021-11-01T09:31:12Z

See my previous question: How does macports package pyobjc? The correct way to package the software is IMHO to package all subprojects, and not use install.py/develop.py.

In order to do that, I'd have to add logic to extract all the many frameworks, list each and every one as a subport and determine on which platforms each one is available. In other words, I'd have to reimplement the logic you already have in develop.py and install.py. Simply having one port containing all frameworks is much simpler and maintainable, and allows me to get the “right” list of frameworks for every single OS without further ado.

Could you file an issues about this with the features you need and would like to have for this?

The main interface for installing pyobjc is to use "pip install pyobjc" using the artefacts on PyPI. That's obviously not entirely compatible with package managers like MacPorts.

The install.py and develop.py scripts slowly evolve over time as my needs change, without knowing your requirements this is bound to result in accidentally breaking your tooling in the future :-(

danchr added 5 commits April 9, 2021 19:11

use distutils from setuptools, it's patched

81ba485

setup: handle MacOSX11.sdk

6224254

setup: don't overwrite user_options

79e6721

This prevents passing e.g. -j to the build

install: don't fail silently

1ad85a4

Should one of the modules fail, the build would just print an error, but still exit with a success.

setup: do not pass -Werror during build

ec8c4c3

This is generally unsuitable for distribution, as it may cause failures on e.g. deprecation.

danchr force-pushed the macports-setup-changes branch from d539aa4 to ec8c4c3 Compare June 13, 2021 17:29

ronaldoussoren requested changes Jun 14, 2021

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Build changes from MacPorts #368

Build changes from MacPorts #368

danchr commented Jun 13, 2021

ronaldoussoren Jun 14, 2021

ronaldoussoren Oct 26, 2021

ronaldoussoren Jun 14, 2021

Bo98 Oct 24, 2021 •

edited

ronaldoussoren Oct 26, 2021

Bo98 Oct 26, 2021 •

edited

ronaldoussoren Oct 26, 2021

ronaldoussoren Oct 26, 2021

ronaldoussoren Jun 14, 2021

ronaldoussoren Oct 26, 2021

ronaldoussoren Jun 14, 2021

ronaldoussoren Jun 14, 2021

ronaldoussoren Jun 14, 2021

ronaldoussoren Jun 14, 2021

ronaldoussoren Jun 14, 2021

ronaldoussoren commented Jun 14, 2021

ronaldoussoren commented Oct 26, 2021

ronaldoussoren commented Oct 26, 2021

danchr commented Nov 1, 2021

ronaldoussoren commented Nov 1, 2021

Build changes from MacPorts #368

Are you sure you want to change the base?

Build changes from MacPorts #368

Conversation

danchr commented Jun 13, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Bo98 Oct 24, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Bo98 Oct 26, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ronaldoussoren commented Jun 14, 2021

ronaldoussoren commented Oct 26, 2021

ronaldoussoren commented Oct 26, 2021

danchr commented Nov 1, 2021

ronaldoussoren commented Nov 1, 2021

Bo98 Oct 24, 2021 •

edited

Bo98 Oct 26, 2021 •

edited