On macOS 10.15/Python3.9.1 build_ext, bdist_wheel produce x86_64-only extensions with 'universal2' tag

[rluce]

Environment

• MacBook Pro (w/ Intel)
• macOS 10.15
• Python 3.9.1 from python.org (universal2 build)

((test-39) ~/tmp  % python3.9
Python 3.9.1 (v3.9.1:1e5d33e9b9, Dec  7 2020, 12:44:01) 
[Clang 12.0.0 (clang-1200.0.32.27)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sysconfig; sysconfig.get_platform()
'macosx-10.9-universal2'

• Xcode 12.3
• Other versions

(test-39) tmp/hello  % pip list
Package    Version
---------- -------
pip        20.3.3
setuptools 51.1.1
wheel      0.36.2

Observation

Compiling a C extension via build_ext (and packaging them via bdist_wheel) results in x86_64 only shared objects, but the Interpreter's platform tag universal2 is used in the wheel's tag nontheless. For me the expected behaviour is that the platform tag being used is 'x64_64', because the shared object does not have a arm64 slice. I know that I can manually override the 'universal2' tag in the wheel, but this default choice seems odd here.

Minimal example to reproduce:

File hello.c:

#include <Python.h>

static PyObject* helloworld(PyObject* self, PyObject* args) {
    printf("Hello, world\n");
    return Py_None;
}

static PyMethodDef myMethods[] = {
    { "hello", helloworld, METH_NOARGS, "say hello to the world" },
    { NULL, NULL, 0, NULL }
};

static struct PyModuleDef myModule = {
    PyModuleDef_HEAD_INIT,
    "hello",
    "only for testing",
    -1,
    myMethods
};

PyMODINIT_FUNC PyInit_hello(void) {
    return PyModule_Create(&myModule);
}

File setup.py:

from setuptools import setup, Extension

setup(name = 'hello', version = '1.0',
   ext_modules = [Extension('hello', ['hello.c'])])

Commands:

(test-39) tmp/hello  % python setup.py bdist_wheel
running bdist_wheel
running build
running build_ext
building 'hello' extension
creating build
creating build/temp.macosx-10.9-universal2-3.9
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -arch x86_64 -g -I/Users/luce/tmp/test-39/include -I/Library/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c hello.c -o build/temp.macosx-10.9-universal2-3.9/hello.o
creating build/lib.macosx-10.9-universal2-3.9
gcc -bundle -undefined dynamic_lookup -arch x86_64 -g build/temp.macosx-10.9-universal2-3.9/hello.o -o build/lib.macosx-10.9-universal2-3.9/hello.cpython-39-darwin.so
installing to build/bdist.macosx-10.9-universal2/wheel
running install
running install_lib
creating build/bdist.macosx-10.9-universal2
creating build/bdist.macosx-10.9-universal2/wheel
copying build/lib.macosx-10.9-universal2-3.9/hello.cpython-39-darwin.so -> build/bdist.macosx-10.9-universal2/wheel
running install_egg_info
running egg_info
creating hello.egg-info
writing hello.egg-info/PKG-INFO
writing dependency_links to hello.egg-info/dependency_links.txt
writing top-level names to hello.egg-info/top_level.txt
writing manifest file 'hello.egg-info/SOURCES.txt'
reading manifest file 'hello.egg-info/SOURCES.txt'
writing manifest file 'hello.egg-info/SOURCES.txt'
Copying hello.egg-info to build/bdist.macosx-10.9-universal2/wheel/hello-1.0-py3.9.egg-info
running install_scripts
creating build/bdist.macosx-10.9-universal2/wheel/hello-1.0.dist-info/WHEEL
creating 'dist/hello-1.0-cp39-cp39-macosx_10_9_universal2.whl' and adding 'build/bdist.macosx-10.9-universal2/wheel' to it
adding 'hello.cpython-39-darwin.so'
adding 'hello-1.0.dist-info/METADATA'
adding 'hello-1.0.dist-info/WHEEL'
adding 'hello-1.0.dist-info/top_level.txt'
adding 'hello-1.0.dist-info/RECORD'
removing build/bdist.macosx-10.9-universal2/wheel
(test-39) tmp/hello  % file build/lib.macosx-10.9-universal2-3.9/hello.cpython-39-darwin.so
build/lib.macosx-10.9-universal2-3.9/hello.cpython-39-darwin.so: Mach-O 64-bit bundle x86_64

Further discussion

On macOS 10.15 is it possible to cross compile for arm64, too, if a recent enough Xcode version (>=12.2) and the appropriate macOS SDK (>= 11.0) is used. So technically it is possible create truly universal2 extensions with that OS (but -arch arm64 would need to be added to the CFLAGS then). In the example above however, the invocation of 'gcc' results in the macOS 10.15 SDK being used, and the result can only be a x86_64 binary.

Pinging @ronaldoussoren who has done a lot of work for the universal2 integration.

[rluce]

There is an upstream bug report for this, https://bugs.python.org/issue42619.

[ronaldoussoren]

The upstream issue contains some more information on the background, in particular a comment in _osx_support.py (in the standard library) that says:

        # Use the original CFLAGS value, if available, so that we
        # return the same machine type for the platform string.
        # Otherwise, distutils may consider this a cross-compiling
        # case and disallow installs.

The side effect of that behaviour is that the sysconfig.get_platform() for a universal build will always return the platform tag for that universal build, even if the current machine cannot build for some of those architectures. I haven't done further research on this and don't know why using a correct tag would cause problems.