-
Notifications
You must be signed in to change notification settings - Fork 231
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: support building universal/arm64 from x64_86 Python 3.8 #704
Conversation
cibuildwheel/macos.py
Outdated
@@ -286,6 +281,11 @@ def setup_python( | |||
# we can do universal2 builds on macos 10.15, but we need to | |||
# set ARCHFLAGS otherwise CPython sets it to `-arch x86_64` | |||
env.setdefault("ARCHFLAGS", "-arch arm64 -arch x86_64") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You need to set _PYTHON_HOST_PLATFORM=macosx-11.0-universal2
and ARCHFLAGS
here (regardless of the os version).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Awesome, thanks! Though I think we could set _PYTHON_HOST_PLATFORM=macosx-10.9-universal2
, since the Intel portion supports 10.9+ (it does require a slightly newer packaging/pip to be supported). Actually, should we get this number from value of MACOSX_DEPLOYMENT_TARGET
? Or does that get computed correctly automatically?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actually, should we get this number from value of MACOSX_DEPLOYMENT_TARGET?
Sure. (That might break a few workflows because the wheel tag will have say 10.11 instead of the expected 10.9 which users might have renamed to correctly reflect MACOS_DEPLOYMENT_TARGET. Though this should be minor and using MACOSX_DEPLOYMENT_TARGET is the correct strategy.)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Setting this as the minimum (10.9) seems to do the right thing:
3 wheels produced in 13 minutes:
boost_histogram-1.0.3.dev15+g7314060-cp38-cp38-macosx_10_12_universal2.whl
boost_histogram-1.0.3.dev15+g7314060-cp38-cp38-macosx_10_12_x86_64.whl
boost_histogram-1.0.3.dev15+g7314060-cp38-cp38-macosx_11_0_arm64.whl
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I went ahead & use MACOS_DEPLOYMENT_TARGET in the strategy. Feel free to drop the commit if not appropriate.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could you set CMAKE_OSX_ARCHITECTURES as well? https://cmake.org/cmake/help/latest/variable/CMAKE_OSX_ARCHITECTURES.html
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
scikit-build
(the official CMake build system for for Python) now supports ARCHFLAGS
, and https://github.com/pybind/cmake_example does too. If you are wrapping CMake yourself, I think you should respect ARCHFLAGS
as well. I don't think we can or should set every build tool's special flags for this, but the tooling should support the standard Python settings.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
See https://github.com/pybind/cmake_example/blob/f6539807e764707f2ee2b9d7ba0c8d017d685853/setup.py#L93-L97 , specifically, if you are not using Scikit-build for some reason.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good to know this. I will try Scikit-build. Would mind telling me who defined the "ARCHFLAGS" standard?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's from setuptools, and therefore probably from distutils in the Python standard library.
37cd2f8
to
3a4f23e
Compare
3a4f23e
to
11a24d9
Compare
It is possible to also add to documentation information on how cross-compile external libraries or some links to such documents? I one project (https://github.com/czaki/imagecodecs_build) time of external libraries build is ~30 min, so I prefer to build it once per architecture. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah, that's a nice solution to the Python 3.8 README footnote issue :D actually make it work!
This is a bit of a head-scratcher, but now that I've read it a few times, it does make sense. I don't have a strong opinion on whether MACOSX_DEPLOYMENT_TARGET should match _PYTHON_HOST_PLATFORM, but I agree that it does seem like the most 'correct' way to do it. So I guess I'm +0.1 on how it is currently.
LGTM, comments are only suggestions.
cibuildwheel/macos.py
Outdated
# macOS 11 is the first OS with arm64 support, so the wheels | ||
# have that as a minimum. | ||
env.setdefault("_PYTHON_HOST_PLATFORM", "macosx-11.0-arm64") | ||
deployment_target = max(deployment_target, (11, 0)) | ||
host_arch = "arm64" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
host_arch
seems like the wrong name, I think... perhaps target_arch
? I know that we're setting the _PYTHON_HOST_PLATFORM
env var, but I think the only reason it's called that is because it overrides a function called get_host_platform
in distutils.
cibuildwheel/macos.py
Outdated
deployment_target = tuple(map(int, env.get("MACOSX_DEPLOYMENT_TARGET", "10.9").split(".")[:2])) | ||
if deployment_target[0] >= 11: | ||
# starting with macOS 11, the minor version in fact represents patch version | ||
deployment_target = (deployment_target[0], 0) | ||
if len(deployment_target) < 2: | ||
# even though it didn't make much sense before macOS 11, it was valid to set | ||
# MACOSX_DEPLOYMENT_TARGET=10 which is 10.0 | ||
deployment_target = (deployment_target[0], 0) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I tripped up on this a little, perhaps something like the following would be easier to read?
deployment_target = tuple(map(int, env.get("MACOSX_DEPLOYMENT_TARGET", "10.9").split(".")[:2])) | |
if deployment_target[0] >= 11: | |
# starting with macOS 11, the minor version in fact represents patch version | |
deployment_target = (deployment_target[0], 0) | |
if len(deployment_target) < 2: | |
# even though it didn't make much sense before macOS 11, it was valid to set | |
# MACOSX_DEPLOYMENT_TARGET=10 which is 10.0 | |
deployment_target = (deployment_target[0], 0) | |
parts = env.get("MACOSX_DEPLOYMENT_TARGET", "10.9").split(".") | |
if len(parts) < 2: | |
deployment_target = int(parts[0]), 0 | |
else: | |
deployment_target = int(parts[0]), int(parts[1]) | |
if deployment_target[0] >= 11: | |
# starting with macOS 11, we retain compatibility across minor versions | |
deployment_target = (deployment_target[0], 0) |
I'm in favor of the original form, for the following reasons:
If there was a clear reason to do this, then that would be different. But the wheel tags produced are already correct; they respect |
As mentioned in my last comment, I mostly agree with @henryiii and think that this PR should drop my commits to proceed in its original form. I've backed up the branch on my fork if we want to resume the conversation on this one later: https://github.com/mayeut/cibuildwheel/tree/backup-mdt |
Sounds good to me |
ddb1b6b
to
11a24d9
Compare
11a24d9
to
6a9cec0
Compare
This adds support for creating wheels with CPython 3.8 for Apple Silicon. This is significant because 3.8 is the default Python on macOS's Xcode CLT.
Orignal message:
This currently produces an "x86_64" tagged wheel instead of the correct universal2 tag (but the wheel is fine if you rename it, other than having the wrong tag also listed in WHEEL). See scipy/oldest-supported-numpy#20 and grab example wheels at https://github.com/scikit-hep/boost-histogram/pull/583.