Build macos arm64 wheels #7

jcfr · 2023-04-26T01:04:49Z

Closes #17

To provide some context, this commit reverts the following:

Turn off arm64 builds in cibuildwheel

Because GitHub Actions does not yet have arm64 runners available,
cross-compiling is the only option for making arm64 wheels.

We are running into several issues with cross-compiling our module,
including the fact that, apparently, part of the compilation process
runs at least one executable (vtkWrapHierarchy-9.2) in the VTK SDK
(which, if we are cross-compiling, will not work since the architecture
does not match). We also seem to be getting some linking errors, which
may or may not be important.

Source: Kitware/LookingGlassVTKModule@3a6888a from Sept 2022

Associated GitHub issues related to official support that would avoid relying on cross-compilation:

jcfr · 2023-04-26T01:20:56Z

Similarly to the report from Sept 2022, the error is related to a failure to execute vtkWrapHierarchy-9.2compiled for macOS arm64 on a macOS intel host:

cd /Users/runner/work/VTKU3DExporter/VTKU3DExporter/_skbuild/macosx-10.10-arm64-3.11/cmake-build/inner-build/vtkU3DExporter && /Users/runner/work/VTKU3DExporter/VTKU3DExporter/_deps/vtk-wheel-sdk-9.2.5-cp311-cp311-macosx_11_0_arm64/bin/vtkWrapHierarchy-9.2 @/Users/runner/work/VTKU3DExporter/VTKU3DExporter/_skbuild/macosx-10.10-arm64-3.11/cmake-build/inner-build/vtkU3DExporter/CMakeFiles/vtkU3DExporter-hierarchy.Release.args -o /Users/runner/work/VTKU3DExporter/VTKU3DExporter/_skbuild/macosx-10.10-arm64-3.11/cmake-build/inner-build/lib/vtk/hierarchy/vtkU3DExporter/vtkU3DExporter-hierarchy.txt /Users/runner/work/VTKU3DExporter/VTKU3DExporter/_skbuild/macosx-10.10-arm64-3.11/cmake-build/inner-build/vtkU3DExporter/CMakeFiles/vtkU3DExporter-hierarchy.data @/Users/runner/work/VTKU3DExporter/VTKU3DExporter/_skbuild/macosx-10.10-arm64-3.11/cmake-build/inner-build/vtkU3DExporter/CMakeFiles/vtkU3DExporter-hierarchy.depends.args
/bin/sh: /Users/runner/work/VTKU3DExporter/VTKU3DExporter/_deps/vtk-wheel-sdk-9.2.5-cp311-cp311-macosx_11_0_arm64/bin/vtkWrapHierarchy-9.2: Bad CPU type in executable
[2/12] Building CXX object vtkU3DExporter/CMakeFiles/U3DExporter.dir/vtkU3DExporter.cxx.o
ninja: build stopped: subcommand failed.

jcfr · 2023-04-26T14:33:55Z

The VTK compile tools are being built successfully on macOS host (Intel) when we are cross-compiling the arm64 wheels:

That said, downloading VTK in a sub-directory of the source tree is having unintended side effects:

Call Stack (most recent call first):
  CMakeLists.txt:263 (vtk_module_scan)

CMake Error at /Users/runner/work/VTKU3DExporter/VTKU3DExporter/_deps/vtk-wheel-sdk-9.2.5-cp311-cp311-macosx_11_0_arm64/vtk-9.2.5.data/headers/cmake/vtkModule.cmake:262 (message):
  A VTK module requires a name (from
  /Users/runner/work/VTKU3DExporter/VTKU3DExporter/_deps/VTKCompileTools/VTK/Accelerators/Vtkm/Core/vtk.module;/Users/runner/work/VTKU3DExporter/VTKU3DExporter/_deps/VTKCompileTools/VTK/Accelerators/Vtkm/DataModel/vtk.module;...).
Call Stack (most recent call first):
  /Users/runner/work/VTKU3DExporter/VTKU3DExporter/_deps/vtk-wheel-sdk-9.2.5-cp311-cp311-macosx_11_0_arm64/vtk-9.2.5.data/headers/cmake/vtkModule.cmake:648 (_vtk_module_parse_module_args)
  CMakeLists.txt:263 (vtk_module_scan)
-- Configuring incomplete, errors occurred!

The VTK modules are discovered along side the one from VTKU3DExporter

jcfr · 2023-08-30T23:31:47Z

@Korijn arm64 wheels can now be built 🎉

Note

This particular wheel can be downloaded at https://github.com/ClinicalGraphics/VTKU3DExporter/actions/runs/6030972183?pr=7

Immediate Next steps:

Consolidate the changes in VTKExternalModule¹
- Add support for finding VTKCompileTools and passing cross-compilation variables KitwareMedical/VTKExternalModule#7
Consolidate the changes in this pull request
- ENH: Update VTKExternalModule adding support for cross-compilation #23
- Re-enable build for all python versions
Locally download wheels uploaded as artifact and confirm they work (due to the fact we cross-compile from a macOS x86_64 GitHub runner targeting arm64, we can not run the tests)
Tag

https://github.com/KitwareMedical/VTKExternalModule/compare/main...jcfr:add-crosscompiling-support?expand=0 ↩

List of changes: $ git shortlog c1906cf12..37ade3c26 --no-merges Jean-Christophe Fillion-Robin (3): Remove redundant BUILD_TESTING option COMP: Support configuring install directories Add support for finding VTKCompileTools and passing cross-compilation variables (#7)

Note this is a general fix and it does not mean that the CMAKE_CROSSCOMPILING and CMAKE_CROSSCOMPILING_EMULATOR will be passed down.

…l_sdk_version()

Update setup.py so that these variables are passed: - CMAKE_CROSSCOMPILING - CMAKE_SYSTEM_NAME - VTKCompileTools_DIR Add support for building VTKCompileTools for x86_64 (the one provided in the wheel sdk are for arm64): - Update setup.py adding "auto_build_vtk_compile_tools()" - Add BUILDVTKCompileTools CMake module and associated test ensuring the tool can be executed on the GitHub runner

jcfr · 2023-08-31T02:29:28Z

After downloading the artifacts¹ and installing on a macOS M1 workstation, the dependencies are successfully installed:

$ pip install vtk_u3dexporter-0.1.dev1+g67a72a4-cp311-cp311-macosx_11_0_arm64.whl 
Processing ./vtk_u3dexporter-0.1.dev1+g67a72a4-cp311-cp311-macosx_11_0_arm64.whl
Collecting vtk==9.2.5
  Downloading vtk-9.2.5-cp311-cp311-macosx_11_0_arm64.whl (58.9 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 58.9/58.9 MB 10.3 MB/s eta 0:00:00
Collecting matplotlib>=2.0.0
  Downloading matplotlib-3.7.2-cp311-cp311-macosx_11_0_arm64.whl (7.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.3/7.3 MB 70.2 MB/s eta 0:00:00
Collecting contourpy>=1.0.1
  Downloading contourpy-1.1.0-cp311-cp311-macosx_11_0_arm64.whl (229 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 229.3/229.3 kB 27.2 MB/s eta 0:00:00
Collecting cycler>=0.10
  Downloading cycler-0.11.0-py3-none-any.whl (6.4 kB)
Collecting fonttools>=4.22.0
  Downloading fonttools-4.42.1-cp311-cp311-macosx_10_9_universal2.whl (2.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.7/2.7 MB 72.2 MB/s eta 0:00:00
Collecting kiwisolver>=1.0.1
  Downloading kiwisolver-1.4.5-cp311-cp311-macosx_11_0_arm64.whl (66 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 66.2/66.2 kB 9.9 MB/s eta 0:00:00
Collecting numpy>=1.20
  Downloading numpy-1.25.2-cp311-cp311-macosx_11_0_arm64.whl (14.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 14.0/14.0 MB 74.3 MB/s eta 0:00:00
Collecting packaging>=20.0
  Using cached packaging-23.1-py3-none-any.whl (48 kB)
Collecting pillow>=6.2.0
  Using cached Pillow-10.0.0-cp311-cp311-macosx_11_0_arm64.whl (3.1 MB)
Collecting pyparsing<3.1,>=2.3.1
  Downloading pyparsing-3.0.9-py3-none-any.whl (98 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 98.3/98.3 kB 17.5 MB/s eta 0:00:00
Collecting python-dateutil>=2.7
  Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 247.7/247.7 kB 26.7 MB/s eta 0:00:00
Collecting six>=1.5
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Installing collected packages: six, pyparsing, pillow, packaging, numpy, kiwisolver, fonttools, cycler, python-dateutil, contourpy, matplotlib, vtk, vtk-u3dexporter
Successfully installed contourpy-1.

https://github.com/ClinicalGraphics/VTKU3DExporter/actions/runs/6030972183?pr=7 ↩

jcfr · 2023-08-31T02:44:22Z

After:

creating a virtual env.
downloading the sources (to have the test available)
installing the testing requirements

I was able to run pytest and confirm the tests pass ✅

Prerequisites

$ git clone git clone https://@github.com/ClinicalGraphics/VTKU3DExporter.git
[...]

$ python -c "import os; print(os.uname())"
posix.uname_result(sysname='Darwin', nodename='grax', release='22.2.0', version='Darwin Kernel Version 22.2.0: Fri Nov 11 02:04:44 PST 2022; root:xnu-8792.61.2~4/RELEASE_ARM64_T8103', machine='arm64')

$ python -c "import os; print(os.uname().machine)"
arm64

$ pip install VTKU3DExporter/Testing/Python/requirements.txt
[...]

Tests

$ pytest -v VTKU3DExporter/Testing/Python 
=========================================================================================== test session starts ===========================================================================================
platform darwin -- Python 3.11.0, pytest-7.4.0, pluggy-1.3.0 -- /Users/svc-dashboard/D/T/vth-u3dexporter-3.11/bin/python3.11
cachedir: .pytest_cache
rootdir: /Users/svc-dashboard/D/T/VTKU3DExporter
collected 2 items                                                                                                                                                                                         

VTKU3DExporter/Testing/Python/test_import.py::test_basic_import PASSED                                                                                                                              [ 50%]
VTKU3DExporter/Testing/Python/test_u3d_generation.py::test_u3d_generation PASSED                                                                                                                    [100%]

=========================================================================================== 2 passed in 14.97s ============================================================================================

jcfr force-pushed the build-macos-arm64-wheels branch from 9d7226e to d83704f Compare April 26, 2023 14:16

jcfr force-pushed the build-macos-arm64-wheels branch 5 times, most recently from 2c499e6 to d2c5449 Compare April 27, 2023 05:09

jcfr force-pushed the build-macos-arm64-wheels branch from d2c5449 to 6c48899 Compare August 30, 2023 14:07

jcfr added 3 commits August 30, 2023 21:09

COMP: Update U3D external project to ensure variables are passed down

1c4b403

Note this is a general fix and it does not mean that the CMAKE_CROSSCOMPILING and CMAKE_CROSSCOMPILING_EMULATOR will be passed down.

ENH: Re-factor setup.py introducing vtk_wheel_sdk_name() and vtk_whee…

98d7578

…l_sdk_version()

jcfr force-pushed the build-macos-arm64-wheels branch from 3d23b0b to 511670e Compare August 31, 2023 01:10

jcfr merged commit c4d15ec into main Aug 31, 2023
4 checks passed

jcfr deleted the build-macos-arm64-wheels branch August 31, 2023 02:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Build macos arm64 wheels #7

Build macos arm64 wheels #7

jcfr commented Apr 26, 2023 •

edited by Korijn

Loading

jcfr commented Apr 26, 2023 •

edited

Loading

jcfr commented Apr 26, 2023

jcfr commented Aug 30, 2023 •

edited

Loading

jcfr commented Aug 31, 2023

jcfr commented Aug 31, 2023 •

edited

Loading

Build macos arm64 wheels #7

Build macos arm64 wheels #7

Conversation

jcfr commented Apr 26, 2023 • edited by Korijn Loading

jcfr commented Apr 26, 2023 • edited Loading

jcfr commented Apr 26, 2023

jcfr commented Aug 30, 2023 • edited Loading

Footnotes

jcfr commented Aug 31, 2023

Footnotes

jcfr commented Aug 31, 2023 • edited Loading

Prerequisites

Tests

jcfr commented Apr 26, 2023 •

edited by Korijn

Loading

jcfr commented Apr 26, 2023 •

edited

Loading

jcfr commented Aug 30, 2023 •

edited

Loading

jcfr commented Aug 31, 2023 •

edited

Loading