Skip to content

Commit

Permalink
Migrate MacOs wheel binary builds to ephemeral M1 runners (#110432)
Browse files Browse the repository at this point in the history
Surprisingly there are no speed difference between running the cross-compilation on `macos12-xl` (x86_64 12 core machine) and `macos-13-xlarge` (m1 6 core machine)

Most of the changes are on the https://github.com/pytorch/builder side:
- pytorch/builder@50a6e91 skips installing mkl on M1 machines
- pytorch/builder@bbb29b0 same for llvm-9
- pytorch/builder@8bcc83d bumps minimal numpy version to 1.19 (as 1.17 is not available for m1)
- pytorch/builder@cc4f1f9 skips building tests/distributed for M1
Pull Request resolved: #110432
Approved by: https://github.com/kit1980
  • Loading branch information
malfet authored and pytorchmergebot committed Oct 3, 2023
1 parent 7f0a659 commit cd0e7d1
Show file tree
Hide file tree
Showing 7 changed files with 134 additions and 51 deletions.
5 changes: 3 additions & 2 deletions .github/scripts/generate_ci_workflows.py
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ class BinaryBuildWorkflow:
branches: str = "nightly"
# Mainly for macos
cross_compile_arm64: bool = False
xcode_version: str = ""
macos_runner: str = "macos-12-xl"

def __post_init__(self) -> None:
if self.abi_version:
Expand Down Expand Up @@ -307,7 +307,8 @@ class OperatingSystem:
build_configs=generate_binary_build_matrix.generate_wheels_matrix(
OperatingSystem.MACOS_ARM64
),
cross_compile_arm64=True,
cross_compile_arm64=False,
macos_runner="macos-13-xlarge",
ciflow_config=CIFlowConfig(
labels={LABEL_CIFLOW_BINARIES, LABEL_CIFLOW_BINARIES_WHEEL},
isolated_workflow=True,
Expand Down
10 changes: 7 additions & 3 deletions .github/templates/macos_binary_build_workflow.yml.j2
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ jobs:
{%- for config in build_configs %}
!{{ config["build_name"] }}-build:
if: ${{ github.repository_owner == 'pytorch' }}
runs-on: macos-12-xl
runs-on: !{{ macos_runner }}
timeout-minutes: !{{ common.timeout_minutes }}
!{{ upload.binary_env(config, true) }}
# For sccache access (only on non-forked PRs)
Expand All @@ -69,11 +69,15 @@ jobs:
- name: Install conda and dependencies
run: |
# Install conda, setup-miniconda messes with the path that messes with the ruby stuff we do later on
curl --retry 3 --retry-all-errors -o "${RUNNER_TEMP}/conda.sh" https://repo.anaconda.com/miniconda/Miniconda3-py310_23.5.2-0-MacOSX-x86_64.sh
curl --retry 3 --retry-all-errors -o "${RUNNER_TEMP}/conda.sh" "https://repo.anaconda.com/miniconda/Miniconda3-py310_23.5.2-0-MacOSX-$(uname -m).sh"
chmod +x "${RUNNER_TEMP}/conda.sh"
/bin/bash "${RUNNER_TEMP}/conda.sh" -b -p "${RUNNER_TEMP}/anaconda"
echo "${RUNNER_TEMP}/anaconda/bin" >> "${GITHUB_PATH}"
echo "DEVELOPER_DIR=/Applications/Xcode_13.3.1.app/Contents/Developer" >> "${GITHUB_ENV}"
if [ -d "/Applications/Xcode_14.3.1.app" ]; then
echo "DEVELOPER_DIR=/Applications/Xcode_14.3.1.app/Contents/Developer" >> "${GITHUB_ENV}"
elif [ -d "/Applications/Xcode_13.3.1.app" ]; then
echo "DEVELOPER_DIR=/Applications/Xcode_13.3.1.app/Contents/Developer" >> "${GITHUB_ENV}"
fi
!{{ common.checkout(deep_clone=False, directory="pytorch") }}
!{{ common.checkout(deep_clone=False, directory="builder", repository=common.builder_repo, branch=common.builder_branch) }}
- name: Install sccache (only for non-forked PRs, and pushes to trunk)
Expand Down
32 changes: 24 additions & 8 deletions .github/workflows/generated-macos-arm64-binary-conda-nightly.yml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

42 changes: 28 additions & 14 deletions .github/workflows/generated-macos-arm64-binary-wheel-nightly.yml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

32 changes: 24 additions & 8 deletions .github/workflows/generated-macos-binary-conda-nightly.yml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

0 comments on commit cd0e7d1

Please sign in to comment.