Builds do not work for Apple M1 target in GitHub Actions

[ofek]

resolving 1 targets
resolving target exe
error[PYOXIDIZER_BUILD]: could not find default Python distribution for x86_64-apple-darwin
 --> app/pyoxidizer.bzl:6:12
  |
6 |     dist = default_python_distribution(flavor="standalone_static")
  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ default_python_distribution()


error: could not find default Python distribution for x86_64-apple-darwin

resolving 1 targets
resolving target exe
error[PYOXIDIZER_BUILD]: could not find default Python distribution for aarch64-apple-darwin
 --> app/pyoxidizer.bzl:6:12
  |
6 |     dist = default_python_distribution(flavor="standalone_static")
  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ default_python_distribution()


error: could not find default Python distribution for aarch64-apple-darwin

name: build

on:
  push:
    branches:
    - main
  pull_request:
    branches:
    - main

concurrency:
  group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.sha }}
  cancel-in-progress: true

env:
  PYTHONUNBUFFERED: "1"
  FORCE_COLOR: "1"
  PYTHON_VERSION: "3.10"

jobs:
  apple-intel:
    name: Build standalone executable for macOS Intel
    runs-on: macos-10.15

    env:
      TARGET: "x86_64-apple-darwin"
      MACOSX_DEPLOYMENT_TARGET: "10.15"

    steps:
    - uses: actions/checkout@v3

    - name: Set up Python ${{ env.PYTHON_VERSION }}
      uses: actions/setup-python@v3
      with:
        python-version: ${{ env.PYTHON_VERSION }}

    - name: Install PyOxidizer
      run: python -m pip install --upgrade pyoxidizer

    - name: Install Rust target
      run: rustup target add ${{ env.TARGET }}

    - name: Build executable
      run: pyoxidizer build exe --path app --release --target-triple ${{ env.TARGET }}

    - name: Strip binary
      run: strip app/build/${{ env.TARGET }}/release/exe/devenv

    - name: Setup artifact directory
      run: |
        mv app/build/${{ env.TARGET }}/release targets
        mv targets/exe targets/${{ env.TARGET }}

    - uses: actions/upload-artifact@v3
      with:
        name: artifacts
        path: targets
        if-no-files-found: error
  apple-m1:
    name: Build standalone executable for macOS M1
    runs-on: ubuntu-20.04

    env:
      TARGET: "aarch64-apple-darwin"
      MACOSX_DEPLOYMENT_TARGET: "11.0"

    steps:
    - uses: actions/checkout@v3

    - name: Set up Python ${{ env.PYTHON_VERSION }}
      uses: actions/setup-python@v3
      with:
        python-version: ${{ env.PYTHON_VERSION }}

    - name: Set up QEMU
      uses: docker/setup-qemu-action@v2
      with:
        platforms: arm64

    - name: Install PyOxidizer
      run: python -m pip install --upgrade pyoxidizer

    - name: Install Rust target
      run: rustup target add ${{ env.TARGET }}

    - name: Build executable
      run: pyoxidizer build exe --path app --release --target-triple ${{ env.TARGET }}

    - name: Strip binary
      run: strip app/build/${{ env.TARGET }}/release/exe/devenv

    - name: Setup artifact directory
      run: |
        mv app/build/${{ env.TARGET }}/release targets
        mv targets/exe targets/${{ env.TARGET }}

    - uses: actions/upload-artifact@v3
      with:
        name: artifacts
        path: targets
        if-no-files-found: error

[ofek]

oh it should be standalone_dynamic https://pyoxidizer.readthedocs.io/en/stable/pyoxidizer_packaging_python_distributions.html#available-python-distributions

for aarch64-apple-darwin on ubuntu-20.04 with QEMU:

error[PYOXIDIZER_PYTHON_EXECUTABLE]: calling pip install

Caused by:
    Exec format error (os error 8)
  --> app/pyoxidizer.bzl:17:30
   |
17 |     exe.add_python_resources(exe.pip_install(["."]))
   |                              ^^^^^^^^^^^^^^^^^^^^^^ PythonExecutable.pip_install()


error: calling pip install

Caused by:
    Exec format error (os error 8)

for aarch64-apple-darwin on macos-11:

error[PYOXIDIZER_PYTHON_EXECUTABLE]: calling pip install

Caused by:
    Bad CPU type in executable (os error 86)
  --> app/pyoxidizer.bzl:17:30
   |
17 |     exe.add_python_resources(exe.pip_install(["."]))
   |                              ^^^^^^^^^^^^^^^^^^^^^^ PythonExecutable.pip_install()


error: calling pip install

Caused by:
    Bad CPU type in executable (os error 86)

[indygreg]

The first error seems to show us attempting to execute a Mach-O binary on Linux.

The second seems to show us attempting to execute an ELF binary for a foreign CPU type.

This points to a bug where we're not detecting the host target correctly and attempting to run the wrong Python distribution.

This should just work in modern versions of PyOxidizer.

I think a potential problem here might be the TARGET environment variable. That's used by Cargo to define the target triple. But I think it respects an already set TARGET environment variable. Try changing to TARGET_TRIPLE and see if you get better results?

[ofek]

Same after changing to TARGET_TRIPLE. Maybe cross compiling is not possible?

[indygreg]

I think maybe the issue here is you are running under ARM QEMU on Linux. What might be happening is PyOxidizer doesn't recognize the current host target and falls back to attempting to use an x86_64-unknown-linux-gnu Python distribution since an aarch64-unknown-linux-gnu distribution isn't available.

I'll try to push code today that adds the aarch64-unknown-linux-gnu distributions to PyOxidizer.

[ofek]

Still not yet working, using artifacts from https://github.com/indygreg/PyOxidizer/actions/runs/2416005657:

Run ./pyoxidizer build exe --path app --release --target-triple aarch64-apple-darwin
resolving 1 targets
resolving target exe
resolving Python distribution https://github.com/indygreg/python-build-standalone/releases/download/20220528/cpython-3.10.4%2B20220528-aarch64-apple-darwin-pgo-full.tar.zst (sha256=31978bf0f6911eb5873eba4fec5b8452e5df2ad6c78de2b71dfbe73cbce44182)
downloading https://github.com/indygreg/python-build-standalone/releases/download/20220528/cpython-3.10.4%2B20220528-aarch64-apple-darwin-pgo-full.tar.zst
Python distribution available at /home/runner/.cache/pyoxidizer/python_distributions/cpython-3.10.4%2B20220528-aarch64-apple-darwin-pgo-full.tar.zst
reading data from Python distribution...
resolving Python distribution https://github.com/indygreg/python-build-standalone/releases/download/20220528/cpython-3.10.4%2B20220528-x86_64-unknown-linux-gnu-pgo-full.tar.zst (sha256=ab27c27d034da39e999efce6bfa416be645918f27c13e89be9794dea0ef2f116)
downloading https://github.com/indygreg/python-build-standalone/releases/download/20220528/cpython-3.10.4%2B20220528-x86_64-unknown-linux-gnu-pgo-full.tar.zst
Python distribution available at /home/runner/.cache/pyoxidizer/python_distributions/cpython-3.10.4%2B20220528-x86_64-unknown-linux-gnu-pgo-full.tar.zst
reading data from Python distribution...
installing modified distutils to /tmp/pyoxidizer-pip-install26fJHn/packages
modifying distutils/_msvccompiler.py for oxidation
modifying distutils/command/build_ext.py for oxidation
modifying distutils/unixccompiler.py for oxidation
pip installing to /tmp/pyoxidizer-pip-install26fJHn/install
error[PYOXIDIZER_PYTHON_EXECUTABLE]: calling pip install

error: calling pip install

Caused by:
    Exec format error (os error 8)
    Caused by:
        Exec format error (os error 8)
      --> app/pyoxidizer.bzl:17:30
       |
    17 |     exe.add_python_resources(exe.pip_install(["."]))
       |                              ^^^^^^^^^^^^^^^^^^^^^^ PythonExecutable.pip_install()

[indygreg]

I'm still perplexed why we're getting an exec format error here.

But if you want to build for Apple M1, you should build from a macOS GitHub Actions runner for best results.

In theory cross-compilation from Linux to macOS is supported. In reality, it requires a lot of manual muckery. It isn't well documented and very brittle. I recommend using macOS to build for macOS.

[ofek]

But if you want to build for Apple M1, you should build from a macOS GitHub Actions runner for best results.

Oh I thought ARM emulation was required. Do you have an example?

[ofek]

Here's a repo you can fork to test it not working https://github.com/ofek/pyoxidizer-build-example

Latest build failing for M1 https://github.com/ofek/pyoxidizer-build-example/actions/runs/2431461199

[ofek]

Any ideas?

[indygreg]

This is due to the use of pip_install(). This method won't work when cross-compiling. This is documented at https://pyoxidizer.readthedocs.io/en/pyoxidizer-0.21.0/pyoxidizer_packaging_python_files.html#choosing-which-packaging-method-to-call.

Essentially, today pip_install() will blindly attempt to run the Python interpreter for the target triple. In your case, that's an interpreter built for M1. That won't work on an Intel macOS system. And attempts to execute the process fail with an opaque error like Bad CPU type in executable (os error 86).

Workaround is to use pip_download() when cross-compiling. Actually, use of pip_download() is straight up preferred. But it likely won't work with standalone_static builds because extension modules need to be recompiled from source in this build mode.

PyOxidizer should absolutely have better documentation and error reporting around this footgun.

[ofek]

Ah okay thanks. So how would one install the current package? I think exe.add_python_resources(exe.pip_download(["."])) would only produce wheels for dependencies.

[ofek]

I can confirm ^ does not include the current package.

Additionally, it seems v0.21.0 broke builds on macOS GitHub Actions runners e.g. macos-10.15:

Collecting pyoxidizer>=0.20.0
  Downloading pyoxidizer-0.21.0-py3-none-macosx_10_9_x86_64.whl (15.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 15.5/15.5 MB 25.7 MB/s eta 0:00:00
Installing collected packages: pyoxidizer
Successfully installed pyoxidizer-0.21.0
cmd [1] | pyoxidizer build exe --path app --target-triple x86_64-apple-darwin --release
dyld: Library not loaded: /usr/local/opt/xz/lib/liblzma.5.dylib
  Referenced from: /Users/runner/Library/Application Support/hatch/env/virtual/pyapp-UYSILkbu/build/bin/pyoxidizer
  Reason: no suitable image found.  Did find:
	/usr/local/opt/xz/lib/liblzma.5.dylib: code signature in (/usr/local/opt/xz/lib/liblzma.5.dylib) not valid for use in process using Library Validation: mapped file has no cdhash, completely unsigned? Code has to be at least ad-hoc signed.
	/usr/local/opt/xz/lib/liblzma.5.dylib: stat() failed with errno=1

[indygreg]

That liblzma dependency feels new and is definitely a bug. Will open a separate issue for tracking that. Will need to release a new version without this dependency. Thanks for reporting it.

Ah okay thanks. So how would one install the current package? I think exe.add_python_resources(exe.pip_download(["."])) would only produce wheels for dependencies.

You may want the read_package_root() method. See https://pyoxidizer.readthedocs.io/en/stable/pyoxidizer_packaging_python_files.html#packaging-an-application-from-a-local-python-package.

[indygreg]

Filed #585 for the liblzma issue.