Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bootloader: Add Dockerfile to cross compile Linux bootloaders for abnormal architectures. #5995

Merged
merged 1 commit into from
Jul 11, 2021

Conversation

bwoodsend
Copy link
Member

Add a Dockerfile which leverages the 'dockcross' family of cross compiler base images to build the Linux bootloaders for almost any architecture.

This required a few preliminary changes:

  • The waf script now always assumes it's building for the C compiler's reported target architecture if the target OS is Linux. Previously, it was determined from the host machine which of course is wrong in a cross compile setup.

  • To avoid bootloaders of different architectures clobbering each other if present in the same installation or git clone, append an architecture family name in the path to the bootloader. e.g. A ppc64le (Power PC, 64 bit, little endian) bootloader would be saved under PyInstaller/bootloader/Linux-64bit-ppc/run. This change was already half implemented as a special case for ARM and sw_64. This change only applies to Linux. All other OSs either don't need this change or, in the case of macOS, have their own special requirements and handling. This use of a loose family name for architectures rather than the exact output from platform.machine() is needed so that waf can predict this name based only on the target reported by the C compiler which is considerably less exact than that of uname -m or platform.machine(). For example, when platform.machine() is armv5l, armv6l or armv7h, the C compiler simply reports arm in all cases. The mapping should go from armv7h to arm rather than the other way around.

A couple of unintended bonuses of this approach:

  • dockcross provide manylinux2014 compatible compilers (see PEPs 513 and 599) to avoid the glibc symbol runtime errors that crop up if you don't keep glibc requirements lower than your target's version. This commit can also be used to build native x86_64 binaries using a non-cross variant which conform to the manylinux2014 standard (which, as of setup: Build per-platform wheels. #5787, is advertised by our wheel tags).

  • dockcross also includes cross compilers for alternative libc implementations meaning that we could use this to build musl bootloaders for Alpine or Bionic bootloaders for Android if we wanted to. However, they'd be difficult to distribute because PyPI currently only allows glibc Linux binaries.

@bwoodsend
Copy link
Member Author

I did some PyPI stats queries on Google Big Query to see is it's worth shipping wheels for any of these architectures. These are the download stats for the last month. The architectures with an asterisk next to them are the ones supported by manylinux and therefore PyPI.

Architecture Count
x86_64 * 240961
aarch64 * 4095
armv7l 1979
i686 * 319
armv8l 133
ppc64le * 72
mips64 71
s390x * 62
armv6l 30
sw_64 7
ppc64 3
loongarch64 1

So I suggest that we start shipping wheels for aarch64 and maybe i686 for now. Although I'd rather we don't check the bootloaders into the repo like we do with the standard ones.

…ormal architectures.

Add a Dockerfile which leverages the 'dockcross' family of cross compiler base
images to build the Linux bootloaders for almost any architecture.

This required a few preliminary changes:

* The waf script now always assumes it's building for the C compiler's reported
  target architecture if the target OS is Linux. Previously, it was determined
  from the host machine which of course is wrong in a cross compile setup.

* To avoid bootloaders of different architectures clobbering each other if
  present in the same installation or git clone, append an architecture family
  name in the path to the bootloader. e.g. A `ppc64le` (Power PC, 64 bit, little
  endian) bootloader would be saved under
  `PyInstaller/bootloader/Linux-64bit-ppc/run`. This change was already half
  implemented as a special case for ARM and `sw_64`. This change only applies to
  Linux. All other OSs either don't need this change or, in the case of macOS,
  have their own special requirements and handling. This use of a loose *family
  name* for architectures rather than the exact output from `platform.machine()`
  is needed so that waf can predict this name based only on the target reported
  by the C compiler which is considerably less exact than that of `uname -m` or
  `platform.machine()`. For example, when `platform.machine()` is `armv5l`,
  `armv6l` or `armv7h`, the C compiler simply reports `arm` in all cases. The
  mapping should go from `armv7h` to `arm` rather than the other way around.

A couple of unintended bonuses of this approach:

* dockcross provide manylinux2014 compatible compilers (see PEPs 513 and 599) to
  avoid the glibc symbol runtime errors that crop up if you don't keep glibc
  requirements lower than your target's version. This commit can also be used
  to build native `x86_64` binaries using a non-cross variant which conform to
  the manylinux2014 standard (which, as of pyinstaller#5787, is advertised by our wheel
  tags).

* dockcross also includes cross compilers for alternative libc implementations
  meaning that we could use this to build `musl` bootloaders for Alpine or
  `Bionic` bootloaders for Android if we wanted to. However, they'd be difficult
  to distribute because PyPI currently only allows glibc Linux binaries.
@bwoodsend bwoodsend marked this pull request as ready for review July 11, 2021 15:00
@bwoodsend bwoodsend merged commit 018fa3f into pyinstaller:develop Jul 11, 2021
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Nov 16, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants