Ability to install `armv7l manylinux/musllinux` wheels on `armv8l` #690

mayeut · 2023-04-23T11:40:58Z

aarch64 systems running under linux32 emulation will report an armv8l machine rather than armv7l.
This shall not prevent packaging to report musllinux/manylinux armv7l compatibiility as long as the running python has a compatible ABI.

fix #476 (partial)

Some choices were made based on what PyPI accepts today to get a first level of support. A more generic approach would require to refactor some things but would probably be doable as well. I'll comment a bit more inline.

src/packaging/tags.py

mayeut · 2023-04-23T11:54:02Z

src/packaging/_manylinux.py

+    if arch == "armv8l":
+        # armv8l wheels are not accepted on PyPI
+        # As long as we pass the the ABI check below,
+        # the armv7l wheels can be installed.
+        arch = "armv7l"


This boils down to the fact that PyPI only accepts armv7l for now.
A more future proof way to do this would be to refactor _have_compatible_abi to return a list of architectures and yield multiple arch per manylinux version.

Is there any reason PyPI has that restriction? Or should we ask PyPI to update to accept the newer architecture?

For reference, the architectures accepted by PyPI are defined there : https://github.com/pypi/warehouse/blob/f2b3ed3724b064fab8621960fc8658a867b95e1e/warehouse/forklift/legacy.py#L95-L149

Those architectures were defined through the various PEPs for manylinux/musllinux, they are more or less equivalent to what distros offers, with, in this case, armv7l==armhf.

While there's nothing preventing PyPI from accepting other architectures, except maybe going through a new PEP, IMHO, it's not worth adding armv8l there.

I think the demand for 32-bit ARM wheels is most likely linked to armv7/armhf wheels availability for platforms like RaspberryPI or other embedded systems (e.g. OpenWRT based ones for musllinux).

When it comes to building/testing such armv7l wheels, there are multiple routes for CI:

use such an armv7 system but then, you're on your own to manage a self-hosted CI node

use qemu which is painfully slow

use an aarch64 system in 32-bit mode, I only tested Travis CI AWS Graviton 2 nodes in 32-bit mode for now but it should also be applicable to at least CirrusCI, Circle CI both using AWS Graviton 2 and the future GitHub-Hosted aarch64 runners.

For that last route, being able to install armv7l is important, publishing armv8l wheels probably less so.

src/packaging/_musllinux.py

brettcannon

I think it would be better to yield armv8l tags followed by armv7l tags since the former is fully compatible with the latter. That way there's no data loss as well as being able to use the wheels uploaded to PyPI.

src/packaging/_manylinux.py

brettcannon · 2023-06-16T22:56:08Z

src/packaging/_manylinux.py

@@ -204,6 +204,11 @@ def _is_compatible(name: str, arch: str, version: _GLibCVersion) -> bool:


 def platform_tags(linux: str, arch: str) -> Iterator[str]:
+    if arch == "armv8l":
+        # armv8l wheels are not accepted on PyPI


Suggested change

# armv8l wheels are not accepted on PyPI

# armv8l wheels are not accepted on PyPI.

src/packaging/_musllinux.py

src/packaging/tags.py

brettcannon

LGTM, but there's unfortunately a merge conflict due to me accepting another of your PRs before getting to this one 😅.

brettcannon · 2023-07-07T21:43:43Z

src/packaging/_manylinux.py

        return _is_linux_i686(executable)
-    return arch in {"x86_64", "aarch64", "ppc64", "ppc64le", "s390x"}
+    return len(set(archs) & {"x86_64", "aarch64", "ppc64", "ppc64le", "s390x"}) > 0


Suggested change

return len(set(archs) & {"x86_64", "aarch64", "ppc64", "ppc64le", "s390x"}) > 0

return any(arch in {"x86_64", "aarch64", "ppc64", "ppc64le", "s390x"} for arch in archs)

aarch64 systems running under linux32 emulation will report an armv8l machine rather than armv7l. This shall not prevent packaging to report musllinux/manylinux armv7l compatibiility as long as the running python has a compatible ABI.

mayeut · 2023-07-08T07:48:41Z

@brettcannon,
I rebased on main to fix the merge conflict & applied your last review suggestion.

brettcannon · 2023-07-14T18:47:01Z

Thanks for your patience with this PR!

This PR contains the following updates: | Package | Update | Change | |---|---|---| | [packaging](https://togithub.com/pypa/packaging) | minor | `==23.1` -> `==23.2` | --- ### Release Notes <details> <summary>pypa/packaging (packaging)</summary> ### [`v23.2`](https://togithub.com/pypa/packaging/releases/tag/23.2) [Compare Source](https://togithub.com/pypa/packaging/compare/23.1...23.2) #### What's Changed - parse_marker should consume the entire source string by [@mwerschy](https://togithub.com/mwerschy) in [pypa/packaging#687 - Create a Security Policy file by [@joycebrum](https://togithub.com/joycebrum) in [pypa/packaging#695 - Add python 3.12 to CI by [@mayeut](https://togithub.com/mayeut) in [pypa/packaging#689 - Remove URL validation from requirement parsing by [@uranusjr](https://togithub.com/uranusjr) in [pypa/packaging#684 - Add types for packaging.version.\_Version by [@hauntsaninja](https://togithub.com/hauntsaninja) in [pypa/packaging#665 - Add PyPy 3.10 to CI by [@mayeut](https://togithub.com/mayeut) in [pypa/packaging#699 - Remove unused argument in `_manylinux._is_compatible` by [@mayeut](https://togithub.com/mayeut) in [pypa/packaging#700 - Canonicalize names for requirements comparison by [@astrojuanlu](https://togithub.com/astrojuanlu) in [pypa/packaging#696 - Add platform tag support for LoongArch by [@loongson-zn](https://togithub.com/loongson-zn) in [pypa/packaging#693 - Ability to install `armv7l manylinux/musllinux` wheels on `armv8l` by [@mayeut](https://togithub.com/mayeut) in [pypa/packaging#690 - Include CHANGELOG.rst in sdist by [@astrojuanlu](https://togithub.com/astrojuanlu) in [pypa/packaging#704 - Update pyupgrade to Python 3.7+ by [@fangchenli](https://togithub.com/fangchenli) in [pypa/packaging#580 - Fix version pattern pre-releases by [@deathaxe](https://togithub.com/deathaxe) in [pypa/packaging#705 - Fix typos found by codespell by [@DimitriPapadopoulos](https://togithub.com/DimitriPapadopoulos) in [pypa/packaging#706 - Support enriched metadata by [@brettcannon](https://togithub.com/brettcannon) in [pypa/packaging#686 - Correct rST syntax in CHANGELOG.rst by [@atugushev](https://togithub.com/atugushev) in [pypa/packaging#709 - fix: platform tag for GraalPy by [@mayeut](https://togithub.com/mayeut) in [pypa/packaging#711 - Document that this library uses a calendar-based versioning scheme by [@faph](https://togithub.com/faph) in [pypa/packaging#717 - fix: Update copyright date for docs by [@garrypolley](https://togithub.com/garrypolley) in [pypa/packaging#713 - Bump pip version to avoid known vulnerabilities by [@joycebrum](https://togithub.com/joycebrum) in [pypa/packaging#720 - Typing annotations fixed in version.py by [@jolaf](https://togithub.com/jolaf) in [pypa/packaging#723 - parse\_{sdist,wheel}\_filename: don't raise InvalidVersion by [@SpecLad](https://togithub.com/SpecLad) in [pypa/packaging#721 - Fix code blocks in CHANGELOG.md by [@edmorley](https://togithub.com/edmorley) in [pypa/packaging#724 #### New Contributors - [@mwerschy](https://togithub.com/mwerschy) made their first contribution in [pypa/packaging#687 - [@joycebrum](https://togithub.com/joycebrum) made their first contribution in [pypa/packaging#695 - [@astrojuanlu](https://togithub.com/astrojuanlu) made their first contribution in [pypa/packaging#696 - [@loongson-zn](https://togithub.com/loongson-zn) made their first contribution in [pypa/packaging#693 - [@fangchenli](https://togithub.com/fangchenli) made their first contribution in [pypa/packaging#580 - [@deathaxe](https://togithub.com/deathaxe) made their first contribution in [pypa/packaging#705 - [@DimitriPapadopoulos](https://togithub.com/DimitriPapadopoulos) made their first contribution in [pypa/packaging#706 - [@atugushev](https://togithub.com/atugushev) made their first contribution in [pypa/packaging#709 - [@faph](https://togithub.com/faph) made their first contribution in [pypa/packaging#717 - [@garrypolley](https://togithub.com/garrypolley) made their first contribution in [pypa/packaging#713 - [@jolaf](https://togithub.com/jolaf) made their first contribution in [pypa/packaging#723 - [@SpecLad](https://togithub.com/SpecLad) made their first contribution in [pypa/packaging#721 - [@edmorley](https://togithub.com/edmorley) made their first contribution in [pypa/packaging#724 **Full Changelog**: pypa/packaging@23.1...23.2 </details> --- ### Configuration 📅 **Schedule**: Branch creation - "before 4am on the first day of the month" (UTC), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://togithub.com/renovatebot/renovate).

mayeut commented Apr 23, 2023

View reviewed changes

src/packaging/tags.py Show resolved Hide resolved

mayeut commented Apr 23, 2023

View reviewed changes

src/packaging/_musllinux.py Outdated Show resolved Hide resolved

brettcannon added the packaging.tags label May 3, 2023

mayeut force-pushed the armv8l branch from fa579e3 to 2066902 Compare June 3, 2023 11:31

brettcannon requested changes Jun 16, 2023

View reviewed changes

mayeut force-pushed the armv8l branch 2 times, most recently from fdcf5dc to 377b114 Compare June 24, 2023 10:08

mayeut requested a review from brettcannon June 25, 2023 14:21

brettcannon requested changes Jul 7, 2023

View reviewed changes

mayeut added 2 commits July 8, 2023 09:17

generate both armv8l and armv7l tags for 32-bit python on aarch64

f2bdb76

mayeut force-pushed the armv8l branch from 377b114 to f2bdb76 Compare July 8, 2023 07:36

Apply review suggestion

06df33b

mayeut requested a review from brettcannon July 11, 2023 12:01

brettcannon approved these changes Jul 14, 2023

View reviewed changes

brettcannon merged commit 1277904 into pypa:main Jul 14, 2023
30 checks passed

mayeut deleted the armv8l branch July 17, 2023 22:40

messense mentioned this pull request Aug 4, 2023

Generate correct wheel name for armv8l PyO3/maturin#1289

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ability to install `armv7l manylinux/musllinux` wheels on `armv8l` #690

Ability to install `armv7l manylinux/musllinux` wheels on `armv8l` #690

mayeut commented Apr 23, 2023

mayeut Apr 23, 2023

brettcannon May 3, 2023

mayeut May 13, 2023

brettcannon left a comment

brettcannon Jun 16, 2023

brettcannon left a comment

brettcannon Jul 7, 2023

mayeut commented Jul 8, 2023

brettcannon commented Jul 14, 2023

	# armv8l wheels are not accepted on PyPI
	# armv8l wheels are not accepted on PyPI.

	return len(set(archs) & {"x86_64", "aarch64", "ppc64", "ppc64le", "s390x"}) > 0
	return any(arch in {"x86_64", "aarch64", "ppc64", "ppc64le", "s390x"} for arch in archs)

Ability to install armv7l manylinux/musllinux wheels on armv8l #690

Ability to install armv7l manylinux/musllinux wheels on armv8l #690

Conversation

mayeut commented Apr 23, 2023

mayeut Apr 23, 2023

Choose a reason for hiding this comment

brettcannon May 3, 2023

Choose a reason for hiding this comment

mayeut May 13, 2023

Choose a reason for hiding this comment

brettcannon left a comment

Choose a reason for hiding this comment

brettcannon Jun 16, 2023

Choose a reason for hiding this comment

brettcannon left a comment

Choose a reason for hiding this comment

brettcannon Jul 7, 2023

Choose a reason for hiding this comment

mayeut commented Jul 8, 2023

brettcannon commented Jul 14, 2023

Ability to install `armv7l manylinux/musllinux` wheels on `armv8l` #690

Ability to install `armv7l manylinux/musllinux` wheels on `armv8l` #690