feat: add linux aarch64 wheel #2103
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mayeut
force-pushed
the
abi3-linux-aarch64
branch
3 times, most recently
from
ddc8969
to
73a80da
Compare
mayeut
force-pushed
the
abi3-linux-aarch64
branch
from
73a80da
to
a99ad94
Compare
|
@mayeut is the failing test a flaky one which can be re-run, or does this PR need another fix before @giampaolo can review? awesome work btw 🙏 oh and I think |
mayeut
force-pushed
the
abi3-linux-aarch64
branch
from
a99ad94
to
b8c1753
Compare
It was a flaky test. There was also another fix needed since the As mentioned in #2070 (comment),
This is intentional. It won't fix those issues which are related to building from sources. Of course, people are much less likely to hit those since they would probably get a pre-built wheel unless enforcing a build from source. Hence |
mayeut
force-pushed
the
abi3-linux-aarch64
branch
from
b8c1753
to
80d814c
Compare
|
For people eager to test these awesome wheels in the meantime: Using $ export PIP_FIND_LINKS='https://github.com/ddelange/psutil/releases/expanded_assets/release-5.9.1' # space separated links
$ pip install --no-cache-dir psutil==5.9.1
Collecting psutil~=5.9
Downloading https://github.com/ddelange/psutil/releases/download/release-5.9.1/psutil-5.9.1-cp36-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (271 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 271.3/271.3 KB 22.2 MB/s eta 0:00:00As this is dynamic, definitely unsafe for production. I recommend forking to have some immutability. |
mayeut
force-pushed
the
abi3-linux-aarch64
branch
from
80d814c
to
997ab3c
Compare
mayeut
force-pushed
the
abi3-linux-aarch64
branch
from
997ab3c
to
e99529c
Compare
|
@giampaolo could you please review this awesome contribution? |
mayeut
force-pushed
the
abi3-linux-aarch64
branch
2 times, most recently
from
19490c3
to
ba013d8
Compare
This comment was marked as outdated.
This comment was marked as outdated.
mayeut
force-pushed
the
abi3-linux-aarch64
branch
2 times, most recently
from
4e4d7bc
to
ca86104
Compare
mayeut
force-pushed
the
abi3-linux-aarch64
branch
from
ca86104
to
5877dce
Compare
|
Whats the progress for this? Seems my docker image won't compile even after installing |
|
as long as this PR is not merged and released, pip will get the source distribution and not the arm64 wheel that this PR will add to the build pipeline. hotfix in my previous comment above |
alas, this didn't seem to work: I guess there is nothing to do but wait? |
|
for completeness this is what I ran: |
|
nice article explaining issue: https://pythonspeed.com/articles/docker-build-problems-mac/ |
|
@mayeut would it make sense to further future proof the CI by setting these? CIBW_MANYLINUX_X86_64_IMAGE: manylinux_2_28
CIBW_MANYLINUX_AARCH64_IMAGE: manylinux_2_28
CIBW_MANYLINUX_I686_IMAGE: manylinux_2_28This will produce even more forward-compatible wheels, built on a RHEL8.6 (instead of RHEL7) docker image ( edit: hmm, on second thought, from reading this, i think it might reduce backward compatibility (despite producing a wheel like |
Indeed it would reduce backward compatibility. I've kept the default image in the PR which builds on manylinux2014 & produces manylinux2010 compatible wheels (thus going back to CentOS/RHEL/OracleLinux 6). |
mayeut
force-pushed
the
abi3-linux-aarch64
branch
from
5877dce
to
99357b2
Compare
mayeut
force-pushed
the
abi3-linux-aarch64
branch
2 times, most recently
from
de29694
to
32934ed
Compare
This commit updates the build workflow to use the latest cibuildwheel as a GitHub Action. cibuildwheel configuration is now in its own file (as there's no `pyproject.toml` yet) Signed-off-by: mayeut <mayeut@users.noreply.github.com>
Signed-off-by: mayeut <mayeut@users.noreply.github.com>
`macos-10.15` runner is deprecated. Signed-off-by: mayeut <mayeut@users.noreply.github.com>
Signed-off-by: mayeut <mayeut@users.noreply.github.com>
Signed-off-by: mayeut <mayeut@users.noreply.github.com>
Signed-off-by: mayeut <mayeut@users.noreply.github.com>
mayeut
force-pushed
the
abi3-linux-aarch64
branch
from
32934ed
to
0daf018
Compare
|
@giampaolo can you please review? |
|
Hello. |
|
Hi @giampaolo 👋 Appreciate you checking in. I don't know @mayeut, but I have seen him putting a lot of time and effort into this PR over the past 4 (!) months. To me your comment is disappointing, I can imagine it similar for @mayeut.
I think the features are nicely separated, the biggest part of the diff comes in #2102: switching to limited ABI (only two tiny changes in
With 60+ million downloads per month, this package comes in very high in the 'hall of fame'. I completely understand your effort considerations, and wouldn't want you to get overloaded due to this change. I think, however, that for such an essential package there are enough professional developers out there for whom it would not be a lot of work to handle some aarch64 bugfixes in their spare/allocated OSS time. So if you don't have time for such fixes, which I understand, there will be a contributor sooner or later that will take it over (this effort from @mayeut as a beautiful example). I'm afraid that waving off aarch64 so quickly isn't that easy in reality: as the PR description points out, aarch64 has already surpassed windows in terms of downloads for this package. Oracle, Amazon, Google, and Microsoft are all offering aarch64 cloud instances at an undeniable price point compared to amd/intel, so the demand will undoubtedly only grow. As for me, I only have experience with Amazon EC2, and know that an install from source on their Graviton processors will fail without further config when there are no wheels available. That's something that many devs will run into during their regular day at work. There is a lot of value in this PR. Please consider merging and releasing, for Python community's sake 🙏 |
Unfortunately that's not how it works, mostly because binaries are directly related to releases, and those are up to me. It’s not about reviewing and merging one PR. It’s about the additional and continuous work required after a PR like this gets merged. CI builds / binaries distribution is that part of the project infrastructure which requires a constant maintenance over time, because OS and python versions constantly change, and so does GitHub infrastructure / config matrix, which keeps breaking with time. Once wheels for a new platform are offered, you basically are forced to keep supporting them. I may change my mind in the future, but this is not sustainable for me at the moment. I may have some ideas on how to try make this sustainable in the long run, but I need time to think about it. |
|
It saddens me to see this PR get closed like that, @giampaolo. Let me try to advocate one more time with regards to your last comment: because CI already tests the binaries right after creating them, and because green CI is required to merge PRs, I don't think you'll run into the drift issues you describe as long as you make a release relatively soon after the last PR went in. However, I can relate to your prudence as person responsible for releases. Maybe CI on a cron job for In the meantime, the show must go on. To offer the community an alternative to this dead end, and to not let @mayeut's efforts be in vain: my fork includes the commits from this PR on export PIP_FIND_LINKS=https://ddelange.github.io/psutil/
pip install ...
# OR
pip install ... --find-links=https://ddelange.github.io/psutil/ |
|
@giampaolo, thanks for your detailed answers. While I'm a bit disappointed, I do appreciate the position you'd be in w.r.t. to maintaining if it got merged in.
I'll clean the other PRs with that in mind. |
|
I trued @mayeut fork and can't install it on NVIDIA Jetson Nano B01 with Python3.8.0 (the OS came with Python2 and Python3.6.9 but anomalib require Python3.8, so I use Python3.8.0) What i've did: |
|
Thanks for the info. I will definitely try python3.8-dev. Previously I install python3.8. I use I will use |
1 task
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
fixes Generate Wheels for CPython 3.11 (or Use Limited API) #2089
fixes [M1 macOS BigSur] no suitable image found #1954
fixes [Linux] Publish armv7l & aarch64 wheels #1966
fixes [Linux][aarch64] Wheel support for aarch64 Linux #1972
closes Build universal2 wheel for Apple Silicon support #2021
closes chore: update cibuildwheel #2040
closes Build wheels for arm64 on linux/mac #2070
closes feature: use ABI3 for cp36+ #2102
mitigates [macOS Apple Silicon] Installation fails due to compiler errors #1945
mitigates [AWS Graviton2] graviton gcc flags break pip install #2090
Description
Build Linux aarch64 wheels.
Linux aarch64 downloads now represents around 8% of downloads. Having binary wheels will help users to install psutil as can be seen in multiple issues.
This PR builds on top of #2102 :
Use Limited API when building with CPython 3.6+ on Linux/macOS/Windows.
This allows to use pre-built wheels in all future versions of CPython 3.
Windows builds for CPython 3.6+ are moved from AppVeyor to GitHub Actions.
This PR builds on top of #2040 :
Updates the build workflow to use the latest cibuildwheel as a GitHub Action and adds a universal2 wheel for macOS.
cibuildwheel configuration is now in its own file (as there's no pyproject.toml yet).
Diff of wheels produced: