[linux] publish manylinux builds on pypi

[ssbarnea]

Currently psutil can only be installed on linux by compiling it, which is quite inconvenient for many usecases, as it increases the installation foorprint a lot (python-devel, gcc,...).

psutil should publish manylinux binaries on pypi as this would remove the need for compilation on linux. This approach is already popular among other packages.

[bristea]

@giampaolo Any chance of making this happen?

[tahigash]

@giampaolo no plan to have manylinux binaries? if psutil provides manylinux binaries, it would be very convenient for linux users.

[giampaolo]

There's been progress on that front, me and @Czaki spent around a week working on #1758, #1762 and other intermediate PRs (#1761, #1764, #1767) but unfortunately there are still 2 blockers:

• on Linux Process.rlimit() API is not available because the wheel is compiled on CentOS 6 (which is old), so the resulting wheel does not have that functionality (but the tarball does)
• on Windows 32 bit we get occasional and hard-to-debug segfaults / crashes because of the VS version used by github, which is more recent than AppVeyor's where the problem does not occur. So I'll have to install that newer VS version locally and try to debug the segfault on Windows.

Both of these things require time which I currently don't have unfortunately. Actually I was thinking about doing some sort of kickstarter / funding, because psutil development is becoming too demanding for me.

[Czaki]

From my point of view big advantage is that macos wheel can be produced. Also there is possibility to build manylinux2014 (Centos 7) wheel for linux. So part of person could install it from wheel (ones with older system/pip still will be installing from tar without problems). But I agree that this two problems need to be solved

[giampaolo]

Also there is possibility to build manylinux2014 (Centos 7) wheel for linux

But that doesn't support Python 2.7 if I'm not mistaken, right?

[Czaki]

Also there is possibility to build manylinux2014 (Centos 7) wheel for linux

But that doesn't support Python 2.7 if I'm not mistaken, right?

Yes. But if there is no wheel for python 2.7 pip will install psutil from source, as it is done now.

[gaby]

Any updates on this ?

[Czaki]

@giampaolo I still think that manylinux2014 wheel is good solution.

[giampaolo]

So a little update on this. I have a working patch in which I converted prlimit() C code to ctypes which will get us python 2.x support:
https://github.com/giampaolo/psutil/compare/prlimit
Other than that I'll have to:

1. update the documentation
2. set up an automated workflow (via makefile/scripts) to "sign" wheels before uploading them to PyPI so that MD5 hashes are kept under revision control somehow.

I will do this for the next release.

[giampaolo]

I'm happy to announce that finally, after a lot of work, release 5.8.0 is now out and it includes wheels for Linux (32 and 64 bits) and macOS (64 bits).

Other than the changes to Linux (prlimit rewritten in ctypes) and Windows, the automation tools that made this happen are:
https://github.com/giampaolo/psutil/blob/master/.github/workflows/build.yml
https://github.com/giampaolo/psutil/blob/master/scripts/internal/download_wheels_github.py
https://github.com/giampaolo/psutil/blob/master/scripts/internal/download_wheels_appveyor.py

psutil/Makefile

Line 256 in db8cd0c

pre-release: ## Check if we're ready to produce a new release.

List of available wheels:

windows (total = 10)                                      size    arch   pyver
psutil-5.8.0-cp27-none-win32.whl                        231.0K      32    py27
psutil-5.8.0-cp27-none-win_amd64.whl                    234.3K      64    py27
psutil-5.8.0-cp36-cp36m-win32.whl                       234.7K      32    py36
psutil-5.8.0-cp36-cp36m-win_amd64.whl                   239.1K      64    py36
psutil-5.8.0-cp37-cp37m-win32.whl                       234.7K      32    py37
psutil-5.8.0-cp37-cp37m-win_amd64.whl                   239.1K      64    py37
psutil-5.8.0-cp38-cp38-win32.whl                        235.3K      32    py38
psutil-5.8.0-cp38-cp38-win_amd64.whl                    239.6K      64    py38
psutil-5.8.0-cp39-cp39-win32.whl                        235.8K      32    py39
psutil-5.8.0-cp39-cp39-win_amd64.whl                    240.4K      64    py39

linux (total = 12)                                        size    arch   pyver
psutil-5.8.0-cp27-cp27m-manylinux2010_i686.whl          277.8K      32    py27
psutil-5.8.0-cp27-cp27m-manylinux2010_x86_64.whl        281.0K      64    py27
psutil-5.8.0-cp27-cp27mu-manylinux2010_i686.whl         277.8K      32    py27
psutil-5.8.0-cp27-cp27mu-manylinux2010_x86_64.whl       281.0K      64    py27
psutil-5.8.0-cp36-cp36m-manylinux2010_i686.whl          282.5K      32    py36
psutil-5.8.0-cp36-cp36m-manylinux2010_x86_64.whl        285.0K      64    py36
psutil-5.8.0-cp37-cp37m-manylinux2010_i686.whl          283.5K      32    py37
psutil-5.8.0-cp37-cp37m-manylinux2010_x86_64.whl        289.4K      64    py37
psutil-5.8.0-cp38-cp38-manylinux2010_i686.whl           286.9K      32    py38
psutil-5.8.0-cp38-cp38-manylinux2010_x86_64.whl         289.1K      64    py38
psutil-5.8.0-cp39-cp39-manylinux2010_i686.whl           284.2K      32    py39
psutil-5.8.0-cp39-cp39-manylinux2010_x86_64.whl         286.6K      64    py39

macos (total = 5)                                         size    arch   pyver
psutil-5.8.0-cp27-cp27m-macosx_10_9_x86_64.whl          230.2K      64    py27
psutil-5.8.0-cp36-cp36m-macosx_10_9_x86_64.whl          230.5K      64    py36
psutil-5.8.0-cp37-cp37m-macosx_10_9_x86_64.whl          230.5K      64    py37
psutil-5.8.0-cp38-cp38-macosx_10_9_x86_64.whl           230.8K      64    py38
psutil-5.8.0-cp39-cp39-macosx_10_9_x86_64.whl           230.7K      64    py39

If you wish, consider making a donation. ;)
Regardless, enjoy!

[cinchent]

How to download one of those wheels then? (No links provided.)

If you mean that one must use one of the downloader scripts (e.g., download_wheels_github.py), that fails when used with my GitHub token. (requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://api.github.com/repos/giampaolo/psutil/actions/artifacts)

Installing psutil behind a proxy is a major PITA.

EDIT: https://pypi.org/project/psutil-wheels/#files has a subset of the .whl files you list above, but not the one I need.

[ssbarnea]

@cinchent The official public urls can be seen at https://pypi.org/project/psutil/5.8.0/#files -- github artifacts are temporary and with limited lifespan.

[cinchent]

@ssbarnea Thanks. Yes, the wheel variant I need is present there.