wheels on PyPI

[carlwgeorge]

I'm using the createrepo_c Python bindings in a project of mine. For the CI test run I'm using the python:3.7 container image. I'd like to be able to pip install createrepo_c, but this fails due to multiple missing system libraries. I can work around this by installing them with apt first (at a minimum cmake, libmagic-dev, and librpm-dev), but I believe if a wheel package of createrepo_c was uploaded to PyPI it would just work.

Can a createrepo_c wheel be uploaded to PyPI?

[hroncok]

Note that the createrepo-c PyPI package appears to be maintained by pulp maintainers.

[carlwgeorge]

/me checks pulp/pulp_rpm commit history for recently active maintainers...

@goosemania @ipanova @daviddavis @dralley Would y'all be able to help here?

[dralley]

Hello @carlwgeorge

Currently uploading a binary wheel of createrepo_c to PyPI is blocked by some changes in the Python build infrastructure pypa/manylinux#279

Since compiled packages link against system libraries, and the system libraries are different on different systems, Python created a build environment called "manylinux" and then built tooling that wraps up all the dynamically-linked libraries into the wheel so that it doesn't depend on any system-provided libraries. In order to upload a binary wheel to PyPI, it has to have been created in this build environment because otherwise a package built on Ubuntu and uploaded would not work on CentOS.

Unfortunately, that common build environment is based on CentOS 5 and createrepo_c won't build on CentOS 5. The new environment discussed in the PR above is based on CentOS 6 and that will be usable. But it's taking a really, really long time for them to get it ready.

That's why, for now, all we've been able to do is upload a source package and have our CI install all of the build dependencies just like you're doing.

https://github.com/pulp/pulp_rpm/blob/master/.travis/install.sh#L44-L46

[carlwgeorge]

Thanks for the clarification @dralley. I am vaguely familiar with manylinux, but I didn't know that it was still building against EL5. That's understandable that a createrepo_c wheel is blocked on the build environment getting refreshed to EL6.

[dralley]

Yup, sorry I couldn't be more helpful! It was disappointing to us too when we realized that.

[carlwgeorge]

Hey, having a clear understanding of the blocker is absolutely helpful. Thanks!

[hroncok]

but I didn't know that it was still building against EL5

manylinux1 will always build against EL5.

manylinux2010 (imagine "manylinux2", but the naming schema has changed) will always build against EL6.

[dralley]

Also, once that environment does come out, there may be added complications since CentOS 6 doesn't have zchunk or libmodulemd packages available. Hopefully they can at least be compiled on CentOS 6. If not, I'm not really sure what the solution is...

https://pkgs.org/download/zchunk-libs
https://pkgs.org/download/libmodulemd

As it stands, these new deps make it (virtually?) impossible to build a fully-featured createrepo_c Python package from source on any Debian-based distro.

[carlwgeorge]

Since those exist in Fedora (and aren't in RHEL) it's just a matter of requesting the respective Fedora maintainers add an el6 branch and build it for EPEL 6. Of course the dependencies will need to be worked out too. Right off the bat I see that meson and gobject-introspection will be needed, and probably a few others.

[sgallagher]

@dralley just pointed me at this ticket.

So, libmodulemd cannot realistically be built on RHEL/EPEL 6. It needs a far newer glib2 and gobject-introspection than will ever be available on that platform. (And we can't ship newer ones in EPEL for a variety of reasons, not least of which being that they would conflict with or replace the official RHEL packages and in so doing cause lots of chaos).

If this is a significant-enough blocker to justify it, we could look into producing a version of libmodulemd that doesn't provide GObject Introspection bindings and statically-links the libyaml and glib2 dependencies, but that's a big undertaking.

[hroncok]

If this was actually desired and some effort put into this, conflicts with RHEL or EPEL wouldn't really matter. A copr repo (for example) would do.

[hroncok]

As an example, with admesh, I just build my own EL5 RPM:

https://github.com/admesh/python-admesh/tree/v0.98.9/travis
https://github.com/admesh/python-admesh/blob/027f9fe1922d7aada4e032632fcfcb1a4bde5f76/.travis.yml#L48

This is of course much easier, because it is not a full stack of dependencies but just one tiny library.

[carlwgeorge]

@sgallagher

• gobject-introspection isn't in RHEL6, so why couldn't it be added to EPEL6?
• What is the minimum version of glib2 needed to build libmodulemd? The libmodulemd spec file build requires pkgconfig(gobject-2.0), which is provided in RHEL6 by glib2-devel-2.28.8-10.el6. If a newer version is necessary it would be preferable to build require at least that version.
• For libyaml, the build requirement is pkgconfig(yaml-0.1). That isn't provided by RHEL6's libyaml-devel, but that could just be a packaging oversight as the difference between RHEL6 and RHEL7 here is just 0.1.3 to 0.1.4.

[carlwgeorge]

I don't need libmodulemd or zchunk support for my use case. I just want pip to be able to install a createrepo_c wheel with basic functionality.

[carlwgeorge]

Also related, I tried to just use the system python3-createrepo_c package on Fedora, but it's missing metadata so pip can't see it.

https://bugzilla.redhat.com/show_bug.cgi?id=1695677

[sgallagher]

I don't need libmodulemd or zchunk support for my use case. I just want pip to be able to install a createrepo_c wheel with basic functionality.

Define "basic functionality"? Because I'm working on the libmodulemd support right now with the plan that if you encounter a repo with module metadata but haven't built with libmodulemd support, it's going to refuse to operate on that repo.

[carlwgeorge]

For me, "basic functionality" means being able to create repomd.xml, primary.xml.gz, filelists.xml.gz, and other.xml.gz for a directory of RPMs. Checking my code, these are the attributes I use from the Python bindings:

• package_from_rpm
• Repomd
• RepomdRecord
• PrimaryXmlFile
• FilelistsXmlFile
• OtherXmlFile

[sgallagher]

@sgallagher

• gobject-introspection isn't in RHEL6, so why couldn't it be added to EPEL6?

gobject-introspection is fairly tightly coupled with glib2. (They don't always rev together, but quite often. The current version, 1.6.0 requires glib2 >= 2.58.0.

So for whatever version of GI we want to include, we're going to need a recent-enough glib2 to match. Which means either replacing the system copy, statically linking, or doing a mess of work to get GI and libmodulemd to point at a renamed/relocated library.

• What is the minimum version of glib2 needed to build libmodulemd? The libmodulemd spec file build requires pkgconfig(gobject-2.0), which is provided in RHEL6 by glib2-devel-2.28.8-10.el6. If a newer version is necessary it would be preferable to build require at least that version.

Honestly, I don't know. It builds successfully against glib2-2.54 at least (which was the earliest one I tried). I'm not sure how far back I could go and still have it work properly.

• For libyaml, the build requirement is pkgconfig(yaml-0.1). That isn't provided by RHEL6's libyaml-devel, but that could just be a packaging oversight as the difference between RHEL6 and RHEL7 here is just 0.1.3 to 0.1.4.

Yeah, that's a packaging issue that we could solve in EPEL (create libyaml-pkgconfig whose sole purpose is to provide that file missing from RHEL and Requires: libyaml-devel).

[carlwgeorge]

Thanks for the insight on gobject-introspection/glib2. The earliest reference I can find to a minimum version is when GI 1.29.0 added a requirement of glib2 2.29.7, which is still too new for EL6's 2.28.8.

https://gitlab.gnome.org/GNOME/gobject-introspection/commit/fe01f575b0328107e64fb0c1f2d40d35e6e669cf

That might be a dead end for libmodulemd support, so hopefully createrepo_c will retain a flag to optionally disable that feature. That would allow a createrepo_c wheel (without libmodulemd support) to be uploaded to PyPI once manylinux2010 is finished.

[dralley]

That might be a dead end for libmodulemd support, so hopefully createrepo_c will retain a flag to optionally disable that feature. That would allow a createrepo_c wheel (without libmodulemd support) to be uploaded to PyPI once manylinux2010 is finished.

I don't know, the fact that it would fail on any repository containing modules almost seems as though it's too crippled to ship on PyPI under the same name. In any case, we do need the libmodulemd functionality, so a package like that wouldn't help us, and I don't think we could then justify the effort to set up the (not completely trivial) infrastructure needed to build those wheels (that we can't use) for each release.

I think the ability to produce Python packages is still useful even if they can't be shipped on PyPI or work generically on any kind of distro, though.

@carlwgeorge, Could you build such a package locally, and upload it to some form of file storage, and then just pip install http://..../createrepo_c-12.0.whl or something along those lines inside your CI environment. That would eliminate the need to compile inside your CI environment. And of course, you could keep compiling the source package with any particular combination of command line arguments you need.

See: https://github.com/pulp/pulp_rpm/blob/master/.travis/install.sh#L46

Or: if the issue with the python3-createrepo_c metadata was resolved, that would also solve your problem?

[carlwgeorge]

Yes, if rhbz#1695677 is resolved I'll just use a Fedora container for my CI and and use the system python3-createrepo_c package. Trying to do that now still attempts to install createrepo_c from PyPI because of the missing metadata.

[sgallagher]

So, I got nerd-sniped here and ended up seeing how far I could get towards making libmodulemd build against EPEL 6. Turns out, it was pretty far (thanks, SCLs!), but I've finally discovered the minimum version of glib2 I need: It's 2.44. I'm going to stop here and say that EPEL 6 isn't achievable.

[dralley]

@sgallagher Thanks for providing that confirmation!

[jdieter]

So, I realize you said zchunk wasn't important for your use-case, but I just took a look at what it would take to get it to compile on CentOS 6, and the main missing dependency is meson. zchunk itself has very few dependencies, just libcurl, openssl and zstd, and all are in CentOS 6. On the other hand, I quite like meson as a build system, and I'm not sure that I'm ready to switch to something like autotools just for CentOS 6 compatibility. I don't suppose anyone's worked out a way to get meson to run on CentOS 6?

[hroncok]

You should be able to pip install meson, if you target Python 3.5+.

[carlwgeorge]

It will need ninja-build first, which I requested earlier. rhbz#1695759

Please excuse my accidental markdown in bugzilla. 😬

[sgallagher]

I have a COPR of meson 0.44.1, which is the newest that will build on EPEL 6 (0.45 and later require at least Python 3.5). It might be possible to do something around SCLs though.

[jdieter]

I have a COPR of meson 0.44.1, which is the newest that will build on EPEL 6 (0.45 and later require at least Python 3.5). It might be possible to do something around SCLs though.

Thanks @sgallagher, I installed your COPR and meson and ninja work fine, but now I'm getting loads of compilation errors, I'm assuming because gcc is so old:

../src/lib/zck_private.h:92: error: redefinition of typedef 'zckCtx'
include/zck.h:49: note: previous declaration of 'zckCtx' was here
../src/lib/zck_private.h:106: error: redefinition of typedef 'zck_log_type'
include/zck.h:47: note: previous declaration of 'zck_log_type' was here

I'd be happy to dig into this more.

[sgallagher]

On Wed, Apr 3, 2019 at 5:13 PM Jonathan Dieter ***@***.***> wrote: I have a COPR of meson 0.44.1, which is the newest that will build on EPEL 6 (0.45 and later require at least Python 3.5). It might be possible to do something around SCLs though. Thanks @sgallagher <https://github.com/sgallagher>, I installed your COPR and meson and ninja work fine, but now I'm getting loads of compilation errors, I'm assuming because gcc is so old: ../src/lib/zck_private.h:92: error: redefinition of typedef 'zckCtx' include/zck.h:49: note: previous declaration of 'zckCtx' was here ../src/lib/zck_private.h:106: error: redefinition of typedef 'zck_log_type' include/zck.h:47: note: previous declaration of 'zck_log_type' was here I'd be happy to dig into this more.

On CentOS 6, you can `yum install centos-release-scl` and then `yum install devtoolset-8` for a recent GCC

…

[jdieter]

On CentOS 6, you can yum install centos-release-scl and then yum install devtoolset-8 for a recent GCC

That worked and passed the test suite. So, zchunk can be compiled on CentOS 6, as long as you're willing to use a COPR and an SCL.

If meson and ninja get into EPEL 6, I'd be happy to figure out how to make zchunk compile with gcc-4.4.

[wohali]

@dralley Hello, it appears pypa/auditwheel#230 has been fixed. Does that unblock merging your fixes for this?

[dralley]

In the sense that it builds, yes, in the sense that it works as intended, no.

pypa/manylinux#823

tl;dr createrepo_c uses the "magic number" database located at /usr/share/misc/magic to determine the types of various kinds of files. It uses libmagic to parse that database. When you build a cross-platform Python wheel, it bundles all the libraries that it links against into the package, including libmagic.

(libmagic) https://github.com/file/file

The old version that gets bundled (from CentOS 7) can't parse the /usr/share/misc/magic databases for more recent linux distros, so the createrepo_c binary wheel works on older distros, but it doesn't work on newer distros.

[dralley]

There are really only a few possible solutions, and I'm not really motivated to push any of them:

• don't use libmagic - Probably not possible since compression_wrapper.c seems to use it quite a lot.
• get the Python packaging folks to make an dynamic linking exception for libmagic the way they do for glibc. I have very low expectations that they would do that.
• I have no idea what the libmagic story around "forwards compatibility is" but obviously this years-old version doesn't have it. Maybe new ones do and it could be linked into the weel?

[wohali]

In that case, what is your recommended "best approach" to installing this on an arbitrary system? Is it python setup.py bdist_wheel and pip install dist/whatever? Or would you just suggest cmake && make && make install?

[dralley]

If the createrepo_c build dependencies are installed on all of the systems you want to install it on pip install createrepo-c will compile the package and install it transparently.

If you build a wheel on one system and then install it on another system, it might work or might not work but it depends entirely on the details. It's been a while since I tried it, I don't remember the details.

[kontura]

* don't use libmagic - Probably not possible since  `compression_wrapper.c` seems to use it quite a lot.

Incidentally I am already considering this as a part of #251 I want to replace parts of cr_detect_compression (the part that uses libmagic) by rpmFileIsCompressed in order to support the rpmio compressions more transparently.

rpmFileIsCompressed uses its own hardcoded magic bytes so that could fix this issue if its the only problem left.

[dralley]

Wow, great news then!

I don't know for certain that it's the only problem remaining, but I can't think of anything else that would prevent it from working. It's not so dissimilar to libcomps and the manylinux wheels for that package works fine.

[dralley]

@lukash Whoops, we should reopen this

[dralley]

Incidentally I am already considering this as a part of #251 I want to replace parts of cr_detect_compression (the part that uses libmagic) by rpmFileIsCompressed in order to support the rpmio compressions more transparently.

rpmFileIsCompressed uses its own hardcoded magic bytes so that could fix this issue if its the only problem left.

Setting aside the debate about rpmio, hardcoding the magic bytes would be pretty straightforwards regardless.

Gzip => 0x1f, 0x8b
Bzip => 0x42, 0x5a
Zstd => 0x28, 0xb5, 0x2f, 0xfd
Lzma => 0xfd, 0x37, 0x7a, 0x58, 0x5a

[dralley]

Honestly it would probably be fine to vendor the whole function. Apart from new compression types being supported most of it is untouched for 11 / 14 years according to git blame.