Use rpmio (add support for zstd)

[kontura]

This adds support for rpmio compression modes, it is possible to use --general-compress-type 9.zstd , 9T4.zstd or 7T16.xzdio for parallel (de)compression with createrepo_c.

It does break the ABI of the C library because cr_CompressionType is no longer an enum but instead a const char* and all its values CR_CW_*_COMPRESSION become defined as const char *. However I believe the API is preserved and anything linking with the C library should work after a recompile. This change is required in order to encompass all the possible rpmio modes.

Should help with: #53. #82

The plan is to keep both rpmio and the current compression methods for a while and after some transition period keep just rpmio and zchunk. This would allow to remove quite a lot of code.

[Conan-Kudo]

I spoke to @dmach about this a while back, and I'm not sure if we want to do this. One of the major reasons for createrepo_c's somewhat weird way of handling RPMs (internal definitions of some tags, internal handling for compression algorithms and payload data, etc.) is to avoid the problem where RPM needs to be upgraded or have backports for createrepo_c to be useful for newer distribution targets. This has saved our bacon in Fedora and openSUSE enough times that I'm not sure I would want to accept changing to use librpmio for this. I can easily foresee this biting us again later as RPM continues to change and people are required to use older distributions for production while wanting to support newer distributions over time.

[dralley]

I have to agree with @Conan-Kudo. Pulp / Satellite would inevitably run into the same problems - any significant changes to the metadata may result in installations on older RHEL (or any other OS) being unable to publish the metadata expected by newer RHEL (or any other OS). The support timeframes make it a very difficult problem.

[lukash]

I may not be fully understanding the problem, so bear with me:

From what I can see we can use any compression functions that are present in rpm-libs (as long as they are present in rpm-libs versions on all systems we build createrepo_c for). If we need to add support for a new compression that is not present in rpm-libs on older systems for which we build createrepo_c, we'd need to add our own implementation (or dependency). So we should keep the code in such a way it's possible to add different compression implementations other than rpm, but that's all? Am I missing something?

[dralley]

That's the crux of it, yes

[kontura]

I couldn't get this to work properly because:

• we cannot be completely transparent to the compression since createrepo needs to know appropriate compression suffix but there is not way to get it from rpmio for any arbitrary rpmio compression mode
• there is no straightforward way to validate the rpmio compression modes and when an invalid mode is specified by the user rpmio creates the file normally but doesn't use any compression

Based on this and the concerns raised above I have reduced this PR to just adding zstd support through rpmio. That should make things simpler hopefully.

[kontura]

Also created some ci tests: rpm-software-management/ci-dnf-stack#1029

[dralley]

It would be great if "none" were also an option

[Conan-Kudo]

I'd rather not have rpmio used for compression at all.

[dralley]

As long as it's supported on RHEL8, we can probably live with it. It will cause problems for the binary Python packages though, which is unfortunate. I do agree about preferring not to rely on rpmio but we can see what problems it causes for us in practice and perhaps update the implementation later.

[Conan-Kudo]

The problem is that it's not supported on RHEL 7, and that's where the majority of Foreman+Pulp/Satellite deployments are. And if RPM is not built with zstd support, this will break (which was the case for RHEL 8.0~8.2 and SLE 15 SP0 and SP1).

[dralley]

It partially depends on how fast the uptake is. If it's not actually used for another 2 years then it's less of a problem than if third parties and Fedora start using it immediately (requisite on having a new-enough createrepo_c available to begin with).

But you're right that such assumptions are probably not a good idea.

[Conan-Kudo]

It was a pain working through the crap caused by RHEL stripping out zstd support when forking from Fedora, and then needing to put it back afterward when Fedora wanted to use zstd for RPM payloads. If I don't have to, I don't want to take on an rpmio dependency for this because I definitely don't want this problem again.

[kontura]

I see, ok then I am closing this.

[dralley]

#145 (comment)

[dralley]

@kontura @Conan-Kudo For what it's worth, as of the most recent releases of Foreman / Satellite only EL8+ is supported. I'm not sure if that changes anything for you.

I would be OK with this moving forwards with this as long as there is an understanding that leaning too heavily on rpmio can be problematic in the general case.

[Conan-Kudo]

I don't think it'd be a good idea for us, because it creates unusual constraints for things like Koji and OBS servers too. They are often deployed on the oldest platforms even when builds are being made more newer ones on those systems.

[dralley]

Ack.