implementation of conda_2_28 (aka centos 8)

[beckermr]

This issue tracks centos 8 implementation PRs

• build sysroots from alma 8
  • This one is subtle since we need to weigh down the newer sysroots on ppc64le / aarch64 / s390x using track_features.
  • We likely will also need the current repodata hack package on ppc64le / aarch64 / s390x.
  • To use the curr repodata hack package, we'll need to patch the cos7 sysroots to depend on it.
  • rebuild current cos7 sysroots with current repodata hack (ENH add current repodata hack on non-x86_64 for 2.17 in prep for 2.28 linux-sysroot-feedstock#46)
  • submit repodata patch to add current repodata data hack to cos7 sysroots (ENH add repodata patch to make sure 2.17 sysroots are in current repo… conda-forge-repodata-patches-feedstock#441)
  • add new sysroots with track_features (ENH glibc 2.28 from alma8 linux-sysroot-feedstock#47)
• build docker images (ENH Add AlmaLinux 8 based image docker-images#235)
• build CDTs from alma 8
  • make sure cdt_name is conda_2_28
  • see issue here for deciding what we build, maybe nothing! (figure out which CDTs we can drop for cos8, maybe drop them all? cdt-builds#66)
• adjust staged recipes for alma8
• adjust pinnings so that
  • default image is alma8
• test compilers

The alma 8 repo is https://repo.almalinux.org/almalinux/8.7/BaseOS/x86_64/os/Packages/ etc.

[jaimergp]

xrefs:

• CentOS 8 #1432
• This week's meeting notes

[jaimergp]

What I understood was:

• We will build the new sysroots from Alma 8
• Alma 8 happens to use Glibc 2.28
• manylinux_2_28 is also derived from Alma 8, so that's the match?
• Anaconda versions their sysroots with the glibc version, and we do too?
• The cosN (CentOs version N) prefixes we had in some places will be replaced by conda_X_YZ (with X.YZ being the glibc version)

And maybe I am missing something else, but we can probably merge the discussion with the CentOS 8 thread mentioned in my comment above.

[beckermr]

Someone, I think @isuruf, mentioned listing versions for other things. libm maybe?

[h-vetinari]

The cosN (CentOs version N) prefixes we had in some places will be replaced by conda_X_YZ (with X.YZ being the glibc version)

Manylinux uses glibc major.minor for its versioning, and I think it's good to match that (I think in the core call the mood was that conda_2_28 would be the best option).

That said, not least after the ill-fated Debian-based manylinux_2_24, in effect it's been just a way to encode the RHEL major version:

manylinux	glibc	RHEL
manylinux1	2.5	5
manylinux2010	2.12	6
manylinux2014	2.17	7
manylinux_2_28	2.28	8

That's because manylinux cannot bring its own updated compiler stack along and so is dependent on having the devtoolset backports. Perhaps keeping RHEL X as a base (through one of its ABI-compatible derivatives like Alma,Rocky,UBI) solves the other versioning questions, even if we call it conda_2_28?

[beckermr]

Yes. We plan to keep alma8 as the base.

[isuruf]

One plus about just using conda instead of conda_2_28, is that the user does not have to deal with cdt_name in the recipes. They can just use the versioning of the sysroot and the cdt_name will become obsolete.

[beckermr]

We'll need to tuck the cdt name in the CDTs somewhere maybe? Idk if the CDT package names will conflict or not.

[beckermr]

Can you send an example recipe where people are dealing directly with cdt_name? I thought the jinja2 function took care of that.

[isuruf]

For eg: https://github.com/conda-forge/qt-main-feedstock/blob/main/conda-forge.yml#L18-L19

[isuruf]

On the other hand, that line in conda-forge.yml serves two purposes. Setting cdt_name and the docker image name.

[beckermr]

Hmmmm. Being able to at least match cdts to the os for our docker containers is inherently useful possibly? This would argue for using conda_2_28 in both places.

[isuruf]

Being able to at least match cdts to the os for our docker containers is inherently useful possibly?

That's not really needed. We use cos6 CDTs in cos7 docker images.

[beckermr]

I'm not saying we alway have to have them matched. I'm saying that using the same notation in both places is helpful.

[jakirkham]

Would it help to start a PR for Docker images? Or are we not ready for that yet?

[beckermr]

Go for it but I don't expect it to be merged anytime soon.

[jakirkham]

Gotcha, what do we see as the required steps before they are merged? Asking since they wouldn't be integrated anywhere by just publishing the image. Or is there something else I'm missing?

[beckermr]

I am not sure what goes in them. If we need the sysroots to put in them, then we'd need that. If they don't have anything special, then we can just build it.

[jakirkham]

Gotcha

Don't think the sysroot is needed

The images cache the compiler packages as a convenience, but that can be disabled temporarily or it can use older compilers for now. Not too worried about this

Can't think of anything else of concern

If something comes up when we start working on them, we can always discuss

[jakirkham]

Started adding an image in PR ( conda-forge/docker-images#235 )

[jakirkham]

From Matt, we need to update os_version for conda_2_28. Also there is a corresponding way to do this in staged-recipes

xref: conda-forge/conda-smithy#1434

[h-vetinari]

adjust smithy to allow for easy alma8 config

What's the intention here? Being able to switch the image? (I'm asking because I'd be willing to give it a shot...)

FWIW, the current 2.17 setup doesn't need any smithy interaction, it's enough to just add sysroot_linux-64 2.17 # [linux64] to the build dependencies. AFAIR that's because we switched the images to cos7, while still using the cos6 sysroot by default. This setup was the result of a bunch of discussions around resolver errors and other issues with older images if there are packages built against the newer sysroot (for details see this summary of the very clever setup proposed by Isuru);

Is there a reason we couldn't switch the images to alma8, but keep the sysroot at cos6 (with opt-in upgrade to cos7 & alma8)?

[beckermr]

There is no reason we couldn't bump images. This may break builds using yum requirements if the packages have changed names or conventions upstream.

[h-vetinari]

This may break builds using yum requirements if the packages have changed names or conventions upstream.

According to this search, there's around 170 recipes that use yum_requirements.txt (for the most part, it's mesa, x11, etc.). I guess it would be possible to audit those for any name-changes, however, given that Alma 8 intends to be bug-for-bug compatible with RHEL 8, I strongly doubt that packages would changes names TBH.

[beckermr]

Fair point. I'm happy with simply bumping the default image.

[h-vetinari]

There seems to be something going awry with the new repodata hack. I'm getting Alma 8 kernel headers together with the COS 7 sysroot on aarch:

The following NEW packages will be INSTALLED [selection]:

    _sysroot_linux-aarch64_curr_repodata_hack: 4-h57d6b7b_13             conda-forge
    kernel-headers_linux-aarch64:              4.18.0-h5b4a56d_13        conda-forge  # <- kernel version in RHEL 8 !!
    sysroot_linux-aarch64:                     2.17-h5b4a56d_13          conda-forge  # <- glibc version in RHEL 7

Interestingly, this is not happening on PPC, where I get:

The following NEW packages will be INSTALLED [selection]:

    _sysroot_linux-ppc64le_curr_repodata_hack: 4-h43410cf_13             conda-forge
    kernel-headers_linux-ppc64le:              3.10.0-h23d7e6c_13        conda-forge  # <- kernel version in RHEL 7
    sysroot_linux-ppc64le:                     2.17-h23d7e6c_13          conda-forge  # <- glibc version in RHEL 7

[beckermr]

What makes you think it is the repodata hack?

[h-vetinari]

What makes you think it is the repodata hack?

It must be related to the sysroot, where the kernel-headers are built, and I couldn't see a difference between aarch/ppc in conda-forge/linux-sysroot-feedstock#46. Both variants are pulling in crdh 4, but looking a bit closer, that divergence between aarch & ppc goes back much further. Seems we've been using newer kernel headers on aarch since conda-forge/linux-sysroot-feedstock#15 (corresponding to linux version in RHEL 8, but apparently still being downloaded through CentOS 7 repos¹). Seems it's not critical.

Footnotes

1. though that PR doesn't document the rationale, so I can't really say why it diverged at all ↩

[jakirkham]

This is worth a read

https://almalinux.org/blog/impact-of-rhel-changes/

[jaimergp]

More news about RHEL: https://techcrunch.com/2023/07/11/why-suse-is-forking-red-hat-enterprise-linux/

[h-vetinari]

SUSE's effort will be a fork and not a clone (presumably based on CentOS stream...? SUSE already has an enterprise offering - very curious that they'll provide a knock-off of the competition...?), which loses the whole bug-for-bug compatibility guarantees with RHEL. But the whole situation is clearly something that has caused a lot of a stir so this'll probably take a while to settle.

I'd prefer to not wait that long with our sysroot. IMO we should move forward with one of the RHEL clones in its current state, as switching between those clones down the line will stay compatible. rhubi would be the safest (if we can make it work), but we could also pretty easily use CentOS Stream (still supported for a while) or even Alma in its current state.

I don't think Debian 10 (also using glibc 2.28) is a realistic alternative, as its EOL is 2024-06-30, which is only a month after the EOL of CentOS Stream 8, so then we could just us that without having to switch between distributions (yum vs. apt; different package names, etc.).

[jakirkham]

Maybe we could try out the UBIs in a feedstock? Idk how compatible they will be with the different RHEL clones, but at least they will be compatible with RHEL

Wonder how the clones are going to handle this? Maybe they will create a set of shared sources they work with? If so, sticking with AlmaLinux could be fine

Since we are discussing yum package availability and weighing our options, it is worth trying maybe one of the harder yum cases we can think of. Naively would start with something like qt-main or maybe qtconsole

Though we could also systematically generate all packages we get from yum by looping over feedstocks and collecting yum_requirements.txt into a set of packages

[h-vetinari]

Wonder how the clones are going to handle this?

Alma has decided to no longer aim to be 1:1 compatible with RHEL, but will still aim to be ABI-compatible (which should normally be good enough for us). But if they're planning to use the CentOS Stream sources for that, then those go EOL much earlier than they promise to. Not sure yet how that'll shake out.

Maybe we could try out the UBIs in a feedstock? Idk how compatible they will be with the different RHEL clones, but at least they will be compatible with RHEL

I continue to think that UBI is probably the best option. All of those under discussion will be ABI-compatible anyway, but UBI has the fewest question marks around availability/stability, resp. the longest EOL.

[jakirkham]

Also with yum packages it might be worth revisiting if some of these can move over to Conda packages. We have a pretty wide set of X11 packages, which should eliminate the need of getting these from yum

The missing mesa packages are a bit worrying since those are needed by packages that have Graphics components. AFAIK they are not easily replaceable so will need some thought

The other wrinkle to keep in mind is CDTs are built from yum packages (and in some cases yum sources). So what RHEL provides (or what clones use) directly affects the CDTs

[beckermr]

The other wrinkle to keep in mind is CDTs are built from yum packages (and in some cases yum sources). So what RHEL provides (or what clones use) directly affects the CDTs

I 💯 agree here. This is an important point. CDTs smell funny, but I don't want us crippled by what RH provides in the UBI.

[h-vetinari]

We have a pretty wide set of X11 packages, which should eliminate the need of getting these from yum

Things are moving to Wayland anyway - slowly but surely -, so perhaps we can indeed get rid of some things there.

Maybe even packaging mesa itself could work? Their installation instructions do cover running against a non-system build, so it sounds like it may be possible to pull off?

In any case, if UBI is not fully featured enough in terms of yum packages, I feel we might just go with CentOS stream for now. If Alma has found a way to keep going by the time stream goes EOL, we can still switch because things are (and will remain) ABI compatible

[h-vetinari]

Looking at the most recent alma update (about a month ago) again, it does seem like they'll be focussing on CentOS Stream. They're not putting it that directly, but

We will also start asking anyone who reports bugs in AlmaLinux OS to attempt to test and replicate the problem in CentOS Stream as well, so we can focus our energy on correcting it in the right place.

certainly points in that direction. Following CentOS Stream will keep them ABI-compatible (which is enough for us), but not the previous bug-for-bug compatible. It's also not clear to me how they'll keep going after a given version of CentOS Stream goes EOL (much earlier than the respective RHEL version), but the parts we need are hardly going to keep changing much anyway.

I couldn't make the core call today, but I see in the notes that:

SUSE as an option potentially? Will wait and see; still unclear where everything stands

I would stay with the RHEL clones. Even CentOS Stream would be good enough, if we just keep using it past its EOL (like we've been doing for CentOS 6 as well). It's not like we get a noticeably longer EOL with OpenSUSE (the latest 15.5 is EOL in Dec. '24, barely half a year more than CentOS Stream 8).

Finally, we could still decide to do OpenSUSE independently of whether we finish off the 2.28 Alma sysroots now, because OpenSUSE is on glibc 2.31, so they would be entirely separate.

[beckermr]

@h-vetinari I meant using this: https://www.suse.com/news/SUSE-Preserves-Choice-in-Enterprise-Linux/

[h-vetinari]

Yeah okay, sorry for misinterpreting that one. For that initiative, aside from opensuse cannibalising its own Enterprise Linux product (so who knows how long that remains attractive to them), they also only have the sources of CentOS stream to leverage for ABI-compat. Once the updates there stop, I don't see how this is any different from any of the other RHEL clones.

More importantly, the clones are all ABI-compatible (which is the number one argument to have a clone in the first place), and likely very tightly derived from RHEL sources and/or CentOS stream sources. So it's very likely that we could change the RHEL clone behind our sysroot (and even CDTs), and it would not be an issue.

So I think we can't really do much wrong by going with Alma for now. Either they find a way to keep going past Stream's EOL, or there's another ABI-compatible distro to switch to that does, or we just accept that the image doesn't receive updates anymore. None of these sound terrible IMO.

[beckermr]

Right yeah. My only thought might be that folks could choose to be bug-for-bug compatible with their distro in which case we might have a preference. Only time will tell though. At the ABI level, totally agreed.

[h-vetinari]

I think no clone will realistically be able stay bug-for-bug. Alma already explicitly abandoned that.

It's not realistic to do without knowing the RHEL sources, which no-one gets anymore (well, and if you sneak them out of a subscription and into your distro, RHEL can trivially observe in the open distro sources whether someone is picking up their internal patches, and then come and hound you).

The only way to do it would be to reverse engineer the bug fix from the bug report and profiling RHEL behaviour, but ain't nobody got time for dat.

It's just too much risk and effort for too little gain I think. Especially as ABI-compat is IMO orders of magnitude more important than whether a given bug behaves exactly the same as on RHEL.

Beyond that, I don't see how bug-for-bug compat would be useful to conda-forge. We need the ABI, but I think we can agree that we'd generally like to have less bugs, not more (and most of our linux users will be on way newer distros that carry those fixes already). So from that POV, following CentOS stream would be attractive, were it not for the shorter EOL.

Luckily, keeping the ABI while backporting their own bugfixes is something that is at least realistically feasible for the clones, so they actually have a chance to extend that EOL as promised, beyond what they get from stream. Overall, I still think the best choice is going forward with Alma now.

[jakirkham]

IIRC in our last conda-forge meeting it sounded like we are ok going ahead with AlmaLinux 8. Did I understand correctly?

If so, it sounds like it comes down to doing these next steps (particularly upgrading CDTs). Does that sound right?

[beckermr]

Yes but see the list above. We are trying to get rid of as many CDTs as we can.

[h-vetinari]

We have merged conda-forge/docker-images#242 recently. What are some next steps here? Are we ready to tackle the first CDTs (modulo conda-forge/cdt-builds#66)?

[beckermr]

We agreed to get rid of as many CDTs as we could. So that's the next step. To go through them and figure out what we can build ourselves.

[jakirkham]

Maybe an earlier step is just getting a list of CDTs we use so we can search conda-forge for fuzzy matches

[jaimergp]

xref conda-forge/cdt-builds#66