Use Alma Linux 9 instead of Alma Linux 8

[MarkEWaite]

Use Alma Linux 9 instead of Alma Linux 8

Alma Linux 9.2 is the most recent Alma Linux release. It has a longer support life and is more likely to include important fixes.

Also incremented the default version of Jenkins that this script packages so that it matches with the next long term support baseline, 2.401.

Testing done

Ran make build-almalinux_jdk11 and make test-almalinux_jdk11 and confirmed that build and test work as expected. Since the UBI container image is already using UBI 9, this should be a safe change.

@saper I'd like a code review from you, since you're listed as CODEOWNER of the Alma Linux container image.

Submitter checklist

• Make sure you are opening from a topic/feature/bugfix branch (right side) and not your main branch!
• Ensure that the pull request title represents the desired changelog entry
• Please describe what you did
• Link to relevant issues in GitHub or Jira
• Link to relevant pull requests, esp. upstream and downstream changes
• Ensure you have provided tests - that demonstrates feature works or fixes the issue

[dduportal]

Blocking the PR from any merge for the same reason as #1586 (review)

=> once approved by maintainers, we have to build it manually for the past 2 weekly and LTS to make sure it does not rebuild (and override) them.

(Have not look at th PR content or goal itself)

[dduportal]

• Technical content looks good
• Waiting for @saper advice
• @MarkEWaite candid question: shouldn't we mark this change as "breaking" to mark that we stop distributing the line 8.x of AlmaLinux?

[MarkEWaite]

Waiting for @saper advice

I agree that we should wait for a code review from @saper

shouldn't we mark this change as "breaking" to mark that we stop distributing the line 8.x of AlmaLinux?

Yes, I think that you are correct. We should mark this change as "breaking" because it changes the container image in a way that may surprise users. I've added the "breaking" label to remind us that it needs an entry in the Jenkins weekly changelog and the Jenkins LTS changelog.

I'm open to consider a blog post as well, if that is helpful. The weekly and LTS changelog seems like the best place to document it, but a blog post provides a very visible location to announce the change.

=> once approved by maintainers, we have to build it manually for the past 2 weekly and LTS to make sure it does not rebuild (and override) them.

I think this one is different than #1586 (review) . I don't think that we need to rebuild any previous releases with this change. I think that it will be better for users if we document the change in the first weekly changelog and the first LTS changelog where the change is visible. I don't think that we should alter the content of any already created almalinux container tags for this change.

[dduportal]

=> once approved by maintainers, we have to build it manually for the past 2 weekly and LTS to make sure it does not rebuild (and override) them.

I think this one is different than #1586 (review) . I don't think that we need to rebuild any previous releases with this change. I think that it will be better for users if we document the change in the first weekly changelog and the first LTS changelog where the change is visible. I don't think that we should alter the content of any already created almalinux container tags for this change.

Alas it is a technical constraint today: if you simulate the execution of the trusted.ci.jenkins.io daily build on the last commit of this PR, the 4 images (past 2 weekly and past 2 LTS) are suddendly seen as "not published":

docker git:(use-almalinux-9) ./.ci/publish.sh -n
Docker repository in Use:
* JENKINS_REPO: jenkins/jenkins
Dry run, will not publish images
2.403 not published yet
# ...
2.404 not published yet 
# ...
2.387.2 not published yet
# ...
2.387.3 not published yet
# ...

=> the reason is because the script .ci/publish.sh checks for existence of a given "version" by looking all the manifests and comparing to the list of tags and CPU architecture defined in docker-bake.hcl

Given the challenge it gives us, I'm inclined to propose a PR later today to switch the trusted.ci.jenkins.io delivery to be triggered by tag only (and remove this challenging script).
=> this would ensure we can always relate 1 "version" in DockerHub to 1 git tag + release here. The only real change would be that the weekly and LTS Core release produce would have to be amended: instead of connect to trusted.ci and launch a build or wait 6h, it would be create a tag in the jenkinsci/docker repository with the Jenkins version and wait 15 min max. WDYT @MarkEWaite ?

[MarkEWaite]

I'm inclined to propose a PR later today to switch the trusted.ci.jenkins.io delivery to be triggered by tag only (and remove this challenging script).

I would like that very much. The "guess which containers to push" mode has many ways that it can fail. Push when a new tag arrives seems less likely to have corner cases and unexpected problems.

[MarkEWaite]

if you simulate the execution of the trusted.ci.jenkins.io daily build on the last commit of this PR, the 4 images (past 2 weekly and past 2 LTS) are suddenly seen as "not published":

Very interesting. I assume we'll have more cases like this. In the future, I expect to propose a pull request to stop publishing the container image(s) that use centos7 as their base image. The upstream centos7 container image has not been maintained since November 2020 and Red Hat is ending support of Red Hat Enterprise Linux 7 in 2024.

I've started a Jenkins developers discussion thread on the idea and have submitted a draft pull request jenkinsci/jenkins#7913 to Jenkins core that will warn CentOS 7 users of the upcoming end of life.

[dduportal]

Started initial work in #1630

[dduportal]

@MarkEWaite the diff shows the following error:

[dduportal]

Adding approval to remove the blockage I've put, since #1630 was merged successfully: the "rebuild" risk does not exist anymore.

[MarkEWaite]

@MarkEWaite the diff shows the following error:

Thanks for detecting that error. I'm not sure if we failed to add @saper to the maintainers of this repository or if @saper has been removed himself from the repository.

@saper are you interested in continuing as a code owner of the Alma Linux container image?

[saper]

I am not sure what the maintainer/code owner duties here are. But I can help with testing/reviewing/whatever is needed.
Speaking of this change:

1. Haven't tested it yet
2. We take Java from the eclipse-temurin:11.0.19_7-jdk-centos7 - this does not seem quite right (although it has a chance to work) - shouldn't we use https://github.com/adoptium/containers/blob/920efae8fe37e2b8f2b288b5f7f9e67134ecad1d/11/jdk/ubi/ubi9-minimal/Dockerfile.releases.full here?

[MarkEWaite]

I am not sure what the maintainer/code owner duties here are. But I can help with testing/reviewing/whatever is needed. Speaking of this change:

1. Haven't tested it yet

If you're willing to test it, that would be much appreciated. That would then let you say, "I approve". We can resolve the repository permissions question later

2. We take Java from the eclipse-temurin:11.0.19_7-jdk-centos7 - this does not seem quite right (although it has a chance to work) - shouldn't we use https://github.com/adoptium/containers/blob/920efae8fe37e2b8f2b288b5f7f9e67134ecad1d/11/jdk/ubi/ubi9-minimal/Dockerfile.releases.full here?

That's a very good point and a good thing to include in a new pull request that prepares us for the June 2024 official end of life of CentOS 7 and the 2023 expected early end of life for CentOS 7 in the Jenkins project. I've started a Jenkins developers mailing list discussion on it and have submitted a Jenkins core pull request preparing for an early end of life for CentOS 7 in the Jenkins project.

[timja]

I agree the ubi image should be used instead of the centos 7 one.

[dduportal]

I agree the ubi image should be used instead of the centos 7 one.

+1 with this

[MarkEWaite]

I agree the ubi image should be used instead of the centos 7 one.

+1 with this

Done in f291ede

[timja]

Looks good, I checked what else is using centos 7, https://github.com/search?q=repo%3Ajenkinsci%2Fdocker%2011.0.19_7-jdk-centos7&type=code

May want to consider updating the ubi8 to ubi9 for Java 11.

[MarkEWaite]

May want to consider updating the ubi8 to ubi9 for Java 11.

I think that is a good idea for another pull request. That pull request will be more complicated because the container image includes the UBI version in its name. Will need discussion with @olivergondza to see if it is better to create a new container image for ubi9 and stop updating the ubi8 container image or if we need to have both of them.

[dduportal]

May want to consider updating the ubi8 to ubi9 for Java 11.

I think that is a good idea for another pull request. That pull request will be more complicated because the container image includes the UBI version in its name. Will need discussion with @olivergondza to see if it is better to create a new container image for ubi9 and stop updating the ubi8 container image or if we need to have both of them.

Good point, may I suggest to separate concerns then?

• Let's update this PR to "only" add the new UBI9 for both JDK11 and JDK17, as a new feature. That would make it simpler and fatser to review, test, approve (and deliver).

• Then, a subsequent PR should remove the UBI8 to have the discussion about "deprecate it or not"

WDYT @MarkEWaite ?

[MarkEWaite]

Good point, may I suggest to separate concerns then?

• Let's update this PR to "only" add the new UBI9 for both JDK11 and JDK17, as a new feature. That would make it simpler and fatser to review, test, approve (and deliver).
• Then, a subsequent PR should remove the UBI8 to have the discussion about "deprecate it or not"

I was assuming an even greater separation of concerns by limiting this pull request to only change Alma Linux and let other pull requests be used to improve or adjust the UBI container images. If this pull request should include additions for the UBI 9 containers, I'm fine with that as well, though I thought it would be easier in the changelog if that was a separate pull request.

[dduportal]

Good point, may I suggest to separate concerns then?

• Let's update this PR to "only" add the new UBI9 for both JDK11 and JDK17, as a new feature. That would make it simpler and fatser to review, test, approve (and deliver).
• Then, a subsequent PR should remove the UBI8 to have the discussion about "deprecate it or not"

I was assuming an even greater separation of concerns by limiting this pull request to only change Alma Linux and let other pull requests be used to improve or adjust the UBI container images. If this pull request should include additions for the UBI 9 containers, I'm fine with that as well, though I thought it would be easier in the changelog if that was a separate pull request.

You are absolutletly correct! Separating Alma from Ubi images is a great idea.
I mixed up the "temurin ubi" change with the "UBI" images in my mind, sorry.

[MarkEWaite]

@ksalerno99 asked a very good question in the platform SIG meeting today. He asked why we deliver an Alma Linux container when UBI containers are the preferred way of delivering Red Hat compatible container images.

I am placing this pull request on hold so that we can consider an alternate approach. I think the alternate is to extend the facility initially defined in

• Warn when operating system end of life is approaching jenkins#7913

so that we warn container users that their container image will no longer be supported after a certain date (like Nov 16, 2023).

We declare end of life for the following container images:

• Alma Linux
• Blue ocean

More discussion is needed for the idea, but it seems sensible to me.

[saper]

He asked why we deliver an Alma Linux container when UBI containers are the preferred way of delivering Red Hat compatible container images.

I don't know if things are different now, but last time I tried to build own derivative of the UBI-based image (and trying to add some RPMs from Red Hat Enterprise Linux there), it turned out I needed to have a valid Red Hat Enterprise Linux subscription.

Even in the organization that has a commercial license from Red Hat, it turned out to be a pain to deal with the subscription manager setup for the build infrastructure to make it work.

From: https://developers.redhat.com/articles/ubi-faq#redistribution

22. Can I add non-UBI RPMs to a UBI image and still redistribute the resultant container image on a non-Red Hat platform?

No, only Red Hat packages assigned to the UBI EULA can be redistributed. The EULA, under which UBI is governed, allows users to distribute a set of container images and RPMs delivered as part of UBI. Container images and RPMs which are covered by a different EULA have different rights and restrictions. Of course, and in general, you can expect any community open source technology to be redistributable.

[ksalerno99]

He asked why we deliver an Alma Linux container when UBI containers are the preferred way of delivering Red Hat compatible container images.

I don't know if things are different now, but last time I tried to build own derivative of the UBI-based image (and trying to add some RPMs from Red Hat Enterprise Linux there), it turned out I needed to have a valid Red Hat Enterprise Linux subscription.

Even in the organization that has a commercial license from Red Hat, it turned out to be a pain to deal with the subscription manager setup for the build infrastructure to make it work.

Hi, are you sure that's still the case? The last time I added RPMs to a UBI image (ubi8 and ubi9) the message from subscription-manager when running dnf was that a subscription is not required for UBI. Meaning, dnf commands to add packages work without any additional registration, and we currently build a ubi9 image of Jenkins controller that also does not require any subscription registration.

[saper]

It works if you 1) use some limited set of UBI-licensed RPMs or 2) do it on an otherwise licensed Red Hat Enterprise Linux machine. If you do build a container on something else (in my case that was some vanilla Kubernetes cluster not running on the Enterprise Linux and not OpenShift), you need to have an active license. Looks like what the FAQ says is still true.

[olivergondza]

@MarkEWaite, @dduportal, regarding introduction of UBI9 and/or Java 17 support: The way we played this is that we will keep https://github.com/jenkinsci/docker/tree/master/11/rhel/ubi8/hotspot for a bit while we will also provide https://github.com/jenkinsci/docker/tree/master/17/rhel/ubi9/hotspot for users to migrate to. We do not intend to provide the remaining 2 combinations and have them hosted by Jenkins upstream.

This nudges users to upgrade the system and java in one go, but we are ok to live with that.

[olivergondza]

@saper, is correct, the list of Red Hat provided RPMs for UBI is constrained unless you have purchased a subscription. That can limit you somewhat in what you can install for downstream images.

---

As the maintainer of the UBI images here, I have an interest for those to stay, and I do not object if Alma ones remains distributed in parallel. Provided there is an interest in reducing the set of images produced by upstream, and we choose to converge Alma into UBI, and this is a good opportunity to do that.

I would suggest instead of actually doing this MR, to simply point users to UBI9&&Java17 as a migration path from Alma 8. Based on user feedback we will learn if this is sufficient, or there continue to be a need to resurrect Alma - it largely depends on what the users install in the downstream containers and if they need to redistribute those.

[MarkEWaite]

@olivergondza I like that recommendation very much. I'm closing this pull request so that we can use the alternate approach of implementing a container end of life warning similar to the operating system end of life warning, then we can use that container end of life to guide people to transition towards preferred container images.

[nmusatti]

I'm a user of the lts-almalinux image to which I moved from the centos7 one. Since I've started getting end of life messages for jdk11 I was looking for ways to upgrade to jdk17 or jdk21 and found this pull request. I don't fulfill the Red Hat licensing requirements mentioned above, so the termination of the almalinux image is most inconvenient for me as it would force me to try and recreate the custom image I've been using for several years on a very different distribution.

[MarkEWaite]

I'm a user of the lts-almalinux image to which I moved from the centos7 one. Since I've started getting end of life messages for jdk11 I was looking for ways to upgrade to jdk17 or jdk21 and found this pull request. I don't fulfill the Red Hat licensing requirements mentioned above, so the termination of the almalinux image is most inconvenient for me as it would force me to try and recreate the custom image I've been using for several years on a very different distribution.

@nmusatti you chose a very good time to ask the question. A number of pull requests have been merged recently to prepare for the release of Java 21 support for the more exotic platforms (s390x and ppc64le) in pull request:

• Replace Java 21 previews with released versions #1822

As part of those pull requests, I think that the end of Java 11 support in Oct 2024 will be the end of the almalinux container image. There are other outdated items in the Java 11 container images. The Oct 2024 end of Java 11 support will simplify future maintenance of container images by removing the container images that have those outdated items.

In Nov 2024, I think that our container images should only support:

• Alpine 3.20 (or 3.19 if 3.20 is late)
• Debian 12 (Bookworm)
• UBI 9

I'm hesitant to add another container image unless we also add one or more active maintainers for that container image. I'd rather have you make the switch to a Debian or an Alpine based container image than add a new container image without an additional maintainer.

Other maintainers of the image may have different opinions and they are welcome to express their opinions.

Are you interested in becoming a maintainer?

[nmusatti]

I might consider becoming a maintainer of a non Red Hat EL-based image, depending on the kind of effort that would be required. However I would prefer to use Rocky Linux as a base, as that is the distribution I use.

[MarkEWaite]

I might consider becoming a maintainer of a non Red Hat EL-based image, depending on the kind of effort that would be required. However I would prefer to use Rocky Linux as a base, as that is the distribution I use.

I don't have a strong opinion between AlmaLinux and Rocky Linux. We transitioned from UBI 8 with Java 11 to UBI 9 with Java 17 and Java 21. I think it would be reasonable to transition from AlmaLinux 8 with Java 11 to Rocky Linux 9 with Java 17 and Java 21, so long as we have one or more active maintainers for Rocky Linux 9 with Java 17 and Java 21.

In order to become a maintainer, you'd need to show that you have the time and the skills to maintain the images. One very good way to do that is to review and respond to issues. If one of the issues needs a code change, then you could submit a pull request to resolve the issue.

[dduportal]

Question about usage: what is the reason for not using debian based image? The context of using Docker means the container is providing an application with a portable behavior, which is the case for Jenkins controllers. As much as agent might have package constraints, it is not clear to me why even having 2+ different OS for a given image? (But i might miss something)

[saper]

Here is my take on it - why I have initially got into this at all.

It is all about maintaining a large number of derived images - in our case, we have almost exclusively a Red Hat-based environment and we produce derived images for example also for Jenkins. Since container images have to be regularly updated and rebuilt, I find it better to deal with one OS (Enterprise Linux derivative) everywhere than a combination of Debian, RedHat, Alpine based images. In the simplest case, I want to have update-ca-trust everywhere. In the most complex case (not related to Jenkins in particular) I want to have smartcard-related configuration done right everywhere.

The reason we do not use UBI right now is that one might run into a licensing configuration issue if not building on Red Hat platform, like running image building pipeline on the Kubernetes cluster not owned by us, or trying to build the image at home on something that is not a licensed RHEL machine.

I'll have a look at this together with #1822

Let's have a look if the above reasons are enough to carry the burden. Most importantly, I don't know how many users derive they own images from the jenkins one.

[nmusatti]

In my case I'd rather stay on an EL derivative rather than switching to Debian because I use a customized image which contains an assortment of tools and system packages that are required by this or that build. Switching to a different distro would require a significant amount of testing, which I'd rather avoid. On the other hand I'm well aware that this is on a voluntary base, I don't think that I'll be able to guarantee the degree of availability and free time that would be required in order to take on maintenance of the EL image, so whatever the decision, I'll find a way to cope.