Migrate Centos 8 agent and server images to Centos Stream 8

[chadlwilson]

The regular Centos 8 is EOL on 31 December 2021. The only further updates will come via Centos stream.

Rather than distinct naming the images, this makes the lowest effort change to migrate to the new Centos Stream base images published by the team.

• Fixes Replace Centos8 images because of EOL #9940
• Introduces support to pull from different registries by distro and version. Not sure if this will have an impact on caching, as Docker cannot mirror from quay.io OOTB.
• Further discussion is needed on whether this is the best thing to do. Centos Stream will now be upstream of RHEL point releases, delivered in more "continuous delivery" style. Given GoCD releases relatively infrequently but rebuilds and sanity tests on latest images on every commit, I don't believe this is likely to be a major issue or will negatively harm its stability.

The base image as built by Centos seems to be much bigger on Centos Stream (231 MB vs 422 MB uncompressed), but it doesn't seem to make a huge difference to the end image size after GoCD is layered on top.

Type	Before	After
Agent	871 MB uncompressed	920 MB uncompressed (+49 MB)
Server	1.05 GB uncompressed	1.10 GB uncompressed (+50 MB)

Looking for comments on

• There are currently no Docker Library official images yet for Centos Stream. This means they are not getting regularly rebuilt for updates and would rely on us yum update-ing them ourselves during our builds - thankfully we do this already. By contrast there are official Docker library images for Rocky Linux.
• Generally are there concerns about use of Stream? If it is not a good idea, moving to Rocky or Alma Linux distros is an option, but is a bit more work to introduce a new distro and might be a bit weird as we'd have Centos 7 hanging around.
  • There are a few different opinions around the place. This one is more positive.
  • I had some generic comments here.

[chadlwilson]

If @ketan has any opinion, that'd be useful too 🙏

[ketan]

@chadlwilson — I saw your comments on #9940. I don't have any particular concerns on it. Given that the image builds are all automated, it's straight forward to add it to the build and have images published — users will obviously need and want both :)

It may be useful to share some recipes on how users could build their images, and we get out of the business of building images. But having several images does make it super simple for users to build their custom images on top of ours.

I don't know what's the best path forward here :-(

[chadlwilson]

Thanks @ketan. They are automated, yes, however there is some cost in terms of build time, feedback loop, maintaining all the repos here and on Docker Hub etc. I suppose I am possibly sensitive as I'm feeling like a one-man-ish maintenance operation at the moment and seem to find new surprises + things to patch+upgrade+maintain every time I look 😉 Haven't even started trying to figure out how to remove deprecated APIs, or release patched-lib-versions of all the plugins yet :-)

I do like that we build images for the same reasons as you; but my personal preference would be to

• build fewer variants, maybe alpine + debian|ubuntu and that's it, but there were probably some good reasons for the previous choices.
• support Alpine variants for less time than upstream support policy. Alpine has 4 different variants in support at any time, since they have a 6 month release cycle and a 2 year support policy. I don't think we need to build agent images for all 4 variants :-) As a user, previously my team built a custom image on top of the gocd-agent-docker-dind image and that typically just tracks latest Alpine based on what docker:dind do. Never had any major issues with that since Alpine is so small, and seems robust/stable with few "breaking" changes. Anecdotally, most projects that produce Alpine-based images I have seen appear to just use latest stable, with few months of lag from release.
• offset that by rebuilding regularly the images we do support on schedule like most projects do (and I believe is required for Docker official library images), publishing to the existing tags as mutable + an immutable tag with a date in it. Without rebuilds, base OS-level libs don't get patched. That was probably OK when GoCD was releasing every month or two, but right now it's not so ideal.

[arvindsv]

I feel like I haven't kept up with the changes in Centos enough to review this. Maybe @marques-work has a better sense of it.

I do think it doesn't make sense for you to maintain all of these versions yourself. So, please prioritize making your life easier. If that is Centos Stream 8, then so be it. If people need a different one, they can manage it, and we can link to it, use it if needed.

The fact that Rocky Linux has regularly built images gives me some pause, but my point above stands.

[chadlwilson]

Fair enough. I think the community were largely upset because

• the earlier support guarantees were dropped/changed
• Centos Stream will now not be an exact copy of RHEL, but instead some approximation of a release candidate build of what will get into the next RHEL/Rocky/Alma point releases.

From Phil Dibowitz's article:

[chadlwilson]

For what it's worth; I migrated build.gocd.org's own Linux elastic agent containers to Centos Stream 8 in gocd-contrib/gocd-oss-cookbooks@f9a8256 and switched over on the build infrastructure. All seems good so far using it as the base for a custom agent image.

[chadlwilson]

In the absence of any strong opinions either way, I will merge this, since it seems it is better than building on what will be EOL base images in a few days. If we want to abandon Centos Stream, deprecate the centos distros entirely and create new Rocky images after, we can do that.