Build BB worker images in CI and push on quay.io

[fauust]

@vladbogo, @grooverdan this is a first attempt to build BB containers in a CI, comments are welcome!

Please take a look at the following TODO list, also @vladbogo has created a jira task so we can discuss and list all the requirements for this (https://jira.mariadb.org/browse/MDBF-273).

@vladbogo I am interested in testing those containers on BB (above all non amd64 arch because they are build with QEMU emulation). You can fetch them directly from https://hub.docker.com/r/fauust/bb/tags?page=1&ordering=last_updated

Example:
docker pull fauust/bb:ppc-debian-10

Finally, take a look at the final step (Check Dockerfile with hadolint) of the CI. This is a non failing test (for the moment) and it gives indications on what should be improved (or ignored) in the Dockerfiles.

TODO:

• handle tags (it is important that we keep an history of tags in order to be able to rollback to a previous container image if build fails):
• improve dockerfile management (there is a lot of redundant stuff ATM);
• define which hadolint checks should be ignored and fix the others (see last CI step);
• define which registry to use;
• rebuild only dockerfiles that have changed;
• schedule build.

[grooverdan]

Nice start @fauust.

Pulling by container hash is recommended for obtaining a guaranteed stable interface on an image rather than a tag. The remain available for a period of time. But yes, more thought needed around the processes here.

Yes there seem to be a large scope for container consolidation into single multiiarch compatible build (and push manifests of images).

[fauust]

Pulling by container hash is recommended for obtaining a guaranteed stable interface on an image rather than a tag. The remain available for a period of time. But yes, more thought needed around the processes here.

By container hash, you mean commit hash?
Then would something like the following be a good solution:

• registry.io/bb-os-version:<hash>

So, for example:

• registry.io/bb-debian-10:<hash>
• registry.io/bb-fedora-33:<hash>
• registry.io/bb-rhel-8:<hash>

Yes there seem to be a large scope for container consolidation into single multiiarch compatible build (and push manifests of images).

Yep indeed.

There is a balance here between having very descriptive Dockerfiles and doing some more dynamic stuff in it, like for instance (does not work, only for example):

-RUN curl https://github.com/Yelp/dumb-init/releases/download/v1.2.2/dumb-init_1.2.2_amd64.deb -Lo /tmp/init.deb && dpkg -i /tmp/init.deb
+RUN curl https://github.com/Yelp/dumb-init/releases/download/v1.2.2/dumb-init_1.2.2_$(arch).deb -Lo /tmp/init.deb && dpkg -i /tmp/init.deb

There is also the option of generating those Dockerfiles from a template.

I also would like us to consider @ottok proposition #43.

Anyway, I don't have a good overview about all those Dockerfiles so I'll let you do suggestion.

Finally, @grooverdan do you think that we could use quay.io (I don't know if we have quota or storage limitation there) or should I create a container registry on Hetzner cloud.

[grooverdan]

I was thinking pull by digest:

$ podman pull quay.io/mariadb-foundation/mariadb-devel@sha256:401211f80183847b7864605104f74ac743ece5fcd3dd25fe65005482bfa172f5

Useful for keeping dev/prod environments identical.

See https://github.com/Yelp/dumb-init/releases now has just the plain exec with as standard OS arch:

RUN curl https://github.com/Yelp/dumb-init/releases/download/v1.2.5/dumb-init_1.2.5_\$(uname -m) -Lo ....

But sure, use template if things get odd. Also build arg TARGETPLATFORM etc https://docs.docker.com/buildx/working-with-buildx/.

use quay.io (I don't know if we have quota or storage limitation there) or should I create a container registry on Hetzner cloud.

There is no limits. Just be clear with with naming if public. Private quay.io repos need a microplan, I haven't checked details. You're welcome to create a repository or use another repository if you want.

[fauust]

I was thinking pull by digest:

$ podman pull quay.io/mariadb-foundation/mariadb-devel@sha256:401211f80183847b7864605104f74ac743ece5fcd3dd25fe65005482bfa172f5

Useful for keeping dev/prod environments identical.

Ok, make sense. Let's concentrate on containers names then and use latest for tag.

See https://github.com/Yelp/dumb-init/releases now has just the plain exec with as standard OS arch:

RUN curl https://github.com/Yelp/dumb-init/releases/download/v1.2.5/dumb-init_1.2.5_\$(uname -m) -Lo ....

👍

But sure, use template if things get odd. Also build arg TARGETPLATFORM etc https://docs.docker.com/buildx/working-with-buildx/.

That's already handled in the CI ;) I will prepare Dockerfiles accordingly.

There is no limits. Just be clear with with naming if public. Private quay.io repos need a microplan, I haven't checked details. You're welcome to create a repository or use another repository if you want.

If we can relay on quay.io, this is less maintenance for us, I will give it a try then.

[illuusio]

Should those long RUN apt-get && lines be just own RUN commands or script which is executed on build as now they are very hard to parse with human eye (they are nicely intended and all) or is it just me?

[grooverdan]

Its harder, but for the greater good:
https://docs.docker.com/develop/develop-images/dockerfile_best-practices/#sort-multi-line-arguments

[illuusio]

Ok I just need to adapt my brain. Where I can see builded images or build process?

[fauust]

@illuusio for the moment, containers are build on workers but this PR tries to move the building process in a CI.

[cvicentiu]

Hi Faustin!

I'm trying to understand how this workflow will work. So here are my questions for you as well as things to consider when implementing this:

1. When do images get built? Is there a button we can press to get them to get "rebuilt"?
2. Do we have control over when a particular image is "pushed" to the registry?
3. Can Github properly build SLES, RHEL and such dockerfiles? SLES is particularly problematic as one needs a SLES host to actually be able to build a docker file with SLES repositories enabled.
4. Does this work with multi-arch? (x86, x86_64, arm64, ppc64le)
5. It seems that one can use self-hosted runners to get more architectures covered. https://docs.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners#linux
6. What is the performance of the machines running the docker build? We normally shouldn't be running this too often, but for the tarball builder (which we currently do not have) we will need to have a dockerfile that compiles practically all of the system libraries from source, so that they can be linked in statically. We need the process to actually be able to achieve that. This could mean run-times of a few hours. I see some specs here:
   https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners#supported-runners-and-hardware-resources Are these it?
7. What is the advantage of using github actions versus running this as a cron job on the buildbot master?

[illuusio]

@cvicentiu I was actually wondering the same as I'm in impression that Github CI works only on commit based or is there webhook based system also?

Can Github properly build SLES, RHEL and such dockerfiles? SLES is particularly problematic as one needs a SLES host to actually be able to build a docker file with SLES repositories enabled.

Actually for SLES https://build.opensuse.org offers all SLE(S) versions for building from source spec to RPM and there is also possibility to build Docker/Podman images with it as we build our Buildbot master/slave images with Kiwi (Which is easier to use in some cases than Dockefile) but it also builds from Dockerfile (link for our Buildbot master/slave https://build.opensuse.org/package/show/home:illuusio:images/buildbot-container-kiwi). Pros are they are update every time rpm/deb updates or by webhook.

[cvicentiu]

@illuusio I have looked at build.opensuse.org. What we need is to be able to compile and run MariaDB in the same environment (libraries, compiler, etc.) as Suse Linux Enterprise Server. With OpenSuse it's easy. We can recreate docker files no problem. The enterprise version however has packages that can only be downloaded from special repositories that are only enabled with a subscription, much like Red Hat.

Can one create that environment (Say SLES 12 SP4) with the build system linked above?

[fauust]

Hi Vicentju!

1. When do images get built? Is there a button we can press to get them to get "rebuilt"?

The CI runs with those triggers:

  on:
    push:
      paths:
        - .github/workflows/bb_containers.yml
        - buildbot.mariadb.org/dockerfiles/ci/*
    pull_request:
      paths:
        - .github/workflows/bb_containers.yml
        - buildbot.mariadb.org/dockerfiles/ci/*

But we can use any of these https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#on. The schedule option maybe what you are looking for.

2. Do we have control over when a particular image is "pushed" to the registry?

ATM, if the image builds correctly and check passes, it will be pushed to the registry. But we can have a better control of this (based on a tag, a particular commit message etc.). Plus we should keep old images, see my previous comments.

3. Can Github properly build SLES, RHEL and such dockerfiles? SLES is particularly problematic as one needs a SLES host to actually be able to build a docker file with SLES repositories enabled.

Good Q. No idea at the moment I should investigate, using our own runner is maybe an option (but see bellow 5.).

4. Does this work with multi-arch? (x86, x86_64, arm64, ppc64le)

Yes, see actual working builds:
https://hub.docker.com/r/fauust/bb/tags?page=1&ordering=last_updated

5. It seems that one can use self-hosted runners to get more architectures covered. https://docs.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners#linux

Yes, but then there is also the option of using BB for those builds. If building particular arch/os becomes too complicated with GH actions, I would vote for that option. Also we should keep in mind what happened with TravisCI and keep the possibility to switch to another CI. But in the present case, there is nothing very specific to GitHub actions as the biggest work is to manage correctly those Dockerfiles (that particular point really need improvement IMO).

6. What is the performance of the machines running the docker build? We normally shouldn't be running this too often, but for the tarball builder (which we currently do not have) we will need to have a dockerfile that compiles practically all of the system libraries from source, so that they can be linked in statically. We need the process to actually be able to achieve that.

This is a new requirement to me, we will have to think a bit more how to handle this. I am not sure that it's a good idea to rely on GH actions for those heavy builds.

This could mean run-times of a few hours. I see some specs here:
https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners#supported-runners-and-hardware-resources Are these it?

Yes

7. What is the advantage of using github actions versus running this as a cron job on the buildbot master?

Those are the pros IMO:

• maintenance cost;
• bandwidth perf/stability;
• integration with this repo;
• GH powered runner;
• easy multi-arch build setup (https://github.com/marketplace/actions/docker-setup-buildx).

If it result better to use BB for that task (which we should definitively consider), I would vote for using actual builders (not the master). After all, this is just another build.

[illuusio]

@cvicentiu You can build SUSE SLE(S) RPM with OBS SLE 12-15 packages which should be the same as subscriptions packages which are behind paywall. Things get problematic if you provide those RPMs how they go legally?
Also one can build Docker image from those SLE RPMs (But using this Docker requires subscription?).
From SLE 15 SP3 and openSUSE Leap 15.3 should be binary and package compatible (but this effort is made by SUSE) same like CentOS was and now Alma Linux and company. Of course there is some openSUSE packages which are only available on that distribution.

Can one create that environment (Say SLES 12 SP4) with the build system linked above?

SLES seems to be available: SUSE SLE-15-SP3, SUSE SLE-15-SP2, SUSE SLE-15-SP1, SUSE SLE-12-SP5 and SUSE SLE-11 SP 4

[fauust]

Hi! This is ready for final review.

@vladbogo images have been pushed on quay.io (https://quay.io/repository/mariadb-foundation/bb-worker). I guess that you can now start to test those images.

Some images are missing:

• SLES;
• RHEL7 (arm64);
• Debian9 (ppc64le);
• Centos7 (ppc64le);
• others?

Maybe @vladbogo you want to give it a try? I suggest you only build the images you are working on by commenting the CI action file, see https://github.com/fauust/mariadb.org-tools/blob/bb-build-container/.github/workflows/bb_containers.yml#L30-L33.

Please take a look at the README.md, since moby folks don't want to add INCLUDE in Dockerfiles, the Dockerfiles need to be generated first with the common.Dockerfile file (this way we can avoid code duplication for buildbot user setup).

[illuusio]

@fauust Can find Containers from quay.io

[fauust]

Hi @illuusio!

The URL is in the commit:
https://quay.io/repository/mariadb-foundation/bb-worker

[fauust]

@vladbogo please can you do the final review and merge so you can start working on the new dockerfiles?
As discussed, rhel building is disabled for the moment.

[vladbogo]

Hi Faustin. Overall looks good. We will probably have a hard time trying to validate the RPM images. Maybe we can think of an automatic way to do that. Other than that I found some minor issues, but looks good. I also tested some of the images. Apart from the WORKDIR problem, they seem to be in a state where at least the build will not fail. Debian-9 seems to have some issues related to Python packages which I failed to debug, but I will look into it after the merge.