Branch management, distro support and CI

[leseb]

Dear contributors,

@theanalyst ,@walkerlee ,@mbessler ,@minshenglin ,@bnoon ,@a-nldisr ,@rochaporto ,@BugRoger ,@tobilg ,@tobegit3hub ,@xcompass ,@nazar-pc ,@lorieri ,@CoRfr ,@fmeppo ,@TelanJ ,@hookenz ,@thecanadianbaker ,@rootfs ,@h0tbird ,@streamnsight ,@hunter ,@dmitryint, @Ulexus, @jimcurtis

Many users reported this issue already and finally got the time to start addressing it. We don't have any proper release management. We are missing a couple of things:

• No branch management
• No CI

Branch management

If we take a little step back, and for those of you who are not familiar with the repo entirely. We are only putting efforts and ceph/base and ceph/daemon.
Why do we need branches? We need to support several distributions and Ceph versions as well. However one of the main issue we are facing is the Docker Hub. Basically each branch in our git repo reflects one image and we don't have the ability to do some templating with the Dockerfile. We explicitly need to set values and can not do any sort of conditions (if distro then etc etc). So for each OS and each Ceph version we need to modify those:

• https://github.com/ceph/ceph-docker/blob/master/base/Dockerfile#L5
• https://github.com/ceph/ceph-docker/blob/master/base/Dockerfile#L10

All the installation steps: https://github.com/ceph/ceph-docker/blob/master/base/Dockerfile#L24-L28
And eventually modify: https://github.com/ceph/ceph-docker/blob/master/daemon/Dockerfile#L7 accordingly. This is really painful and not practical to manage branches with all of that, spending time to resolve conflicts every time we are trying to merge master into a branch.
So because of all of that we need to arrange our own workflow.
After discussing with one of my colleague (@ErwanAliasr1), we ended up with what I believe could be a pretty good way to manage our branches.

The basic idea is to manage everything from the master branch, so would have several files in the base directory with this format Dockerfile-<ceph version>-<distro>, ie Dockerfile-infernalis-ubuntu.
Then every time someone wants to PR something it will end up in master.
After this, repo's owner (and later a CI) will execute a script that will:

• create branches like this: <ceph-version>-<distro>, ie: infernalis-ubuntu.
• push each Dockerfile-<ceph version>-<distro> into their respective branch by simply copying it to `Dockerfile``

This will then later trigger a new build on the Docker Hub. Then when the next PR comes and gets merged the script will delete those branches and redo the operation above.

In order to not disrupt production deployments, I think we need to keep the default Dockerfile on Ubuntu so we don't disrupt production setups by generating a new image. So initially we will have the following Dockerfiles:

• Dockerfile, default to Ubuntu
• Dockerfile-hammer-ubuntu
• Dockerfile-infernalis-centos

Btw, I think this might be a good opportunity to start killing some branches (like Firefly). Well it's up to you guys, let us know which image version you're using and we will do our best.
A really important point is to never ever touch these branches as they are ephemeral til the next PR merged.

CI

We don't have a CI and this means we have no guarantee that some PR won't screw the next image, then when someone pulls it and runs it this might be broken.
So having a basic CI to do a couple of Docker runs and then a docker exec <id> ceph -s would be extremely useful.
@alfredodeza, @andrewschoen, is there any chance to plug the Ceph CI here? If not, @jimcurtis do you think this is something your team could handle?

Before I start doing anything, I'd like to get some inputs, just to be sure we are heading in the right direction.
Sorry for the long post and thanks for reading!

[ErwanAliasr1]

As a consequence, the "infernalis-ubuntu"-like branches are just temporary branches used to pass the DockerHub build. That's just a trick to workaround the very few features of that tool.

Creating and submitting that temporary branches could be easily automated.

[tobilg]

Thanks @leseb! Just another question: As you wrote nothing about tags, I'd like to know whether there will be tags for the Docker image. If not, I guess it will lead to the same "potentially broken images" problem...

This was exactly what #204 was saying...

[leseb]

@tobilg sorry I forgot to mention tags but yes there will be tags for sure, definitely :).
And yes this thread is also aiming to solve #204.

[ErwanAliasr1]

As all the code is hosted in master, tags are applied to all flavors (distro & versions)

[tobilg]

That's great news, thanks for your effort. I guess every new tag must be specified then in the image's Docker Hub build settings if it's an automated build AFAIK...

[BugRoger]

Awesome. Good news. Looking forward to getting rid of my own images 😄

[Ulexus]

@leseb @ErwanAliasr1 Good ideas and thanks!

Tags may still be a problem, though. While the intention of a tag is to have a static image pushed to Docker Hub, I have found it (in the past, at least) to be all too easy to accidentally trigger a rebuild of an already-built tag when using automated/trusted building. I have definitely noticed changes and optimizations in Docker Hub in the last year or so, though... this may not be an issue anymore.

The Docker Hub docs are particularly mute on the semantics for tags and automatic rebuilds, so experimentation is likely the only way to find out.

[ErwanAliasr1]

The tagging scheme will insure keeping a single build per release tag.

[nazar-pc]

I'm 👍 on approach. Travis CI supports Docker, so might be a solution to consider.

Also about different OS flavors: are they really needed? I mean official images are usually built on Debian Jessie and nothing else. Why both Ubuntu and CentOS needed?

I guess more people are familiar with Ubuntu and considering Docker image as standalone thing one flavor might be enough. Since you'll not do a lot of changes in it when extending (I mean installing bunch of software and stuff), so there is no big practical benefit from having multiple flavors.

[fmeppo]

I'm not a fan of the proposed approach. I very much want branches for each major Ceph release - knowing that time will be spent resolving merge conflicts. Simply renaming Dockerfile to Dockerfile- isn't sufficient; we'll also have conflicting versions of entrypoint.sh, config.kv.sh, etc. Yes, we can put logic in each to determine what release is being used....but at that point, we're making extra complexity to avoid having to use branches. I'd rather a small amount of pain on a rebase than adding stacks of case/esacs whenever I try building something specific to a particular Ceph release.

Practical example: I'd like to open up a Jewel branch. I want to add support for BlueStore, which adds new keys to the KV store, probably a new env var for the entrypoint, and a bunch of extra partitioning logic. Jewel also incorporates a patch I made to ceph-disk a while back, so I'd drop the patch package from ceph/daemon and delete the ceph-disk.patch file entirely. If everything lives in master, then building this logic gets complicated (or we have entrypoint.sh-infernalis and entrypoint.sh-jewel, and the next change someone proposes has to be duplicated...) If we use branches, that's easy - just make changes in master, and only backport to earlier releases if necessary (and possible).

So far, Docker Hub's autobuild seems to only be triggered when I push a branch - and only rebuilds the branch in question. Seems simple enough to build tagged container releases that effectively don't change (unless we backport a bug fix from master) - and that's exactly what I'd want for production. Assuming I'm not missing anything, we'd just need to write a "how to upgrade" section in README.md, define a "latest" tag I can put on my alpha-test rack (no guarantee of stability there), and we're set.

Regarding OS choice, I whole-heartedly agree with @nazar-pc - there's no real benefit in having multiple OS flavors. I'm following ceph-docker because I want an appliance in production; I really don't care what distro is used, only that things work (and don't change) on a couple hundred machines.

[nazar-pc]

Well, thinking more about it, I'd suggest to support single Ceph version (current stable) on single OS flavor. Lets look at Docker itself: when new major version released - developers offer upgrade guide and do not release any patch versions for previous releases anymore.

Having only 1 configuration would simplify dramatically maintenance complexity comparing to matrix of configurations. Also single version will get more attention and testing, which (hopefully) will result in better overall quality or at least bugs will be found much faster.

[leseb]

@fmeppo, @nazar-pc thank you for your feedback. Let me separately address all your questions.

Structure

I don't think it is an issue to create a directory tree such as:

.
├── hammer
│   ├── centos
│   │   ├── base
│   │   │   ├── Dockerfile
│   │   │   └── entrypoint.sh
│   │   └── daemon
│   │       ├── Dockerfile
│   │       └── entrypoint.sh
│   └── ubuntu
│       ├── base
│       │   ├── Dockerfile
│       │   └── entrypoint.sh
│       └── daemon
│           ├── Dockerfile
│           └── entrypoint.sh
└── infernalis
    ├── centos
    │   ├── base
    │   │   ├── Dockerfile
    │   │   └── entrypoint.sh
    │   └── daemon
    │       ├── Dockerfile
    │       └── entrypoint.sh
    └── ubuntu
        ├── base
        │   ├── Dockerfile
        │   └── entrypoint.sh
        └── daemon
            ├── Dockerfile
            └── entrypoint.sh

Since a simple script will be managing all the workflow, I don't think it is a real issue to add the management of the entrypoints too as it also saves us (@Ulexus and I) the burden of managing all the branches and resolving all the conflicts. But no worries @fmeppo there is no issue creating branches anyway. If we take back the Jewel and BlueStore example. I'll never say no to you pushing a PR for another branch or work. If people need to collaborate on a branch it is fine. Then as soon as the work is done we can merge the work into master and there won't be any conflicts since the structure will be identical.

Multi-distro support

I can feel we are not on the same page here but let provide you more arguments on why supporting multiple OS is important.

First and foremost this is the open source world in general, there are multiple distros and different interests. Sticking with a single OS (Ubuntu) might sound easier but in the end it is not since the promise of the new repo structure is to keep things as is and just add the ability to plug other distributions, so once again Ubuntu will remain the default. So from your perspective (pure user) this will not change anything.

On the other side, companies (like my employer Red Hat) or Suse or other might have interest in the work we have been doing. So allowing them to bring the support for their own distro is vital. It will allow us to get more traction and more contributors. If we don't do this and stick with one distro, we will lose that and you might lose some of my contributions too.

I don't want to end up in the situation where people are tempted to fork our repo and do their own stuff because they feel this can't be done here. This will be a waste, once again from my perspective, we have been putting a lot of efforts @Ulexus and I on this repo and I don't want to be in a position where I have to decide where to put my work (internally for Red Hat downstream product) or here in upstream. Usually the best path for companies is to invest in the upstream and then eventually get a downstream product out of it.

Multi-distro support is also about trust and confidence in everyone favorite distributions.
Finally distro specific stuff are only part of ceph/base now, so the changes are rather small.

I'm just trying to align everyone here and closing the door of external distribution contributions is IMHO a bad idea. So I believe sticking with the initial proposition is likely the best way to handle broken images and general repository management.

[nazar-pc]

Well, we definitely have different opinions on OS flavors. For me Docker image is like binary. You do not generally select language binary was originally written in - you just take it and use it as is. The same here, images based on different OSs will works exactly the same (and if not - it is a bug), so I do not see any real benefit for Red Hat or whoever else to care what exactly OS it is based on - it should just work according to documentation as @fmeppo said.

It is up to maintainers to choose, obviously, but I personally prefer avoid any scripting and automation if we can avoid problem in the first place. And this specific case seems pretty achievable.

[dang]

It's about support for the containers. Red Hat, for example, can't and won't support a docker container based on Ubuntu. Neither should Canonical support one based on Fedora or SuSE. As far as a user is concerned, if the want to support it themselves, then it may not matter; then again, it may. If I have to self-support, I probably want a distro I'm familiar with, so that I can debug it. If, however, I want to pay for support, I must have a container that my support organization will be willing to support.

[nazar-pc]

Red Hat, for example, can't and won't support a docker container based on Ubuntu. Neither should Canonical support one based on Fedora or SuSE.

Lets have only CentOS image then. I mean there is no difference. There is no point to support many of them, otherwise let's add images based on Debian, Fedora, Alpine and other distros.

[dang]

I'm planning on submitting a Fedora image. I'm sure if you want to support a specific distro, it'll be fine.

[ErwanAliasr1]

Cool looks like we get a plan !

[ErwanAliasr1]

@nazar-pc our argument about distro support is that commercial distro like Suse/Redhat are providing support to their customers and this is what they mostly buy. In such case, they need to create containers based on their packaging which guarantee the tracability & the support to the product.
Every vendor can build the product with different configure, compilers, libraries which ends up with a different binary even if its from the same base.

As a user, I perfectly understand your position : ceph is ceph whatever is the integrator. But for those who distributed & support their clients, that's very important to keep that tracability of "what have been used/installed where".

So if we want to keep the support of thoses distro, which provides man power & ressources, it's better having the OS flavor choice in our project. That doesn't cost us anything, even more will clarify the workflow.

[theanalyst]

I also agree with @leseb here, +1 for multi-distro support, and supporting the distros upstream here in this repo rather than maintaining a downstream fork for our distros etc.

[leseb]

Awesome, it looks like we have a plan.
Thanks everyone for your comments and feedback, I'll start a prototype next week.

[ErwanAliasr1]

@leseb ping me if you need help on that

[leseb]

@ErwanAliasr1 will do thanks!

[leseb]

FYI I'll submit a patch today.

[leseb]

Here we go: #259

[leseb]

We are almost done, I had to fix a couple of issues with the builds, it's currently building and should be done soon, I'll you know the outcome.

[leseb]

It is all green now. If someone wants to have a look and test, it'd would be nice.
For the complete list of tags, please see: https://hub.docker.com/r/ceph/daemon/tags/

[leseb]

Last update, I think we are good to go now, we just fixed something with @ErwanAliasr1.
We now have a solid build system, so we won't be accepting any more contributions to the several branches but will recommend to use our tags.
I'm going to update the README for the new contributions and rules to follow.

[dang]

So, how are supposed to build non-ubuntu images locally? docker build will not accept symlinks. I can manually resolve all of those; is that how I'm supposed to do it?

[leseb]

Hum we could build a script to generate a dev env from ceph-releases's content. What do you think?

[dang]

It would make contributions much easier, that's for sure.

[leseb]

With this last message, I'm happy to tell everyone that we know have a proper branch strategy with tags. We currently have the following branches available:

build-master-hammer-centos-7
build-master-hammer-ubuntu-14.04
build-master-hammer-ubuntu-16.04
build-master-infernalis-centos-7
build-master-infernalis-ubuntu-14.04
build-master-infernalis-ubuntu-16.04
build-master-jewel-centos-7
build-master-jewel-fedora-23
build-master-jewel-ubuntu-14.04
build-master-jewel-ubuntu-16.04

All the images will soon be available through the Docker Hub (currently building).

We also have a CI running on Travis that does a great job.
As a side note these are the important recent patches: #277 #281 #279 #278.

I'd like to thank everyone for your feedback and particularly @ErwanAliasr1 who initially came up with the idea and helped a lot of the script implementation.

Without further ado, I'm closing this thread 💃

[ErwanAliasr1]

My pleasure to help you @leseb