RFE: Re-think Kata Containers releases

[fidencio]

Folks,

I'm taking the time to write down something that's been going on on my mind for a while, and I'd really like to gather feedback from the @kata-containers/architecture-committee, @kata-containers/kata-containers-maintainer, and users of our project.

Kata Containers has had, in the past, the commitment to do one release per month on each maintained branch, be this release an alpha one or a stable one. In the past we used to maintain 2 stable branches, currently we've been maintaining only one.

However, the last release we had was 5 months ago, and the last backported commit to the stable branch is from 3 months ago. This is not because we haven't had content to be released, quite the opposite, we had content, but we didn't have people's power to actually commit to backport patches and cut a release.

With this said, and knowing that historically we've struggled to get our releases out of the door, I'd like to propose a few things here:

• Drop the stable branch
  • We're not able to maintain one, and this is basically just increase the burden whenever we have to cut a release
  • Downstreams are already maintaining their own stable branches, based on either one fixed release of Kata Containers (be it a stable or a non-stable one), and applying patches atop
    • We cannot fit the needs of all downstream customers with our stable branch, each downstream customer has its own need, and they know better what they need
• Considering we trust our CI ...:
  • Cut one minor release per month, cut one major release per year
    • This meanas that we'd cut:
      • 2024.02.0 -> in February
      • 2024.03.0 -> in March
      • ...
      • 2024.11.0 -> in November
      • Do not cut a release in December, as part of the community is off!
      • Then next year we'd have:
        • 2025.01.0 -> January
        • 2025.02.0 -> February
        • 2025.03.0 -> March
        • ....
        • 2025.11.0 -> November

It's importanta to say that if a CVE needs to be fixed, we'd be able to cut a YYYY.MM.n+1 release at any moment.
For instance, if a CVE shows up in January 2025, after our release, we'd be able to cut a new release, 2025.01.1 as soon as the patches land on the main branch.

Theoretically, this can help us to focus on running one simple script, which would get what we have passing CI, and consider this a release. This would also eliminate the burden to do the maintenance for stable releases (which we currently struggle quite hard to do anyways), and it'd also simplify a lot all the tools around the project (such as kata-deploy and other installers).

Please, I'd like to hear from everyone who's interested in the topic, and we can have a final discussion about this during the vPTG, but the sooner we start discussing this the better.

It's also important to note that what I have in mind for the release process is a GitHub action that would be manually triggered, and that would automagically do the release for us, including tagging, and everything else that's manually done Today.

[stevenhorsman]

So my two pence - I've not lead a release (I was signed up to shadow one, but there haven't been many to go around), but the think that intimidates me about it is the process to confirming all the things that need to be backported, backporting them and getting the CI passing (which should be much easier without the separate tests repo and after the GHA changes in fairness). So I can see clear benefits in this approach.

I don't have enough information to know if any users of the stable release would be impacted - but given that they haven't had anything to use since Oct 23rd, I don't think I've heard any complaints at AC meetings asking for more and to get involved in the release? I would like to hear more from those users though to find out more.

I personally have some concerns about our CI (our nightly hasn't passed for 5 months, but I think that's mostly due to non-required jobs failing), but it feels like any resource diverted from backporting to maintaining and fixing the CI might be a net positive for the project and I strongly agree that we should have the goal to maintain the main branch such that it is ready for a 'surprise' release at any point.

Also think the vPTG is the place to discuss this more and we've already put it on the agenda.

[ildikov]

Thanks @fidencio for writing up the proposal. As @stevenhorsman mentioned the vPTG, I added the link to this issue to the planning etherpad: https://etherpad.opendev.org/p/kata-ptg-planning-april-2024

[zvonkok]

I would love to see more frequent releases. We're adding so many new features per month that we should make them part of a release sooner rather than later. Downstream projects will add the needed features anyway because they were not released. Also, I agree with Steve about the state of the CI.
I've seen many flakes where folks just restarted those tests, again and again, to make it green, which wasted time on both sides, the CI machines' time and developers' time.
We should make main our stable branch, as suggested by Fabiano, targeting more of a rolling release, and there are enough
examples that this can work.

[gkurz]

Having led several stable releases, I agree with @fidencio that the backport effort to the stable branch is a huge burden. It requires a close inspections of a lot of PRs and manual backport of the ones that are considered worthy. It is error prone and often takes several rounds to complete.

@zvonkok we can certainly do rolling releases on main but it would not match the current definition of stable since the branch would get new features, not just fixes. I don't think we'd be able to release at any point in time.

Another option, currently experimented by https://github.com/openshift/sandboxed-containers-operator/ , is a so called "single stream" release process where the project has two branches:

• a development branch, which is the equivalent of our current main, where all the work happens
• a release branch that is only used to cut releases

When releasing a major or minor version, i.e. when content of the development branch matches what was planned for the release, it is merged to the release branch and the release branched is tagged. If some hot fix is needed between two releases, it can be fixed in the development branch and applied to the release branch. A fix release can then be cut.

With this model, the project only has one single version to support at any given time and is able to quickly release a hot fix.

[zvonkok]

Well, I am only concerned about another main, CCv0 thing.

Our goal should be that our main is stable and rock solid. Then it does not matter if you do point releases or rolling releases.

main could have rolling releases 2024.1 and 2024.2, ... fixes should go in first to main anyway and then be cherry-picked into a release, not the other way around.

Based on that, we could have additional "stable" releases if we have the manpower to do that, in a different release branch.

[fidencio]

@gkurz, I sincerely fail to see how much your suggestion differs from what we're doing, and it'd still require backports and people to work on both fronts, which is exactly the proiblem we currently have.

Let me expand here a little bit. The main problem we have is commitment from someone to actually go there and take the bullet to cut the release. By doing a stable branch and a development branch (which, from my point of view, is not much different from what we have), we'd still need someone to adapt tests and similar things in case a hot fix is needed on the stable branch, and the commitment to do so continues to be one the biggest problems which will not be addressed.

[fidencio]

Another thing that I'm suggesting is to NOT have a stable branch, from the original proposal:
"""

• Drop the stable branch
  • We're not able to maintain one, and this is basically just increase the burden whenever we have to cut a release
    Downstreams are already maintaining their own stable branches, based on either one fixed release of Kata Containers (be it a stable or a non-stable one), and applying patches atop
  • We cannot fit the needs of all downstream customers with our stable branch, each downstream customer has its own need, and they know better what they need

"""

[gkurz]

@gkurz, I sincerely fail to see how much your suggestion differs from what we're doing, and it'd still require backports and people to work on both fronts, which is exactly the proiblem we currently have.

This reduces the backport load to only hot fixes, e.g. mostly CVEs, where we want to release right away without risking any interference coming from the current development effort. Non-hot fixes would come with the next usual release, just like the rest.

Let me expand here a little bit. The main problem we have is commitment from someone to actually go there and take the bullet to cut the release. By doing a stable branch and a development branch (which, from my point of view, is not much different from what we have), we'd still need someone to adapt tests and similar things in case a hot fix is needed on the stable branch, and the commitment to do so continues to be one the biggest problems which will not be addressed.

The release branch isn't exactly a stable branch. No development happens there and it should ideally only be a collection of merges from the development branch. It doesn't require commitment from anyone to track all changes in the development branch and decide whether they are valuable fixes to be released. It just leaves an opportunity to be able to ship a critical fix if the need arises.

[fidencio]

It doesn't require commitment from anyone to track all changes in the development branch and decide whether they are valuable fixes to be released. It just leaves an opportunity to be able to ship a critical fix if the need arises.

It requires the commitment of backporting tests changes to cover the proposed fix, which may or may not be a big chunk of work depending on how much the CI changes. I'd not be willing to change what we have to add something that has the chance (and not a small one) to have exactly the same issues as we currently have.

There's also the commitment to be testing both branches and make sure that thing won't get rotten in the stable branch. That's also something we currently struggle to do, and I don't see how your proposal would improve it.

[jodh-intel]

The release branch isn't exactly a stable branch. No development happens there

I'm not sure that's going to be true. The dev branch is always going to be ahead of ― or very briefly at the same point as ― the release branch. But that means we have two options for handling legacy environments:

• Keep all the legacy code in the dev branch, which will accumluate cruft forever (ugh)

  We don't want to do this. And I'm not sure how we'd even test that: I guess we'd only (somehow) test the non-legacy code in the dev branch and potentially have the legacy/release tests bitrot until they are synced to the release branch yet?!?

• Add "legacy features" directly into the release branch (not from the dev branch)

  We don't really want to have to do this either, but would probably have to. But that would mean the branches would diverge, as happens naturally with our stable branches. But that implies we would still have the backport burden we have today since:

  1. The code from the dev branch won't necessarily merge cleanly to the release branch.
  2. We won't necessarily want all the features from the dev branch in the release branch.
     (For example, features that are "too new" (require newer hardware), or features that have only landed partially in the dev branch, or features that have landed but don't have any tests so it's unclear if they are of "release quality" yet).

[jepio]

Here's my take: lets first define "what is our stable API" and "how quick do we promise to be wrt CVEs".

CVEs: If we say - CVEs will be patched within 1 month and we do monthly releases then there is no problem - we ship CVE fixes with the next release. If we see that doing monthly releases is a bit too much and we switch to releasing every 2 months, then doing what @gkurz proposes makes sense (leave open the possibility of a security release based on the latest release).

Stable API: I think rolling, time based releases make the most sense for a most non-library project like kata. But we need to clearly define expectations for users/downstreams. Is the stable API the CRI interface or is it the kata-config file (I prefer including the kata config in the stability promise)? "Stable" != "no new features/fields" - it just means we don't break existing configurations, and handle additional features by having sane defaults.

How much work is a release currently, and how much work could it be with automation? Are there artifacts that need to be published to destinations that can't be automated?

[fidencio]

Let me touch in one specific part here:

How much work is a release currently, and how much work could it be with automation? Are there artifacts that need to be published to destinations that can't be automated?

Currently the release is partially automated, but backports to the stable branch are not. So, on every release someone would have to go through what's been merged in the last 4 weeks and do the backports, considering the release schedule would work as planned (which is not the case).

With what I'm proposing, the release will be fully done by a GitHub action that can be manually triggered, but apart from "click a button", nothing else is needed from the maintainer.

[studychao]

Drop the stable branch
We're not able to maintain one, and this is basically just increase the burden whenever we have to cut a release
Downstreams are already maintaining their own stable branches, based on either one fixed release of Kata Containers (be > it a stable or a non-stable one), and applying patches atop
We cannot fit the needs of all downstream customers with our stable branch, each downstream customer has its own
need, and they know better what they need

Hi @fidencio , I have some questions about this proposal.
Does this mean that we are no longer maintain a stable release with bugfixes, and it will be downstream's duty to maintain their own stable version upon our monthly releases?

[fidencio]

Does this mean that we are no longer maintain a stable release with bugfixes, and it will be downstream's duty to maintain their own stable version upon our monthly releases?

@studychao, exactly. And the reason I'm proposing this is because we're really not mainaining the stable release branch at all. Right now it's just a non maintained burden for releases, which IMHO gives the wrong impressions to the consumers of those branches.

[lifupan]

Hi @fidencio

First of all, I strongly support canceling maintenance on stable versions, which will save us a lot of porting bug fix patches. As an open source community product, it is reasonable for the downstream to be responsible for the stable version. But naming the version with the year and month, unlike the previous major version numbers such as 1.x, 2.x, 3.x, will it not be easy to distinguish some major features brought by each major version? So, can we still keep the previous version naming? Of course, this will indeed make automatic publishing more difficult.

[studychao]

Does this mean that we are no longer maintain a stable release with bugfixes, and it will be downstream's duty to maintain their own stable version upon our monthly releases?

@studychao, exactly. And the reason I'm proposing this is because we're really not mainaining the stable release branch at all. Right now it's just a non maintained burden for releases, which IMHO gives the wrong impressions to the consumers of those branches.

Thanks, I think that is good idea since it release our burden on maintaining a stable branch

[fidencio]

First of all, I strongly support canceling maintenance on stable versions, which will save us a lot of porting bug fix patches. As an open source community product, it is reasonable for the downstream to be responsible for the stable version. But naming the version with the year and month, unlike the previous major version numbers such as 1.x, 2.x, 3.x, will it not be easy to distinguish some major features brought by each major version? So, can we still keep the previous version naming? Of course, this will indeed make automatic publishing more difficult.

So, a few things to mention about this.

• It makes things slightly more complicated, as I can see that we'd have to have 3 workflows
  • One for major releases
  • One for minor releases (the most used one)
  • One for patch releases
• Ideally, on a time based release we wouldn't care much about the major releases, as features would be added monthly
• Documentation could solve the situation of "this feature was introduced in this release"

With all that said, keeping the versioning number we have is still do-able and still a big improvement just for dropping the stable branch, and I think we can make this work.

WDYT?

[lifupan]

First of all, I strongly support canceling maintenance on stable versions, which will save us a lot of porting bug fix patches. As an open source community product, it is reasonable for the downstream to be responsible for the stable version. But naming the version with the year and month, unlike the previous major version numbers such as 1.x, 2.x, 3.x, will it not be easy to distinguish some major features brought by each major version? So, can we still keep the previous version naming? Of course, this will indeed make automatic publishing more difficult.

So, a few things to mention about this.

• It makes things slightly more complicated, as I can see that we'd have to have 3 workflows

  • One for major releases
  • One for minor releases (the most used one)
  • One for patch releases

• Ideally, on a time based release we wouldn't care much about the major releases, as features would be added monthly

• Documentation could solve the situation of "this feature was introduced in this release"

With all that said, keeping the versioning number we have is still do-able and still a big improvement just for dropping the stable branch, and I think we can make this work.

WDYT?

Hi @fidencio

If we use time-based version numbers to release, will the 4.0 version that we are all anticipating or planning now cease to exist? If this is the case, do we need to plan at the beginning of each year what major features will be released in the coming year?

[fidencio]

If we use time-based version numbers to release, will the 4.0 version that we are all anticipating or planning now cease to exist?

No, that's not accurate. We do time based releases on the minor till we decide that we need a major one.
So, we'd be cutting 3.3.0, 3.4.0, 3.5.0, ..., 3.17.0, 3.18.0. If we agree that at 3.18.0 we're good to cut a major, then we start with 4.0.0, 4.1.0, ..., 4.16.0, ...

@lifupan, does it sound reasonable?

[wainersm]

Hi all! First of all, thanks @fidencio for bringing up this topic.

My personal opinion, not necessarily of my employer, I agree with the proposal of dropping the stable branch as we are clearly over-committed. It was implemented an automation to automatically backport pull requests which wasn't broadly advertised and isn't maintained. Even if we fix that automation, I still believe that downstream consumers should keep stable branches (except for CVEs, that we own them).

I wanted to go back to @jepio 's comment on stable API, related with @lifupan 's point on time-based versioning: inevitably at some times we will have to break the backward compatibility (API and configuration files). How to proper advertise those breakages with time-based versioning?

[fidencio]

How to proper advertise those breakages with time-based versioning?

This is a very good question, and the answer is orthogonal to whether we do this change or not.
What's the current approach to that? We don't have one very well defined, and we need to do so.

[fidencio]

Just a summary from the AC meeting from February 20th.
I presented this proposal there, and you can find details and slides used as part of https://etherpad.opendev.org/p/Kata_Containers_2024_Architecture_Committee_Mtgs

Copying and pasting from the etherpad:

• https://docs.google.com/presentation/d/1138SejC6Q0IC8eZftMTZ3q5ATS-5XjtQ-vzNmOPJ0AU/edit#slide=id.p
• Agreements from the AC:
  • Fabiano Fidêncio (the one who proposed the idea): yes
  • Peng Tao: yes
  • Fupan Li: yes
  • Steve Horsman: yes, with the caveat of waiting one more week so other folks can jump into the proposal
  • Greg Kurz: asked for one week to forward this inside Red Hat and have a conversation with his team
  • Jiang Liu (Gerry): yes -- EDITED, based on Gerry's comment: RFE: Re-think Kata Containers releases #9064 (comment)
  • Samuel Ortiz: yes -- EDITED, based on Samuel's comment: RFE: Re-think Kata Containers releases #9064 (comment)
• Points raised from the community:
  • Wainer raised the need to proper align this with Confidential Containers
  • Steve raised the need of a well defined process when a major change that leads to change in behaviour
  • Archana raised the importance for some customers to rely on a stable branch provided by the community
  • Archana raised the idea of actually looking for volunteers instead of dropping the stable branch

[wainersm]

How to proper advertise those breakages with time-based versioning?

This is a very good question, and the answer is orthogonal to whether we do this change or not. What's the current approach to that? We don't have one very well defined, and we need to do so.

Yes, agreed. And it seems we will stick with the semantic versioning, so this (breakages) should be discussed in another occasion.

[sprt]

I agree dropping the stable branch and having a consistent release cycles would make things easier for downstreams.

Following up on @jepio's question: how exactly do we address hotfixes/CVEs with this approach? I think we'd still want to branch out of main in those cases, otherwise we'd end up releasing more changes than just the fixes.

[stevenhorsman]

how exactly do we address hotfixes/CVEs with this approach? I think we'd still want to branch out of main in those cases, otherwise we'd end up releasing more changes than just the fixes.

I think the proposal is that we would release more changes than just the fix (hence no patch releases any more, only minor ones), with the thought being that if our CI is good enough (which is a big if) then there shouldn't be extra risk with a maximum of ~24days of code going into the release.

Based on the discussion from the AC call, my understanding is that under this proposal, if people want to just have CVE fixes on top of a previous release then they've have to fork it downstream and handle that.