SEP 40: LTS releases

[dwoz]

No description provided.

[OrangeDog]

Each LTS release will be supported for N years, but how frequently will LTS releases be made?
In order to facilitate reasonable a reasonable upgrade timeline, there should be some overlap.

How will non-LTS releases work? Continuing (or returning to) a release every six months, and support only the current and previous release?
Will bugfix releases on these be even less frequent (3004, for example, still hasn't had a bugfix release after six months, despite multiple regressions) given the increased number of active branches?

Ending support when the OS does is fair, as long as you keep timely releases, and don't have a bunch of fixes saved up only to miss the window.

How will backport candidates be identified, and will the core team commit to doing them all within a certain time period?
Ideally all backports would be part of the original merge process and not have to wait.

I consider Ubuntu to have the best (for the consumer) LTS support at the moment. That's an LTS release every two years, and each one has free support for five years. Non-LTS releases every six months. The security support lasts longer than the bugfix support. OpenJDK is similar. That's a big commitment, so you don't have to go that far.
Jenkins is another good project, with a more frequent but still fair LTS release model.

[ggiesen]

I agree with you on the Ubuntu model being a great one, but I don't think it's realistic. Something more realistic might be LTS every 2 years with 3 years of support. That being said, I'm not sure that's even attainable at this point as it seems the Salt core team may be struggling even under the current model. Would have to wait for someone from Salt core team to weigh in.

[takeda]

Just a feedback from a random guy. While having LTS releases would be great, but I think following things should be solved before LTS is introduced, as these things will diminish the work put in maintaining it.

Some issues I noticed:

• changes are made without understanding what is being fixed (i.e. trying things blindly) for example saltstack/salt@070597e which was later reverted Remove max zeromq pinned version due to issues on FreeBSD salt#61163
  • a solution perhaps should be to clearly define what is being solved then verify that change indeed solves the issue before merging it. Ideally (I don't think this example is good) include an unit test for the mentioned bug to prevent regression. Although that can have its own issues.
• introducing other unrelated changes in the same commit (like refactoring code that didn't need to be refactored for the bugfix). I don't have example right now, but I saw that the same bugfix was also including removal of Python2.7 code. You can include it in a PR, but at least make it in a separate commit. It would make it easier for you to cherry-pick fixes to supported releases
• there are some weird repo changes (perhaps due to rebasing?), for example if you select 3004.1 release git will warn you that commit is outside of the repo. I guess it likely happens, because when fix is made someone checks up the tag, and then add a commit to it and creases next release. This is how other projects handle maintenance of releases:
  • when release is about to happen a branch is created (for example v3004)
  • a release candidate can be invoked, if there are fixes needed they are included
  • at one point the branch is being tagged with the release version and released
  • if there are issues like for example CVE the fix is applied to master (following practices mentioned above i.e. ideally fix should be just in a single commit, and contain nothing else but the fix)
  • then you go to the release branch and cherry-pick the fix, and resolve any conflict if you have any (although if you do it this way, git will typically work with you and not against you and might surprisingly handle it well even if things significantly changed)
  • create new tag in that branch and do a release

At that point you can designate certain branches as LTE. If you do LTE without the other stuff you will constantly be introducing regressions. And it will be a lot of effort and still not much benefit to it.

Another thing, and I mentioned this in another ticket. I believe is salt should migrate from setup.py + bunch of of custom code + many different requirements.txt to something that is declarative and doesn't allow custom code. I personally like poetry, but others likely will work of to (even using setuptools with specifying all dependencies in setup.cfg should be better).

The way things work appear to cause a lot of burden to anyone make changes. As there are many places that need to be updated. The custom setup.py also appears to have custom parser of requirements.txt (which BTW seems to have a bug and appears to think that python 3.10 is older than 3.9. Essentially telling me that there's no pyzmq version that can be used).

I believe that if there was a single file specifying all supported version ranges, and an environment markers should satisfy platform/python specific dependencies.

[waynew]

Ubuntu probably does have the gold standard for LTS releases -- but they also are a much much larger endeavor than Salt 🙂

One thing that we've seen with a not-insignifcant-number of Salt users is that they're still on significantly outdated versions of Salt. I honestly don't think that an LTS will change that -- that's an organizational problem. An LTS might help them out - and having an overlap of N months may actually provide them with enough of a window to be able to upgrade.

Another thing, and I mentioned this in another ticket. I believe is salt should migrate from setup.py + bunch of of custom code + many different requirements.txt to something that is declarative and doesn't allow custom code.

FWIW, we actually have to do that, since distutils is deprecated and will be removed in Python 3.12, which is rapidly approaching. We are already tracking that. In fact, I jumped straight to a setup.cfg based approach and at least the minimal testing I did seemed to work just fine. But I realize that I'm just one data point and there are many, many more salt users out there.

---

Thanks everyone so far for your feedback! This gives us a lot of things to consider already. If anyone has any points about LTS models that haven't been brought up yet, do please drop them here!

[OrangeDog]

significantly outdated versions of Salt

Probably because of the large number of regressions every release. For example, there is still no 3004.2, and there were some very major regressions in 3004.1. Everyone that uses syndic or TCP had to either stay on a vulnerable release, or patch it themselves.

If there was an LTS branch, it might actually get bugfix releases.

[Oloremo]

If there was an LTS branch, it might actually get bugfix releases.

To be fair bugfix policy was never clarified for the current release process and it's not yet clarified for a new one.

It's unclear what Salt considers bugfix-worth and what is not.

[Oloremo]

It's also unclear how the new process handles the the original "backporting hell" issue that leaded to the current release process

[OrangeDog]

The proposal reads like they assume an LTS system would be less work for the core team, when actually it should be more or it won't work.

[dmurphy18]

@OrangeDog If you take the example of the recent CVE Release for 3002.8, 3003.4 and 3004.1 verses the LTS branch, then that is a reduction where only one branch has to be stabilized with tests green etc. Getting 3002.8 into a stable state for tagging was difficult due to its age. Definitely a reduction in work effort.

[OrangeDog]

only one branch

At least two - the master branch and the LTS branch. That's assuming you also aren't supporting the latest non-LTS release.

Four branches if it's during an upgrade period - master, latest release, latest-LTS, previous-LTS.
So exactly the same number.

[max-arnold]

In this SEP I'd expect to see a retrospective of the previous release process overhaul: https://github.com/saltstack/salt-enhancement-proposals/blob/master/accepted/0014-dev-overhaul.md

• Previous goals (stability, number of supported versions, what should be backported, release cadence, etc)
• What worked and what didn't (in terms of selected solutions to achieve the goals)? Why?
• New goals and how they compare to the old ones. Why the goals had to change?
• How the proposed solutions will achieve the new goals?

A biggest downside of the discussed SEP is that it lacks explicit description of user needs and how they will be met. By users I mean not only regular users, but also companies who drive major features (VMware itself, SUSE, etc). So far it looks like it is only focused on how to reduce the efforts to maintain Salt (which is a totally valid point of view), but it is not clear how the proposed changes will affect users. And I truly believe that user needs should be the primary focus, otherwise there would be no community and no point in maintaining open source software for it (and the efforts will be ultimately reduced to zero).

Below I'll try to summarize my experience with Salt releases over the years:

Pre Neon 3000

• [+] A version support period was longer (subjectively)
• [-/+] Major features were sometimes merged into minor releases, often driven by companies that heavy rely on Salt (i.e. utilize it in their own products, or have very large installs and thus have some influence on SaltStack Inc.)
• [+] It was easy to push a non-security bugfix into an old version
• [-] Due to insufficient testing and merging of new features, Salt had major regressions in minor versions
• [-] Release cadence was unpredictable

Post Neon 3000

• [-] A version support period is shorter (feels like a non-stop major version upgrade process)
• [+] Major features are only merged into major releases
• [-] Non-security bugfixes are an exception. It is a pain to wait for a fix in a module to be included into a next major version
• [-] Regressions introduced in a major version are typically fixed only in a next major version, which means that every major version is regressed in some way. Examples [salt-api] 3002 Multiple args in yaml formated requests are ignored salt#59182 [BUG] On 3003 grains.get returns an empty string when called from minion's id_function salt#60219
• [-] Even regressions introduced by a security bugfix sometimes take a lot of time to get fixed. It is been two months and the tcp regression is still not fixed [BUG] TCP transport broken on 3004.1 and relevant bugfixes salt#61865
• [+/-] Release cadence was predictable, until recently

What I'd like to see:

• I'm fine with 2-3 major releases per year (esp. because major things can be shipped through salt extensions much faster)
• Non-security related bugfixes in minor versions (mandatory bugfixes for introduced regressions, with some room for other types of bugfixes)
• Longer support intervals (2 years minimum, ideally more). Maybe like Django? https://www.djangoproject.com/download/#supported-versions
• Extended support interval for security patches (4 years?)

From a more high level perspective that means:

• I can install a major Salt version on any mainstream distro, set it to auto-update (within minor bugfixes) and forget about it until my Linux distro is EOL. This means having security updates, minor version stability, and no need to upgrade to a major version to fix any regressions
• Reasonably good pace of innovation in terms of new features. I'm ok with upgrading to a major version to get them

[OrangeDog]

Further to a retrospective, see also the discussion on the previous process change: #20

[dwoz]

Each LTS release will be supported for N years, but how frequently will LTS releases be made? In order to facilitate reasonable a reasonable upgrade timeline, there should be some overlap.

How much overlap do you feel is reasonable?

How will non-LTS releases work? Continuing (or returning to) a release every six months, and support only the current and previous release? Will bugfix releases on these be even less frequent (3004, for example, still hasn't had a bugfix release after six months, despite multiple regressions) given the increased number of active branches?

Non-LTS releases would be feature/bugfix combined release and ideally would happen regularly (Like monthly).
Bugfix releases for LTS would also ideally happen regularly.

There are multiple factor effecting the core teams' ability to release more frequently. One of the most significant of these obstacles is our current packaging / testing of packages process. This process can take anywhere from several days to almost a month depending on what kind of release we're doing. CVE releases take longer right now due to the number over versions being released. We currently have a project slated for 3006 that aims to resolve our packaging problems. The goal of this project is to completely automated our package builds and tests.

Ending support when the OS does is fair, as long as you keep timely releases, and don't have a bunch of fixes saved up only to miss the window.

Agreed, release velocity and frequency needs to be sped if we have any chance of making this work.

How will backport candidates be identified, and will the core team commit to doing them all within a certain time period? Ideally all backports would be part of the original merge process and not have to wait.

I think it'll be up to the developer fixing a bug to know where it needs to be fixed and open the appropriate PRs on any branches where it's needed. This, IMO, is needed to prevent merge forward hell. I don't think anything should be 'backported' from master to LTS. The only thing going in LTS is bug-fixes and if a bug exists in both LTS and master, you'd be obligated to open a PR that resolves (and tests for the resolution) in both branches.

I consider Ubuntu to have the best (for the consumer) LTS support at the moment. That's an LTS release every two years, and each one has free support for five years. Non-LTS releases every six months. The security support lasts longer than the bugfix support. OpenJDK is similar. That's a big commitment, so you don't have to go that far. Jenkins is another good project, with a more frequent but still fair LTS release model.

I think we'll be aiming for something a little more frequent.

[dwoz]

I agree with you on the Ubuntu model being a great one, but I don't think it's realistic. Something more realistic might be LTS every 2 years with 3 years of support. That being said, I'm not sure that's even attainable at this point as it seems the Salt core team may be struggling even under the current model. Would have to wait for someone from Salt core team to weigh in.

We are struggling and there are multiple factors involved in why. As I mentioned in my last reply, One reason we're struggling is the amount of time it's taking to package Salt. Another reason we've been struggling is because we have not been consistently running and monitoring tests on older branches. We need to fix both of those problems regardless of our release / support timeline.

Part of the motivation for this SEP is to limit the number of versions we're supporting at any given time. Which would intern limit the number of test suites we're running and having to monitor. When we made the change to the 3000.x release model we intended to increase velocity and we were able to slightly improve there, but that model breaks when we start looking at CVE releases. The primary realization that we've come to is that we absolutely need to keep tests running and monitored on older versions. Another, is we absolutely need to automate package builds and testing.

[dwoz]

Just a feedback from a random guy. While having LTS releases would be great, but I think following things should be solved before LTS is introduced, as these things will diminish the work put in maintaining it.

All feedback is welcome, thanks.

Some issues I noticed:

• changes are made without understanding what is being fixed (i.e. trying things blindly) for example saltstack/salt@070597e which was later reverted Remove max zeromq pinned version due to issues on FreeBSD salt#61163

  • a solution perhaps should be to clearly define what is being solved then verify that change indeed solves the issue before merging it. Ideally (I don't think this example is good) include an unit test for the mentioned bug to prevent regression. Although that can have its own issues.

I think this type of thing is the exception, not the rule. All of the engineers on the core team know it's important to try and fully understand the issue before making changes. Issues with dependencies can be tricky, especially with the amount of OSes and OS versions we support. It should also be noted this is why we run the tests on so many OSes and platforms. Looks like this particular issue happened before we had FreeBSD in the pipelines.

• introducing other unrelated changes in the same commit (like refactoring code that didn't need to be refactored for the bugfix). I don't have example right now, but I saw that the same bugfix was also including removal of Python2.7 code. You can include it in a PR, but at least make it in a separate commit. It would make it easier for you to cherry-pick fixes to supported releases

This was an issue, yes, and is a case where we realized the errors of our ways. We've since put all of the syntax upgrades into a single PR.

• there are some weird repo changes (perhaps due to rebasing?), for example if you select 3004.1 release git will warn you that commit is outside of the repo. I guess it likely happens, because when fix is made someone checks up the tag, and then add a commit to it and creases next release. This is how other projects handle maintenance of releases:

  • when release is about to happen a branch is created (for example v3004)
  • a release candidate can be invoked, if there are fixes needed they are included
  • at one point the branch is being tagged with the release version and released
  • if there are issues like for example CVE the fix is applied to master (following practices mentioned above i.e. ideally fix should be just in a single commit, and contain nothing else but the fix)
  • then you go to the release branch and cherry-pick the fix, and resolve any conflict if you have any (although if you do it this way, git will typically work with you and not against you and might surprisingly handle it well even if things significantly changed)
  • create new tag in that branch and do a release

This is something that has happened as of the last few release and is a process thing. To be clear, we do have a release checklist but it likely needs to be updated.

At that point you can designate certain branches as LTE. If you do LTE without the other stuff you will constantly be introducing regressions. And it will be a lot of effort and still not much benefit to it.

Another thing, and I mentioned this in another ticket. I believe is salt should migrate from setup.py + bunch of of custom code + many different requirements.txt to something that is declarative and doesn't allow custom code. I personally like poetry, but others likely will work of to (even using setuptools with specifying all dependencies in setup.cfg should be better).

Some of this is being worked on right now as a part of the distutils deprecation work.

The way things work appear to cause a lot of burden to anyone make changes. As there are many places that need to be updated. The custom setup.py also appears to have custom parser of requirements.txt (which BTW seems to have a bug and appears to think that python 3.10 is older than 3.9. Essentially telling me that there's no pyzmq version that can be used).

I believe that if there was a single file specifying all supported version ranges, and an environment markers should satisfy platform/python specific dependencies.

The complexity here stems from us supporting system package versions for our dependencies. It vastly complicates things. We are aware of the issue and are working on initiatives to help resolve this.

[waynew]

Re: the setup.py issue, I have that effort slated for Sulphur/3006. I basically am planning to (re)start tackling that as soon as Phosphorus is released. If you want to follow along with those efforts you can track them here: saltstack/salt#60476 - my initial experiments (that are in no way comprehensive) are promising 👍

[dwoz]

only one branch

At least two - the master branch and the LTS branch. That's assuming you also aren't supporting the latest non-LTS release.

Four branches if it's during an upgrade period - master, latest release, latest-LTS, previous-LTS. So exactly the same number.

Master and Latest Release are one branch here. Latest LTS and previous LTS are two.. By my count that is 3 branches. With our currently states relase/support cycle, We end up with 5 supported branches at times.

[OrangeDog]

Master and Latest Release are one branch here

So you're going to stop supporting the latest non-LTS release with security fixes?

[dwoz]

Master and Latest Release are one branch here

So you're going to stop supporting the latest non-LTS release with security fixes?

No, but all non-lts release would be able to have both features and bugfixes. So, if we do a security release, the fixes would go into master and and we'd cut a release with anything else on the head of master. The logic here is that if you are following the master release you expect to have features rolled out with bugfixes, otherwise you can use LTS.

[ITJamie]

what specifically is meant by non-lts branch here? are we suggesting that builds of the "main" branch might not get a re-release at the same time as the "LTS" branch if a cve came out?

What about users who want to be on "latest"

[dwoz]

I have added a diagram to hopefully show what we'll be doing better. Security releases are a priority. When we encounter a CVE all supported releases get a patch for the CVE at the same time, when we make the CVE public.

[welcome]

Congratulations on your first PR being merged! 🎉