Discussion: TLS branch selection & lifecycle

[rvagg]

The intent of this issue is to gather some initial discussion on how to run a TLS process. Ultimately this is up to the working group to decide but feedback is important here.

I don't think this needs to be pinned down firmly any time soon, we can start simple and ease in to it, ramping up the serious as we go along and work it out. That would be my preference anyway. For instance, I'd like to propose that our first LTS is very soon, perhaps to coincide with the next V8 release, and we announce support for only 3 months with the option of extending that later on. Then we can build momentum around the process and iron out any wrinkles. Beyond that, we can brainstorm some possibilities of how this is run.

These are just some options that come to mind for me to start the discussion.

Option 1: Major-version-based LTS releases

In this option we would run an LTS branch as soon as the first minor is bumped after a major, and we'd stick with only patch releases in the strict semver sense. How long they are supported is a matter for further discussion.

One positive about this is that it's easy to understand and communicate.

There are multiple negatives though. It assumes we'll have major-version bumps regularly, which may not be the case given the pain they're going to cause. We also can't do anything time-based with these branches other than how long they exist for.

Example

1. Standard io.js release train moves from 2.0.4 to 2.1.0, an LTS is triggered.
2. A 2.0.x branch is created to run the LTS.
3. 2.0.5 is cut as soon as there is something to warrant it, all patch releases from 2.0.5 onward are part of the LTS.
4. Standard releases continue as usual.

Option 2: Regular time-based, non-overlapping LTS releases

In this option we would have a time period that would trigger a release, perhaps 6 or 12 months. This would be a little like the way Ubuntu does standard releases (not LTS). At the appropriate date, we would take the current major.minor and use that to start an LTS branch. There would be a period of time where there's overlap between standard and LTS so we probably wouldn't end up making actual LTS releases until the next minor bump for standard releases.

Example

We're doing a 12-month cycle starting 1-May each year.

1. Standard io.js release train is at 1.9.2 as we roll over in to May.
2. As soon as there is a major-version bump to 1.10.0 we create a 1.9.x branch and LTS starts where standard releases left off in that minor.
3. Standard releases continue as normal.

Option 3: Regular time-based, overlapping LTS releases

In this option we do the same as the previous but we are maintaining multiple LTS branches at the same time. The overlap allows people a smoother upgrade path than the strict stop/start of option 2. The amount of time we run an LTS differs from the amount of time between creating an LTS branch. Perhaps 18 months for the lifecycle and 12 months between each LTS branch creation, giving a 6 month window for upgrading between LTS.

Variation 1: Time period is the same as the amount of time between creating an LTS branch but we run 2 (or more?) LTS releases, staggered by 6 months, so we're always creating a new LTS each 6 months and run them for X months each, where X could be 12 months or more.

Variation 2: We have two different classes of LTS and we create something similar to the Ubuntu process. We have a long-running one one that runs for a shorter time. So for example, we could make a new LTS every 6 months like Ubuntu does and only support them for 6 months but every X LTS branch is supported for an extended period, perhaps every 3rd gets supported for 24 months or more.

Discussion

So, given those ideas, let's discuss LTS models that we know work well in the wild and how we can adapt those ideas to what we have to work with in io.js.

The major concern I have with io.js is that supporting V8 for longer than a stable version of Chrome is going to be on our heads. However, it's somehow managed to work for Node.js 0.10 for >2 years now so perhaps it's not going to be all that hard. I think the main risks are security-related, we'd be the only eyes supporting that codebase so we'd better have people that understand it.

There is also the issue of possible extension of LTS. There was a suggestion, floated early in the life of io.js (or was it node-forward?) that "if someone wants to contribute then the can keep a branch alive", or something to that effect. I'm uncomfortable with the looseness of this approach because it becomes difficult to communicate exactly what kind of support a particular release train has from the io.js team. I would rather leave it to the LTS WG to make decisions about precisely what the lifecycle of an LTS branch is and if, for instance, a company was to step forward and say that they are willing to throw engineers at a branch to make it live for 5 years and the LTS WG became convinced this was possible and acceptable then they could make it happen. The alternative is to let people release their own versions outside of the official io.js banner, which may work great for situation like IBM wanting to continue releasing POWER-specific versions beyond a standard LTS cycle. This is all open for discussion of course.

[Fishrock123]

Any reason why we couldn't do LTS on majors as x.minor.patch? (i.e. LTS on 1.x would just be 1.x (or 1.latest.x))

see nodejs/dev-policy#3 (comment)

Edit: see also for more detail: https://github.com/jasnell/dev-policy/pull/11/files#diff-04c6e90faac2675aa89e2176d2eec7d8R160

[mikeal]

Ya, I don't see the point in doing LTS for minor lines if we're following semver.

[Zayelion]

Discussion points I'd like to add

1.) How long is the LTS for? Security Patches for how long, in months. After a point it becomes a burden especially if the LTS cuts are high in number. (Option 1 wont work with a long lifespan.)
2.) A 4 releases a year model works fairly well and coincides with corporate life cycle and budgeting.
3.) In my developer experience company's have this business related but not particularly intelligent habit of keeping dead code alive by throwing money and resources at it till it breaks in a critical way. If we support this then we run into the same issue Joynet had. If a company gets to love a specific flavor of an API or feature/bug it will pump life into that branch. I do not see a major issue with that. I think there needs to be a protocol in that case to bring the company back upto mainline and explore what it found and enjoyed where it can not stay on mainline.

It is very important that the companies stay on mainline else it becomes in their best interest to slow the project in specific regards to keep stability in its own products. This can be handled with very strict no support timelines. This may sound strange but this can be done by just filing bugs, someone has to stop and fix the gears specifically someone experienced. They could be spending time stabilizing or improving mainline. Not fixing something that is likely already fixed. Nor should there exist a mechanism for them to do so via financial incentive.

The code is open source for a reason. I feel they should fork it. If they decide to work publicly a notification of intent should be sent and the repository monitored. After the branch expires they should be on their own. With 4 releases a year I believe 6 months of support should be enough. With releases softly placed around quarter start/ends.

Once a year is far to long, patches will be released they should update fully not partially as long as changes are not breaking. We are beyond v1 ( soon v2 ) so this should not be an issue.

I could not find any offhand statements from Google with a quick search for how long it supports a Chrome version but I believe we need to stick close to that.

Or do you believe the classical 5yr model would be better?

[mikeal]

Comparisons to Chrome don't really work. Chrome has an auto-updater which is transparent to customers. Updating io.js is much more involved and potentially requires a recompile of addons or writing fixes or taking new versions of dependencies to deal with incompatibilities.

I don't think it's a good idea to set an arbitrary timeline like 5 years. We should use download and npm statistics to see which versions are still actively used, as well as surveying a few companies that host a lot of io.js apps. Let the usage drive the support length, along with the availability of contributors who care enough about a particular release line to continue to write patches.

[trevnorris]

@mikeal I think we should first address the more practical issue of feasibility. I assume V8 upgrades that break API/ABI compatibility won't fly, so we should look at the history of V8 development and discuss how far back is considered infeasible to maintain/back port patches/etc.

[rvagg]

I believe that major-version-based LTS is fundamentally flawed and will be difficult to maintain and lead to a very confused release cycle. I'm going to be strongly advocating for time-based LTS branching based on whatever the current semver-minor is, regardless of what the major is.

I'm going to write more about why I disagree with using semver-major as the thing we used to make LTS decisions around, this but here's some of my reasoning (some repeated from the original comments above):

• It assumes we'll have major-version bumps regularly, which may not be the case given the pain they're going to cause
• We can't do anything time-based with LTS releases, including being able to predict how long we might want to be supporting an LTS branch for e.g. announce an LTS based on 2.0, a 3.0 may only be a few months away or it may be a few years, what do we say about the support timeframe of a 2.0 LTS branch? This is important for many people, mostly enterprise users. All of the LTS schedules in the wild that I know of that work really well are strictly time-based, usually both the time that they start and the length of time they last for. This provides predictability for users who can build adoption and migration plans around it.
• We consistently break stuff with semver-minor bumps, this is one of the flaws of semver and goes back to the human factor messing with the perfection of semver. In io.js and in almost any other piece of software, a feature addition generally means messing with internals of the software and this opens up the potential for breaking changes, particularly if you have lots of users. Just consider the broken require('.') with NODE_PATH issue as the latest in a series of these kinds of breakages and we've had a string of these. Node is so widely used that we can't have test cases that cover all possible ways it's being used so unless we're treading very likely we're going to break stuff. We are also getting a strong message from users that this kind of instability is making them hold off on adoption so we need to be super-conservative about LTS. From a general-user perspective, semver-minor can be just as unstable as semver-major so we should not treat semver-major as a special case as far as LTS is concerned.

(and more!)

I'll be arguing for strict time-based overlapping LTS branches. We have a schedule where we create a branch X times per year, and those branches will be supported for Y months (possibly two variants with Y and Z months so we can have longer-LTS and shorter-LTS according to a schedule). This gives users the ability to plan and build expectations around our releases and gives enterprise users the ability to map releases to their multi-year plans. It will also build discipline in us so we have a clear POV on what needs to happen and when it needs to happen and that can be communicated very simply to users.

I've also seen discussion somewhere else (on the dev-policy repo?) about possibly needing to bump semver-minor within an LTS branch and I strongly disagree with this. We need to have the discipline to be only patching LTS releases. Anything that gets in should be justifiable as a patch rather than a feature addition. If we decide that an LTS branch contains code that is in some way fatally flawed and we can't fix it with a simple patch (I can't even imagine a scenario where this would happen to be honest!) then it's not our responsibility to fix what users have decided to pin to but perhaps our best path is to communicate the need to upgrade to a newer, non-flawed LTS branch.

LTS users want to choose a version and simply have it "supported" and that just means having serious bugs fixed and security flaws taken care of with no additional shenanigans creating a shifting sand. We're already dealing with an ecosystem that doesn't believe that the patch version number really means a patch thanks to the lose application of semver in joyent/node. I've had to do walk-throughs with large enterprise clients of series of 0.10.x versions and what has changed in them because they have very little faith that a simple upgrade within 0.10 isn't going to break their deployment. We absolutely must establish trust in the stability and security of what we release as LTS.

[rvagg]

Further to the discussion of "how long":

• The users who will want very long support cycles (>2 years particularly) are enterprise users
• There exists an ecosystem of companies that who's mission is to serve those users (NodeSource, StrongLoop, etc.)

Therefore, I would expect that there is an incentive for corporates to pick up the slack for very-long-LTS and it's reasonable for us to set shorter timeframes here for general users. Unless there is an agreement by those collectively putting in the work here (again, NodeSource and StrongLoop in all likelihood) to do that support work here in a collaborative way.

However, this is still overthinking it, we need to just get started so we can practice and develop our methodologies and discover what works best and evolve into something more mature over time. That's why I've suggested starting with a mini-LTS and growing from there. We can kick that off very quickly, as soon as we form the working group in fact!

[trevnorris]

+1 to almost everything @rvagg said.

Though, support for a minor version > 2 years... That's not a long time. That's a freaking eternity. Can we start off smaller?

[rvagg]

@trevnorris 2 years is Node 0.10! but yes, it's a bloody long time.

I'm proposing that we start small and do our first LTS ASAP and time it just for 3 months, purely as an exercise in getting started, experimenting and learning about this process. I'd be keen to start this tomorrow even but it's probably best to wait till we have a WG going!

[bnoordhuis]

FWIW, I wholeheartedly agree that time-based LTS is the way to go but we'll need to bikeshed about frequency and duration.

I'm leaning towards fewer and shorter[0] while StrongLoop and NodeSource are directly footing the bill. It's different when businesses are willing to enter into support contracts.

[0] Straw man: say twice a year and for a period of nine months?

[jasnell]

Finally getting a chance to review this and I believe I'm +1 on the direction @rvagg is suggesting. Going with a generally time based LTS cycle would likely be the best overall, with at least two LTS targets per year. I would certainly hesitate putting strict target dates on it and we would need to factor in a release candidate mechanism but otherwise this looks good. Another definite requirement here would be some form of LTS roadmap that's laid out in advance -- not to restrict the things that can be landed in master but to help guide the effort and communicate expectations so that those of us who rely on the LTS releases can plan ahead accordingly.

One other important aspect to consider is how we track semver-major and semver-minor changes that end up being required to go into an LTS maintenance branch due to a critical fix (typically to address security vulnerabilities). I've been staring at this problem for a few days and so far the only reliable solution I've come up with is to use post-release extension metadata. For instance, if we're on v2.2.10 and we're rolling over to v2.3.0, the LTS is cut on v2.2.10 and the maintenance branch becomes v2.2.10 (not v2.3.x as suggested). Then, rather than continuing to bump up the patch level when changes need to go into the maintenance branch, we append a secondary post-LTS version (e.g. v2.2.10~0.0.0, v2.2.10~0.0.1, v2.2.10~0.1.0, v2.2.10~1.0.0) whenever a change goes in. Yes, it's annoying as hell but it allows us to reliably track maintenance branch changes without stepping on the versioning in the master.

    v.2.2.10
      ---------------------------------------> LTS maintenance
     ⇗ ⇣                         ⇣
    ⇗  tag:v2.2.10~0.0.0   tag:v2.2.10~1.0.0
   ⇗ 
 ----------------------------------------------> master
    ⇣           ⇣           ⇣              ⇣
tag:v2.3.0  tag:v2.3.1  tag:v2.4.0 ... tag:v3.0.0

[Zayelion]

Maybe name the LTS? v.2.2.10 cut for LTS => node-unicorn 1.0.0, node-unicorn 1.1.0.

[mhdawson]

+1 for the time based approach. I also concur that we need a distinct namespace for changes made in the LTS lines. What's suggested by James makes sense if we want to maintain full semver semantics. Alternatives might be a completed different namespace for the LTS releases (ie not include the version number we branched from) or adding a one more digit to the version we branched from. I'll also echo that in my past experience customer's want LTS timelines to be years not months.

[rvagg]

So as a strawman, I propose:

• Cut an LTS every 6 months, on a semi-predictable cycle/date
• Officially maintain each LTS branch for 9 months
• Allow corporates who have an interest in shipping for longer than 9 months to take over the burden of responsibility outside of the official capacity of the LTS group, which would most likely mean maintaining separate release channels distinct from this org (or the foundation) but it doesn't necessarily preclude cooperating to maintain git branches within core and testing that code on the shared infrastructure, just not distributing official releases of that code that place the burden of responsibility back here rather than the downstream party.

There will be parties that want longer cycles than 9 months, but the various companies involved here are in a much better position to serve them directly rather than placing that burden back on this org.

And in any case, this is still too forward-reaching for my tastes and I'd rather not see these kinds of rules and requirements locked down until we start getting experience with the process. My preference is to make some rough predictions of what we think the ideal end-point is but to actually start work on something more manageable and achievable in the short term. One of the key downfalls of the joyent/node process that has lead us to this point is the deadlock that comes from anticipating a perfect future scenario and trying to wedge the current realities in to that. i.e. action speaks much louder than words, shipping code wins, Just Do It^™, etc.

Regarding versioning, I'm still unconvinced about the need to bump minor during an LTS cycle. Can someone illuminate some examples of why this might be the case and why you can't justify forcing those changes into a patch release? LTS assumes stability, if we find ourselves in a state where the underlying code has a fatal flaw perhaps it's better to abandon an LTS release rather than start adding functionality to it?

[jasnell]

One example is adding new API and breaking default behavior to address a
security issue (such as the recent RC4 vulnerability). Since the change
breaks backwards compatibility, forcing it into a patch level change can be
dangerous.
On Apr 15, 2015 6:35 PM, "Rod Vagg" notifications@github.com wrote:

So as a strawman, I propose:

• Cut an LTS every 6 months, on a semi-predictable cycle/date
• Officially maintain each LTS branch for 9 months
• Allow corporates who have an interest in shipping for longer than 9
  months to take over the burden of responsibility outside of the official
  capacity of the LTS group, which would most likely mean maintaining
  separate release channels distinct from this org (or the foundation) but it
  doesn't necessarily preclude cooperating to maintain git branches within
  core and testing that code on the shared infrastructure, just not
  distributing official releases of that code that place the burden of
  responsibility back here rather than the downstream party.

There will be parties that want longer cycles than 9 months, but the
various companies involved here are in a much better position to serve them
directly rather than placing that burden back on this org.

And in any case, this is still too forward-reaching for my tastes and I'd
rather not see these kinds of rules and requirements locked down until we
start getting experience with the process. My preference is to make some
rough predictions of what we think the ideal end-point is but to actually
start work on something more manageable and achievable in the short term.
One of the key downfalls of the joyent/node process that has lead us to
this point is the deadlock that comes from anticipating a perfect future
scenario and trying to wedge the current realities in to that. i.e. action
speaks much louder than words, shipping code wins, Just Do It™, etc.

Regarding versioning, I'm still unconvinced about the need to bump minor
during an LTS cycle. Can someone illuminate some examples of why this might
be the case and why you can't justify forcing those changes into a patch
release? LTS assumes stability, if we find ourselves in a state where the
underlying code has a fatal flaw perhaps it's better to abandon an LTS
release rather than start adding functionality to it?

—
Reply to this email directly or view it on GitHub
#3 (comment).

[mhdawson]

As a data point 9 months is shorter than we'd like, something more like 2 years was what we had in mind. If as you mention the LTS work after 9 months is picked by a company is there any reason the changes could not be contributed to the repos within the foundation and builds generated in the build environment ?

[bnoordhuis]

@mdawsonibm Only if the corporate maintainer(s) are collaborators. Otherwise a collaborator would have to review their changes or changes would have to be accepted unvetted. The first option is arguably too much work, the second option irresponsible.

[mhdawson]

I agree you'd need the corporate maintainer(s) to be collaborators.

[rvagg]

My concern is more about the org/foundation taking responsibility for what it publishes -- if enough corporate contributors are able to make a commitment via the foundation for support then I see no problem with longer release cycles. But again, I don't think we should be making big commitments just yet, we need to get started first. The biggest concern for me in just agreeing that we should sign up for longer release cycles is the fact that the maintenance burden of V8 will be entirely on us after the Chromium team move on to newer releases and that's no small task.