Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Release cadence due to COVID-19 Pandemic #553

Closed
MylesBorins opened this issue Mar 19, 2020 · 26 comments
Closed

Release cadence due to COVID-19 Pandemic #553

MylesBorins opened this issue Mar 19, 2020 · 26 comments

Comments

@MylesBorins
Copy link
Member

@MylesBorins MylesBorins commented Mar 19, 2020

Hey All,

Was curious if we should consider adopting a different release policy for the foreseeable future in light of coronavirus and COVID-19. For example chrome is currently delaying upcoming releases due to the current situation.

One thing to consider is that we want to limit the amount of work that downstream consumers need to do to keep Node.js up to date and their services running

Some questions we need to ask:

  • Should we delay 14.x?
  • Should we put a moratorium on new features?
  • Should we temporarily make all LTS "maintenance only"

Stability in our release lines will likely help folks downstream not have additional fire drills during a trying time, alternatively creating too much of a backlog could create other hardships down the road. So to me it seems like we could do 3 different things (more options welcome of course).

  1. Change nothing, keep currently release schedule and candence
  2. Continue to release Current (e.g. 14) at planned schedule but limit LTS releases to maintenance only
  3. Feature moratorium across all release lines, security fixes + major regressions only.

Thoughts?

@codebytere
Copy link
Member

@codebytere codebytere commented Mar 19, 2020

I think it might be prudent to consider that our productivity may generally be lower as we collectively deal with all the other externalities created by this pandemic, and so we may want to optimize for what we consider to be best for the ecosystem. Chromium and Electron are both pressing pause, and I would imagine that V8 may be as well.

I think I personally would be in favor of reducing feature-motivated changes to existing codepaths as much as possible in agreement with this:

Stability in our release lines will likely help folks downstream not have additional fire drills during a trying time, alternatively creating too much of a backlog could create other hardships down the road

as I think consumers will have a harder time maintaining recency and less bandwidth to address issues. I think i personally would be in favor of 3 but also happy to pursue 2, though there are most likely things i may not be considering or aware of and would want to know y'alls thoughts as well!

@targos
Copy link
Member

@targos targos commented Mar 20, 2020

I think 2 is reasonable if we apply it after all the releases that happen next week.

@addaleax addaleax changed the title Release cadence in lieu of COVID-19 Pandemic Release cadence due to COVID-19 Pandemic Mar 20, 2020
@BethGriggs
Copy link
Member

@BethGriggs BethGriggs commented Mar 20, 2020

I think 2 is reasonable if we apply it after all the releases that happen next week.

+1 as a starting point.

We could also consider at increasing the time between current releases (every 3-4 weeks, rather than 2), if we see that as appropriate.

@BridgeAR
Copy link
Member

@BridgeAR BridgeAR commented Mar 20, 2020

I think 2 is reasonable if we apply it after all the releases that happen next week.

+1 from me as well.

Downstream mostly uses the LTS releases (as can be seen looking at the download numbers) and therefore limiting those seems fine to me. I would keep on releasing current as frequent as possible though to get people to test new things as soon as possible. Earlier and therefore more testing will likely prevent regressions to "leak" into the TLS releases.

@indutny
Copy link
Member

@indutny indutny commented Mar 20, 2020

+1 from me for 2. We should minimize the chances of breaking others code.

@sam-github
Copy link
Contributor

@sam-github sam-github commented Mar 20, 2020

I'm not sure who we are trying to help with this. Certainly not people waiting for features to be released so they can use them.

When we release new features, it doesn't cause any extra work, does it? People can adopt the new releases, or not, can't they, as they have time and inclination?

If as a project we get less contributions because people have other things on their minds, that's OK, Node.js can continue to release when we want (and when volunteers step up to do the release), and the releases will continue to contain what's ready. If that's less things ready, that's OK.

@davidmarkclements
Copy link
Member

@davidmarkclements davidmarkclements commented Mar 20, 2020

@sam-github breaking changes can cause issues, moving versions into maintenance or end of life tends to trigger the need to update and beyond that new features can absolutely cause more work. Think about platforms that provide sandboxing services etc.

@MylesBorins
Copy link
Member Author

@MylesBorins MylesBorins commented Mar 20, 2020

@sam-github in just the last month I've had to do multiple "emergency" follow up 13.x releases as changes to streams broke people down stream. The last 12.x release also required a follow up as it broke a bunch of things (we had to do a post-mortem).

This caused extra work for our team, although that is expected, but also lots of work for folks downstream who needed to upgrade, test, and report breakages.

If a security release drops after a couple of these releases that breaks things it does become much harder for folks to adopt them. My concern is that a faster release cadence on our end could end up with folks downstream sticking to older versions to avoid extra labor while they are focused on other things.

@bnb
Copy link
Member

@bnb bnb commented Mar 20, 2020

As someone who works to help enable product teams to more aggressively ship Node.js releases as parts of their various platforms, I would like to share a +1 to slowing down however the release team feels is most appropriate for y'all.

When we release new features, it doesn't cause any extra work, does it? People can adopt the new releases, or not, can't they, as they have time and inclination?

This is incorrect. People who are implementing Node.js as a part of a platform often end up with a non-trivial amount of work when features are shipped that is not simply opt-in - regardless of whether they're large features or not.

Whether that platform is a cloud or developer tooling, every time Node.js ships a feature there is churn for engineers who have to prepare both for potential unexpected stability changes as @MylesBorins mentioned and for their consumers to use any kind of new features that we ship.

@sam-github
Copy link
Contributor

@sam-github sam-github commented Mar 20, 2020

Is the code flow into projects like Chromium and Electron dominated by single companies? If so, those companies can choose to throttle the rate of contribution, as well as release rate, so they aren't creating bottlenecks.

Node.js contribution is quite distributed. Its hard to get a read on rate of incoming changes, but I haven't noticed a slow down yet. As with everything related to COVID19, the situation is unprecedented and unpredictable.

Releasing less often will likely mean that when releases do come out, they will be much larger. That itself has some risks, but if we are hearing downstream consumers say larger releases, less often, are more easily consumable, I've no objection.

@mhdawson
Copy link
Member

@mhdawson mhdawson commented Mar 20, 2020

I'd add another option for consideration:

Continue to release Current (e.g. 14) at planned schedule but limit LTS releases to the model for maintenance + specifically requested/important changes only.

The main different being that if there are features that have and advocate and for which there is a reason to include them we still can versus maintenance only which would make that more difficult even in cases where it makes sense.

It still has the problem of future larger releases as @sam-github mentioned or the possibility that 12.x will never get back out of maintenance making the move to 14.x potentially a bigger jump/challenge from 14.x but is a bit more flexible than full maintenance mode.

@joyeecheung
Copy link
Member

@joyeecheung joyeecheung commented Mar 20, 2020

+1 to option 2 from me. On features, I think if the conclusion is the productivity is going to be seriously affected in, say, the next 2 weeks, we can just hold off landing any big features during that time frame (i.e. the regular feature freeze used by Chromium, which is happening as an exception due to the COVID19 situation), and revisit after that. If we indeed decide to release as few features as possible for a short period of time, simply stopping the release of them but still keeping them landing in master would be unhelpful because then we could end up with backport headaches. We don't need to settle down on one specific plan over the course since this is probably going to take long.

@bcoe
Copy link
Member

@bcoe bcoe commented Mar 20, 2020

I think it's important to be cognizant of the fact that engineering teams are going to be significantly slowed down right now, and that new release lines can create a significant amount of engineering work: lambda, Azure Functions, and Google Cloud Functions will want to add v14 support, CI/CD systems will need to add it to their matrix.

I think it might at least be worth preparing ourselves for the v14 schedule to slip, and we should evaluate based on how things are looking over the next few weeks -- being mindful that it's not just the work for the Node.js project, but upstream folks as well that we impact (@bnb echos this point of view).

I think I personally would be in favor of reducing feature-motivated changes to existing codepaths as much as possible in agreement with this:

I agree with @joyeecheung, not that folks shouldn't open up PRs (some folks are finding coding a good escape right now), but I think a moratorium on releasing features to master right now would be good; concentrating on critical bug fixes/security patches.

@MylesBorins
Copy link
Member Author

@MylesBorins MylesBorins commented Mar 20, 2020

So, fwiw, my personal thinking at the moment is that we could leave 14.x and the current release line at large unchanged, and just slow down LTS. As mentioned by Sam coding can be a good outlet for folks right now, and we don't want to halt all innovation or stop the project from moving forward. What we do need to ensure is that our LTS consumers are not going to be broken or have any major fire drills. If we stick with the planned releases for Tuesday the biggest thing that we could do is delay the April 12.x Semver-Minor release by at least a month. Another thing we could do is push off the start of 14.x LTS, but we have some time to figure that out

@mcollina
Copy link
Member

@mcollina mcollina commented Mar 22, 2020

I agree with @MylesBorins and @mhdawson: slow down 12.x (and possibly 13.x) and keep 14.x coming as before.
If we need 14.x to slip due to some key PRs that are missing, it's not a bit deal.

@richardlau
Copy link
Member

@richardlau richardlau commented Mar 24, 2020

V8 8.1 has been demoted to beta: https://groups.google.com/forum/#!msg/v8-dev/QCzlW_toPTY/Xk5-pggnBgAJ

What does that mean for Node.js 14 if we were to stick to the current schedule? Master currently has V8 8.1 and we've never landed V8 8.0.

@MylesBorins
Copy link
Member Author

@MylesBorins MylesBorins commented Mar 24, 2020

I've been in touch with various individuals on the V8 team and have received a thumbs up on us sticking with 8.1

@mhdawson
Copy link
Member

@mhdawson mhdawson commented Mar 25, 2020

@MylesBorins can you clarify what you mean by that. Do you mean that it will be out of beta by the time we ship 14.x or that you are saying that its ok to use a beta in 14.x based on discussion with the V8 team?

@MylesBorins
Copy link
Member Author

@MylesBorins MylesBorins commented Mar 25, 2020

@mhdawson
Copy link
Member

@mhdawson mhdawson commented Mar 25, 2020

Going to mark for TSC agenda so that we have discussion on whether our project is comfortable shipping with a beta.

@mhdawson
Copy link
Member

@mhdawson mhdawson commented Mar 26, 2020

To clarify, the decision on using a beta is in the scope of the Release team, but I think it's an important enough change from our regular process that I think it's important the TSC has an FYI which is why I've added to the agenda.

@jasnell
Copy link
Member

@jasnell jasnell commented Apr 1, 2020

Sorry for being late on the discussion, I'm definitely +1 on maintaining the regular schedule for current releases but restricting LTS updates to maintenance through the remainder of 2020.

@MylesBorins
Copy link
Member Author

@MylesBorins MylesBorins commented Apr 3, 2020

So here is a summary of the consensus we had in Thursday's Release WG call. We should likely get this out in a blog post, I've been meaning to write it and will try to do so tomorrow.


  • 10.x
    • 10.20.0 was delayed a week and comes out 2020-04-07
    • Maintenance date has been delayed until 2020-05-19
    • Potentially one more 10.x Semver-Patch if needed before Maintenance on 2020-05-05
  • 12.x
    • 12.16.2 was delayed a week and comes out 2020-04-07
    • 12.17.0 delayed until 2020-05-26
    • 12.18.0 delayed until 2020-08-25
    • Maintenance is still scheduled for 2020-10-27 but subject to change with appropriate notice
  • 13.x
    • No changes to Current Release schedule or planned end of life in June 2020
  • 14.x
    • Still schedule to be released 2020-04-21
    • Will follow Current Release Schedule
    • LTS is still scheduled for 2021-10-19 but subject to change with appropriate notice

@codebytere
Copy link
Member

@codebytere codebytere commented Apr 3, 2020

@MylesBorins i can also try to take the post if that'd be easier!

@MylesBorins
Copy link
Member Author

@MylesBorins MylesBorins commented Apr 3, 2020

@BethGriggs
Copy link
Member

@BethGriggs BethGriggs commented Apr 14, 2020

Blog summarising the changes that we made to the schedule - https://nodejs.org/en/blog/announcements/adjusted-release-schedule-covid/

We will continue to review the schedule.

@BethGriggs BethGriggs closed this Apr 14, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet