Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: clarify backporting policy #34536

Open
FiloSottile opened this issue Sep 25, 2019 · 41 comments

Comments

@FiloSottile
Copy link
Member

@FiloSottile FiloSottile commented Sep 25, 2019

The policy for what does and doesn't get backported is currently documented at https://golang.org/wiki/MinorReleases.

Our default decision should always be to not backport, but fixes for security issues, serious problems with no workaround, and documentation fixes are backported to the most recent two release branches, if applicable to that branch.

Fixes for experimental ports are generally not backported.

A “serious” problem is one that prevents a program from working at all. "Use a more recent stable version" is a valid workaround, so very few fixes will be backported to both previous issues.

This is pretty vague, and it detaches from practice in quite a few ways. We discussed a more complete framework that reflects the current reality with @dmitshur.

I propose we document it at https://golang.org/doc/devel/release.html#policy (in keeping with #34038) with the following text.

/cc @golang/osp-team


Backporting policy

Most recent major release. The following changes are are eligible for backporting to the latest major release.

  • Security fixes, in their own release, according to the security policy.
  • Fixes for serious problems with no workaround. A “serious” problem is one that prevents a program from working at all. This includes, for example, miscompilation issues.
  • Early fixes for regressions and issues in new functionality. As the release matures, the bar for these changes gets higher: while the first minor release in a series will accept most fixes for user-visible regressions, after five months they will be mostly rejected. As more of the ecosystem upgrades, fixing regressions with workarounds becomes less and less valuable, and the tradeoff with stability shifts.
  • Documentation changes that fix incorrect public docs.

Fixes for experimental ports are generally not backported.

Previous major release. The only fixes eligible for backporting to the previous major release (for example, to Go 1.10, once Go 1.11 has been released) are those that address external changes that would make the release unusable. This includes security fixes and platform compatibility fixes (for example, if a new version of an OS breaks Go programs).

We consider upgrading to the latest major release a valid workaround, and the purpose of maintaining the previous major release is only not to force users to upgrade unexpectedly, so pre-existing serious issues and regressions are only fixed in the latest major release.

@gopherbot gopherbot added this to the Proposal milestone Sep 25, 2019
@gopherbot gopherbot added the Proposal label Sep 25, 2019
@networkimprov

This comment has been minimized.

Copy link

@networkimprov networkimprov commented Sep 26, 2019

I find this a little confusing, because fixes to the most-recent release are also called "backports". The above text seems to address only the previous release.

@FiloSottile

This comment has been minimized.

Copy link
Member Author

@FiloSottile FiloSottile commented Sep 26, 2019

Would changing "latest" into "most recent" here help?

The following changes are are eligible for backporting to the latest major release.

@networkimprov

This comment has been minimized.

Copy link

@networkimprov networkimprov commented Sep 26, 2019

Ah, I didn't see that the bullet list and paragraph pertain to different categories. It would help if the policy had two subsection heads.

Re the policy, a lot of folks would rather not upgrade deployments to a new major release until they've had the chance to evaluate its performance in their systems, and iron out any wrinkles. So early in the life of a most-recent release, they'd want to see all backports land on both releases.

Tangentially, I think the best policy would be long-term-support versions of the toolchain & runtime, and a rolling-release of the stdlib... But that's prob outside the scope here :-)

@FiloSottile

This comment has been minimized.

Copy link
Member Author

@FiloSottile FiloSottile commented Sep 26, 2019

Edited it with paragraph headings, thank you.

Re the policy, a lot of folks would rather not upgrade deployments to a new major release until they've had the chance to evaluate its performance in their systems, and iron out any wrinkles. So early in the life of a most-recent release, they'd want to see all backports land on both releases.

That's why we still backport things that are necessary to stay on the previous major release, but after six months, there are very few regressions that can still be found, and stability becomes more important (as well as limited team resources).

@networkimprov

This comment has been minimized.

Copy link

@networkimprov networkimprov commented Sep 26, 2019

It would help if the policy gave a time frame for full support of the previous release, e.g. 90 days. That would let deployment teams know the earliest date on which they might have to either roll out the most recent release, or consider custom builds.

@FiloSottile

This comment has been minimized.

Copy link
Member Author

@FiloSottile FiloSottile commented Sep 26, 2019

I'll find somewhere to link https://golang.org/s/release if that's what you mean?

@networkimprov

This comment has been minimized.

Copy link

@networkimprov networkimprov commented Sep 26, 2019

No. The previous-release policy seems vague and perhaps contradictory:

the purpose of maintaining the previous major release is only not to force users to upgrade immediately

How long is "not immediately", i.e. how long will the previous release get all backports? It's apparently less than 6 months. Could the policy give a time frame?

The only fixes eligible for backporting to the previous major release ... are those that address external changes that would make the release unusable

For fixes outside that set, one apparently has to upgrade immediately?

A lot of sites would prefer to postpone major upgrades as long as possible, and some even skip over an entire cycle.

@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Sep 26, 2019

@networkimprov, sites who intend to skip an entire cycle should be very careful to test the beta and/or RC releases of the next release they do intend to pick up, since the window for the previous release closes as soon as the new release occurs.

You really don't want to be in a situation where you're, say, stuck on 1.11 and neither 1.12 nor 1.13 works for you because of some 1.12 regression that you didn't detect until after the 1.13 release.

@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Sep 26, 2019

For fixes outside that set, one apparently has to upgrade immediately?

Yes: if something non-critical is broken and has been for a long time, then it can presumably remain non-critically broken for a little while longer while you upgrade.

@FiloSottile

This comment has been minimized.

Copy link
Member Author

@FiloSottile FiloSottile commented Sep 26, 2019

Oh, I see what's unclear. What I am trying to say is that if you upgrade your Linux kernel and Go 1.12.5 stops working with it, and we only backport the fix to Go 1.13, you are now forced to upgrade to Go 1.13 immediately. You can't stay on Go 1.12 until you are ready.

OTOH, if you are running Go 1.12.5 and it was working for you, it's fine to say you need to upgrade to Go 1.13.2 to get some fix.

The difference is external forces.

@networkimprov

This comment has been minimized.

Copy link

@networkimprov networkimprov commented Sep 27, 2019

It's awkward to be the only one asking about this; I hope I haven't misunderstood something basic.

But I'm startled to learn that regression backports cease on the day the next major release appears. I'd assumed that a previous release would be fully supported for 6 months. (Would anyone expect otherwise?)

Suppose I wait until 1.12.5 to upgrade from 1.11.x, and then discover a regression on the day 1.13 comes out, I'm outta luck? Either upgrade again to 1.13 or resort to a custom build?

What I suggested above is that regression backports cease at a later date, e.g. 90 days following the next major release, and that the policy should document the time frame.

The difference is external forces

A distinction between regressions and "external forces" seems arbitrary. If one could fully evaluate a deployed application running on a new release within a known period, that would surface any regressions before the backport window closes. But that's not achievable in many cases.

stuck on 1.11 and neither 1.12 nor 1.13 works for you because of some 1.12 regression that you didn't detect until after the 1.13 release.

@bcmills I gathered that fix would be backported to 1.13 since it broke something working in 1.11?

@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Sep 27, 2019

I gathered that fix would be backported to 1.13 since it broke something working in 1.11?

Yes; but if you're still on 1.11 due to some unreported regression in 1.12, it may take a while to identify and backport the fix to 1.13, and in the meantime you'd either be stuck with the regression or stuck with an unsupported Go release.

So it's a risky situation either way — it's much less risky to either upgrade through every release, or ensure that you test the betas and RCs of the next release after you skip one.

@mdempsky

This comment has been minimized.

Copy link
Member

@mdempsky mdempsky commented Sep 27, 2019

A lot of sites would prefer to postpone major upgrades as long as possible, and some even skip over an entire cycle.

It would be interesting to know how widespread this preference is, and what motivates it.

E.g., my first assumption was this only affects stubborn enterprise customers (who presumably could afford support contracts). But until Debian's latest stable release earlier this year, they were still shipping Go 1.7. Ubuntu 16.04LTS and 18.04LTS provide Go 1.6 and Go 1.10, respectively. Presumably other Linux distros are similarly affected.

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Sep 27, 2019

Honestly I would expect that most large sites skip releases. I would be astonished if they adopt every release. At a large site each new release requires full re-qualification and testing. Nobody has time for that, unless they anticipate some large benefit.

(That's not to say that we need to do anything different. Our resources are limited. There is clearly space for commercial support organizations.)

@networkimprov

This comment has been minimized.

Copy link

@networkimprov networkimprov commented Sep 27, 2019

Ubuntu 16.04LTS and 18.04LTS provide Go 1.6 and Go 1.10, respectively

That is a great argument for LTS releases.

@mdempsky

This comment has been minimized.

Copy link
Member

@mdempsky mdempsky commented Sep 27, 2019

Honestly I would expect that most large sites skip releases.

[cries in security engineer]

@VitaliyRT

This comment has been minimized.

Copy link

@VitaliyRT VitaliyRT commented Sep 28, 2019

✋️👍

@networkimprov

This comment has been minimized.

Copy link

@networkimprov networkimprov commented Sep 30, 2019

No one's addressed #34536 (comment)

Should I open a new issue to suggest changing the regression-backports window for previous releases?

@FiloSottile

This comment has been minimized.

Copy link
Member Author

@FiloSottile FiloSottile commented Sep 30, 2019

I think the current text reflects the reality of the current policy, and it would be good to document it properly before we discuss changing it. If you are ok with this text going into release.html and then discussing changing the policy it reflects in another proposal, I think that would be a good way forward.

@networkimprov

This comment has been minimized.

Copy link

@networkimprov networkimprov commented Sep 30, 2019

Then I think you should add a sentence to the effect of "Any regressions found in the previous release will not be fixed."

EDIT: filed #34622

@FiloSottile

This comment has been minimized.

Copy link
Member Author

@FiloSottile FiloSottile commented Sep 30, 2019

Then I think you should add a sentence to the effect of "Any regressions found in the previous release will not be fixed."

That's not the current policy, serious regressions without workaround are still fixed, but are only backported to the latest major release, because we consider it ok to make someone who is upgrading to Go 1.12 late and finds a regression simply jump forward to Go 1.13.

@networkimprov

This comment has been minimized.

Copy link

@networkimprov networkimprov commented Sep 30, 2019

Sorry I meant "Any regressions found in the previous release will not be fixed in that release."

The current language could be misinterpreted. The above makes it clear.

@FiloSottile

This comment has been minimized.

Copy link
Member Author

@FiloSottile FiloSottile commented Sep 30, 2019

Does that add anything over "The only fixes eligible for backporting to the previous major release [...] are those that address external changes that would make the release unusable."?

@networkimprov

This comment has been minimized.

Copy link

@networkimprov networkimprov commented Sep 30, 2019

Indeed it does. The language you quote is easily misinterpreted, especially if you assume (as most would) that a previous release is fully supported.

"External changes" could be read to mean CLs produced outside the core team, for example.

@FiloSottile

This comment has been minimized.

Copy link
Member Author

@FiloSottile FiloSottile commented Sep 30, 2019

Updated the last paragraph to

We consider upgrading to the latest major release a valid workaround, and the purpose of maintaining the previous major release is only not to force users to upgrade unexpectedly, so pre-existing serious issues and regressions are only fixed in the latest major release.

@networkimprov

This comment has been minimized.

Copy link

@networkimprov networkimprov commented Sep 30, 2019

Ok, thanks.

@rsc

This comment has been minimized.

Copy link
Contributor

@rsc rsc commented Nov 14, 2019

I really don't understand this thread. The current policy says:

Each major Go release is supported until there are two newer major releases. For example, Go 1.5 was supported until the Go 1.7 release, and Go 1.6 was supported until the Go 1.8 release. We fix critical problems, including critical security problems, in supported releases as needed by issuing minor revisions (for example, Go 1.6.1, Go 1.6.2, and so on).

What part of this is unclear?

@FiloSottile

This comment has been minimized.

Copy link
Member Author

@FiloSottile FiloSottile commented Nov 14, 2019

@rsc Empirically, there have been a few discussions around different interpretations of the current text, so I am suggesting a more explicit version here. For example, what qualifies as a "critical problem"?

The OSP team developed consistent guidelines to decide what to backport, and I think it would be good to try to capture them in the policy. It might not even be proposal-worthy, really. Should I just send a CL with the suggested text (in the issue description, under the <hr>)?

@rsc

This comment has been minimized.

Copy link
Contributor

@rsc rsc commented Nov 14, 2019

After reading also #34713 and #34622 I see that what's unclear is how our recent decision in #34713 follows our stated policy. I am not sure that it does. If not, this issue would be about changing, not clarifying our policy. That would be very significant and absolutely worth treating as a proposal. Please do not send a CL.

I will talk to @andybons to get more context about the state of the world and then follow up on this issue.

Edit: Fixed issue, #34713 not #34536 as I originally typed.

@FiloSottile

This comment was marked as resolved.

Copy link
Member Author

@FiloSottile FiloSottile commented Nov 14, 2019

how our recent behavior in #34536

This is #34536, typo?

@rsc

This comment was marked as resolved.

Copy link
Contributor

@rsc rsc commented Nov 14, 2019

Thanks, fixed (meant #34713).

@rsc

This comment has been minimized.

Copy link
Contributor

@rsc rsc commented Nov 14, 2019

I closed #34622 as a duplicate of this issue, because both of these issues are about deciding and then making sure we are all clear about the rules for what does and does not get backported.

@networkimprov

This comment has been minimized.

Copy link

@networkimprov networkimprov commented Nov 14, 2019

Even tho this issue was opened first, it has little discussion about choosing/changing the policy, whereas #34622 is entirely about that. If you're open to changing the policy, I'd suggest reopening that one and closing this. (EDIT: Also there are many more substantive comments on that issue.)

EDIT: the current policy doesn't mention regressions, but the de facto policy is to stop backporting them after six months, forcing users to either upgrade or locally backport regression fixes.

@rsc

This comment has been minimized.

Copy link
Contributor

@rsc rsc commented Nov 21, 2019

I spoke to @andybons yesterday to get more context about the team's actions and decisions here and in #34713. I think there has been some confusion about the release policy itself and in particular perhaps not enough awareness of a change we made two years ago.

Background: Release Policy Change in 2017

The original Go release policy (before 2017) was that we did backports to the most recent one release for critical fixes, and to the most recent two releases for security releases. This had the unfortunate effect that on the day a major release was issued, the previous release was instantly unsupported, which left users feeling, well, not very supported. Not everyone can upgrade instantly: bugs and other incompatibilities do happen in new releases, and qualification tests take time, so it can quite reasonably take users a while to be ready to upgrade their production uses. If they hit critical bugs in the interim, they feel abandoned if we don't provide fixes. It's important not to abandon users who have not yet been able to upgrade to the latest release at the instant that the next release comes out. This was a common complaint when we talked to companies using or thinking about using Go, at least those who had read the release policy carefully.

In early 2017, after a brief, uncontentious discussion on #19069, we changed the policy to support the most recent two releases equally for all critical problems, not just critical security problems.

Omitting illustrative examples, the old release policy was:

Each major Go release obsoletes and ends support for the previous one.
We fix critical problems in the current release as needed by issuing minor revisions.

As a special case, we issue minor revisions for critical security problems in both the current release and the previous release.

After the discussion on #19069, we changed the policy, in CL 37531 aka 86abfbb, to:

Each major Go release is supported until there are two newer major releases.
We fix critical problems, including critical security problems,
in supported releases as needed by issuing minor revisions.

The implication of this policy—not just an accident of the wording but the intended effect—is that the two most recent releases are supported equally. Again, this was an explicit recognition that not everyone can quickly upgrade to the latest release to work around a critical problem.

Recent Actions

  • In December 2018, the https://golang.org/wiki/MinorReleases wiki page was edited. Originally it said:

    Our default decision should always be to not backport, but fixes for security issues, serious problems with no workaround, and documentation fixes are backported to the two supported major releases. For example a fix developed during the 1.11 cycle will be backported to a 1.9.x (if applicable) and 1.10.x release. Fixes for experimental ports are generally not backported.

    A “serious” problem is one that prevents a program from working at all.

    The change added one more sentence:

    "Use a more recent stable version" is a valid workaround, so very few fixes will be backported to both previous issues.

    This sentence seems to me to be in direct conflict with the policy change we made in 2017. It is stating the pre-2017 policy, not the current policy.

  • In September 2019, this issue (the one you are reading right now) was filed to elaborate on the December 2018 change and add it to the official release policy. I believe the title is “clarify backporting policy”—clarify, not change—because it wasn't appreciated that this a deviation from the stated policy.

  • In October 2019, there was a request for a backport of the fix for #33953, deemed a critical problem. The backport to Go 1.13 was #34714 (accepted and done), and the backport to Go 1.12 was #34713 (declined). In the discussion on #34713, there were multiple comments echoing the December 2018 wiki page change. For example, “Upgrading to 1.13 will be a valid workaround after #34712 is released.”

The Options

The release policy states that the last two major releases are supported equally. That is, assuming releases are on time, a release is supported for six months during which it is the most recent release, and then six more months during which it is the “previous release.” Let's call this the 6+6 policy.

Before 2017. we had a 6+0 policy, in which a release is only supported when it is the most recent release. Our actions over the past year or so have in some cases been more consistent with the 6+0 policy than the stated 6+6 policy. This issue (the one you are reading right now) suggests that we “clarify” that the policy is 6+0, but that would be a reversion to the pre-2017 policy from the current stated 6+6 policy.

Faced with the claims and recent implementation of a 6+0 support policy, #34622 suggests that we at least adopt a 6+3 policy, or 6+N for some N > 0. I closed that issue as a duplicate of this one because both that one and this one are really about deciding what the policy is and making sure everyone agrees.

Is it 6+6, as currently stated?
Is it 6+0, as some of our recent actions would suggest?
Or should it be 6+3 or something else in the middle?

What Should We Do?

As far as I can tell, no one over the past twelve months intentionally deviated from our 6+6 release policy. No one intentionally rolled back the 2017 change. People simply assumed the policy was 6+0, as if the change had never happened, and acted accordingly. But the change did happen. Our stated policy is 6+6, and companies have adopted Go based on that stated policy, that promise of support.

We could decide to change the stated policy, officially moving to 6+0 or 6+3. That would mean providing less support in 2020 than we said we would in 2017, possibly (likely?) alienating users relying on the stated policy.

Or we could decide to keep the current stated policy, staying at 6+6. That would mean admitting we made some mistakes over the past 12 months, correcting them as much as possible, and taking steps to avoid them in the future. In particular I think it would mean fixing the MinorReleases wiki and reopening #34713 for a backport in a future Go 1.12 point release.

To be very clear, I believe everyone was acting in good faith, and it is not my purpose here to point blame at anyone by pointing at specific actions that happened. In particular, I've omitted the names of the actors because they are unimportant. We the Go team did these things and are responsible for them together. The specifics actions we took, however, are important: we need to understand what we did in order to decide what to do moving forward, including correcting anything we now believe to be a mistake.

@rsc

This comment has been minimized.

Copy link
Contributor

@rsc rsc commented Nov 21, 2019

Personally, I think we should honor and keep the commitment we made in 2017.
That is, I think we should keep 6+6 and correct the things we've done that are inconsistent with 6+6.

The reasons seemed compelling to me at the time, and with twice as many Go users today, they only seem more compelling to me now. But even more important is that we committed to doing so. Backing down to 6+0 or even 6+3 breaks the promise we made. Doing that without a very good justification and a clear explanation will erode user trust. I realize that wasn't the intention, but I believe it would be the effect.

Please thumbs up if you agree with 6+6 + fixing mistakes;
thumbs down if you think we should do 6+0 or 6+3 instead (and maybe comment to elaborate).

Thanks.

@FiloSottile

This comment has been minimized.

Copy link
Member Author

@FiloSottile FiloSottile commented Nov 26, 2019

Thank you @rsc for the detailed background, I indeed didn't know about the 2017 discussion, and I agree that the wording of the policy I'm suggesting is a meaningful change from #19069.

As additional background, I'll try to provide my perspective on why we veered away from it, and I'll try to argue that the policy in this issue does address #19069. However, while I was sharing the release management responsibilities with the rest of the OSP team during this discussion, they are now handled by a subgroup I am not part of, so my opinion carries little weight at this point.

Over the past year, as the OSP team expanded, we made an effort to document and distribute various tasks, including approving backports. That required formalizing, at least internally, what made an issue critical. We found that particularly hard: "critical" is fairly subjective, and we struggled to elaborate on the criteria. Moreover, we noticed that at the beginning of a cycle many less-critical regressions were fixed by backport, which made sense, but was not encoded in the policy. Eventually, we resorted to making these decisions in meetings to make them consistent.

What worked fairly reliably in the decision process was asking two questions:

  1. Is this going to stop many people from upgrading?

  2. Is this going to force many people to upgrade?

What's nice about (1) is that it naturally scales with time: if many months have passed, and no one noticed an issue, surely it's not blocking many people from upgrading. (2) nicely includes security issues.

Note how (1) becomes pretty meaningless when applied to the previous release: why are users upgrading to the old release when a new one is available? This policy suggests fixing issues passing (1) in the current release, and issues passing (2) in both.

I'd argue that provides a good level of long(-ish) term support to slow-to-upgrade users: we promise we'll not force you to upgrade more often than every 12 months, because such issues would pass (2) and get backported all the way. However, you don't get to upgrade to the previous release when there's a new one available: just always upgrade to the latest if you are upgrading, and if you need a long lead time to qualify it, start with the beta.

Note how #19069 would pass (2) and would have been backported to both releases under this proposed policy.

I think a policy based on those two questions (which I might or might not have done a good job of formalizing in this proposal) would provide a clear and consistent framework to the team, while still delivering 12 months of support for releases, in the same sense that LTS distributions support staying on an old release.

@rsc

This comment has been minimized.

Copy link
Contributor

@rsc rsc commented Nov 26, 2019

@FiloSottile, I can't tell whether you agree with my suggestion that we honor our commitment to the 6+6 support policy. Are you arguing that we should change the policy away from 6+6?

@FiloSottile

This comment has been minimized.

Copy link
Member Author

@FiloSottile FiloSottile commented Nov 26, 2019

I am arguing for the following policy: for 6 months, we backport fixes for issues that would stop many users from upgrading; for the next 6 months, we only backport fixes for issues that would force current users of that release to upgrade (like security issues and incompatibilities with new OS versions, but not newly observed regressions).

That's what I tried to propose with this issue. As far as I understand, this is a different kind of 6+6 than what you called 6+6, because the second 6 months have a stricter criteria.

(Unless that was implicit in your interpretation of "critical", the subjectivity of which is what I've been trying to solve.)

@rsc

This comment has been minimized.

Copy link
Contributor

@rsc rsc commented Nov 27, 2019

I don't understand your criteria. How does a newly observed regression in a 6-month old release not force current users to upgrade if they start running into it?

The answer to that aside, the fact that I don't understand your criteria means users won't either and is exactly why the current 6+6 policy is better: it's much easier to explain "if something is deemed critical to fix in any release, it gets fixed in the last two."

And again, that's what we promised users in 2017, that the last two releases are equally supported. In the absence of very compelling new information that that was a mistake, we should do what we said we would do. Your argument seems like it is splitting hairs in a situation where hair-splitting is not something we want to inflict on users.

The consensus on this issue, including from the open source team who has to do the work, seems to be overwhelmingly in favor of keeping 6+6. I suggest we reaffirm that the last two releases are equally well supported.

@FiloSottile

This comment has been minimized.

Copy link
Member Author

@FiloSottile FiloSottile commented Dec 2, 2019

The consensus on this issue, including from the open source team who has to do the work, seems to be overwhelmingly in favor of keeping 6+6. I suggest we reaffirm that the last two releases are equally well supported.

Sounds good. As I mentioned, the open source team's opinion is strictly more relevant than mine here.

Would you like me to close this issue, or leave it open to update the wiki? (I'm happy to do it myself, but I might not be the best person to word the policy now.)

@rsc

This comment has been minimized.

Copy link
Contributor

@rsc rsc commented Dec 3, 2019

Let's leave the issue open and let the open source team use it to track any doc changes they need to make.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Proposals
Incoming
8 participants
You can’t perform that action at this time.