New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: faster release cycle #21296

Closed
zegl opened this Issue Aug 3, 2017 · 21 comments

Comments

Projects
None yet
9 participants
@zegl
Contributor

zegl commented Aug 3, 2017

Background

Currently two new minor versions of Go are shipped per year, as according to the Go Release Cycle.

The Go blog post Contributors Summit mentions under "The standard library" that the standard library suffers from the 6 month release schedule, because new fixes and features might have to wait up to 9 months (from freeze to next release). The proposal in the blog post is to move packages out from the stdlib, to allow them to exist in their own silos, and release new versions of a package when read.

Proposal

An alternative solution to this problem would be to make releases more often.

My proposal is to speed up the release schedule to one release at least every 3 months (but hopefully faster then that). The main reason to do this would be to decrease the time to market for new fixes and features.

This would of course lead to smaller and less "impactful" upgrades, but could also allow to run experiments (such as GO15VENDOREXPERIMENT) faster.

An alternative solution could be to release standard library updates 4 or more times per year, and larger compiler changes at the current rate of 2 times per year.

Examples of languages that releases often

@gopherbot gopherbot added this to the Proposal milestone Aug 3, 2017

@gopherbot gopherbot added the Proposal label Aug 3, 2017

@EddieRingle

This comment has been minimized.

Show comment
Hide comment
@EddieRingle

EddieRingle Aug 3, 2017

This is something I'd like to see as well. I've mentioned it before, including today, where @zegl let me know that he had taken the initiative to bring up the issue here. :)

I haven't heard a convincing argument against faster release cycles yet, so I'm open to hearing others' thoughts.

EddieRingle commented Aug 3, 2017

This is something I'd like to see as well. I've mentioned it before, including today, where @zegl let me know that he had taken the initiative to bring up the issue here. :)

I haven't heard a convincing argument against faster release cycles yet, so I'm open to hearing others' thoughts.

@ianlancetaylor

This comment has been minimized.

Show comment
Hide comment
@ianlancetaylor

ianlancetaylor Aug 3, 2017

Contributor

I think the main argument against doing more frequent releases, at least in the way we do them now is, basically, it's a lot of work, and we have a limited number of people.

A secondary argument is that it would make it harder to get enough testing for major changes like garbage collection improvements, where problems often only show up over time.

I would be more sympathetic to finding a way to separate library release cycles from language/compiler/runtime release cycles. Then we could discuss a faster release cycle for the standard library, as you mentioned.

Contributor

ianlancetaylor commented Aug 3, 2017

I think the main argument against doing more frequent releases, at least in the way we do them now is, basically, it's a lot of work, and we have a limited number of people.

A secondary argument is that it would make it harder to get enough testing for major changes like garbage collection improvements, where problems often only show up over time.

I would be more sympathetic to finding a way to separate library release cycles from language/compiler/runtime release cycles. Then we could discuss a faster release cycle for the standard library, as you mentioned.

@EddieRingle

This comment has been minimized.

Show comment
Hide comment
@EddieRingle

EddieRingle Aug 3, 2017

I think the main argument against doing more frequent releases, at least in the way we do them now is, basically, it's a lot of work, and we have a limited number of people.

Why? :)

A secondary argument is that it would make it harder to get enough testing for major changes like garbage collection improvements, where problems often only show up over time.

This is handled in projects like Rust by having multiple release trains. The major changes you describe could be feature-gated or otherwise would incubate in a 'beta' or 'unstable' train before being promoted as stable. This would be in addition to the incubation that occurs before a change is even accepted to begin with.

EddieRingle commented Aug 3, 2017

I think the main argument against doing more frequent releases, at least in the way we do them now is, basically, it's a lot of work, and we have a limited number of people.

Why? :)

A secondary argument is that it would make it harder to get enough testing for major changes like garbage collection improvements, where problems often only show up over time.

This is handled in projects like Rust by having multiple release trains. The major changes you describe could be feature-gated or otherwise would incubate in a 'beta' or 'unstable' train before being promoted as stable. This would be in addition to the incubation that occurs before a change is even accepted to begin with.

@bcmills

This comment has been minimized.

Show comment
Hide comment
@bcmills
Member

bcmills commented Aug 3, 2017

Why?

See @dsnet's talk: https://youtu.be/OuT8YYAOOVI

@dsnet

This comment has been minimized.

Show comment
Hide comment
@dsnet

dsnet Aug 3, 2017

Member

I would be more sympathetic to finding a way to separate library release cycles from language/compiler/runtime release cycles.

I support this course of action more than increasing the release cycle of everything.

See @dsnet's talk: https://youtu.be/OuT8YYAOOVI

Relevant takeaway: (see 20:18 from the video). It takes 3 months to thoroughly test each release, triage the regression bugs, and actually fix them. Having more people testing each release might lower that time, but sometimes bugs are only discovered when running in canaries for some time and that's a process that doesn't get faster with more people.

Bottom-line: It's unlikely that we can increase the speed of releases without damaging the quality of each release.

Member

dsnet commented Aug 3, 2017

I would be more sympathetic to finding a way to separate library release cycles from language/compiler/runtime release cycles.

I support this course of action more than increasing the release cycle of everything.

See @dsnet's talk: https://youtu.be/OuT8YYAOOVI

Relevant takeaway: (see 20:18 from the video). It takes 3 months to thoroughly test each release, triage the regression bugs, and actually fix them. Having more people testing each release might lower that time, but sometimes bugs are only discovered when running in canaries for some time and that's a process that doesn't get faster with more people.

Bottom-line: It's unlikely that we can increase the speed of releases without damaging the quality of each release.

@EddieRingle

This comment has been minimized.

Show comment
Hide comment
@EddieRingle

EddieRingle Aug 3, 2017

It takes 3 months to thoroughly test each release, triage the regression bugs, and actually fix them. Having more people testing each release might lower that time, but sometimes bugs are only discovered when running in canaries for some time and that's a process that doesn't get faster with more people.

@dsnet (I haven't had the opportunity to watch your talk yet.) I've mentioned having separate release trains. Wouldn't that be a potential solution for much of what you describe?

EddieRingle commented Aug 3, 2017

It takes 3 months to thoroughly test each release, triage the regression bugs, and actually fix them. Having more people testing each release might lower that time, but sometimes bugs are only discovered when running in canaries for some time and that's a process that doesn't get faster with more people.

@dsnet (I haven't had the opportunity to watch your talk yet.) I've mentioned having separate release trains. Wouldn't that be a potential solution for much of what you describe?

@zegl

This comment has been minimized.

Show comment
Hide comment
@zegl

zegl Aug 4, 2017

Contributor

it's a lot of work, and we have a limited number of people.

@ianlancetaylor: Is there any other part of making a release today that is costly in terms of resources, except for waiting for users to run production workloads on the new version?

I would be more sympathetic to finding a way to separate library release cycles from language/compiler/runtime release cycles. Then we could discuss a faster release cycle for the standard library, as you mentioned.

Good, this is probably the easiest route forward if this is something that the Go team wants to do. I can't imagine there being lots of arguments against it.

See @dsnet's talk: https://youtu.be/OuT8YYAOOVI

@dsnet: Thanks for an interesting talk. At 20:18 you're showing details about the amount of failures, is this data publicly available?

Contributor

zegl commented Aug 4, 2017

it's a lot of work, and we have a limited number of people.

@ianlancetaylor: Is there any other part of making a release today that is costly in terms of resources, except for waiting for users to run production workloads on the new version?

I would be more sympathetic to finding a way to separate library release cycles from language/compiler/runtime release cycles. Then we could discuss a faster release cycle for the standard library, as you mentioned.

Good, this is probably the easiest route forward if this is something that the Go team wants to do. I can't imagine there being lots of arguments against it.

See @dsnet's talk: https://youtu.be/OuT8YYAOOVI

@dsnet: Thanks for an interesting talk. At 20:18 you're showing details about the amount of failures, is this data publicly available?

@ianlancetaylor

This comment has been minimized.

Show comment
Hide comment
@ianlancetaylor

ianlancetaylor Aug 4, 2017

Contributor

@zegl I would say that the most time consuming part is not running the tests, it's analyzing the test failures. Changes in the compiler and runtime can cause changes in program behavior. We have to determine the root cause of those changes in order to decide whether the test is faulty--depending on something it shouldn't have--or whether it is due to some incorrect change in Go. This is reasonably straightforward for a unit test, but all too often those bug reports show up in the form of "using the new version of Go causes response latency to double for this specific action," where the action is implemented by several different servers communicating with each other.

I don't want to exaggerate this too much. There are many other things we spend time on, like bug triage and so forth. But I think that test regression root cause analysis is probably the biggest and least predictable time sink, and it's the main reason why our releases tend to be a bit late.

Contributor

ianlancetaylor commented Aug 4, 2017

@zegl I would say that the most time consuming part is not running the tests, it's analyzing the test failures. Changes in the compiler and runtime can cause changes in program behavior. We have to determine the root cause of those changes in order to decide whether the test is faulty--depending on something it shouldn't have--or whether it is due to some incorrect change in Go. This is reasonably straightforward for a unit test, but all too often those bug reports show up in the form of "using the new version of Go causes response latency to double for this specific action," where the action is implemented by several different servers communicating with each other.

I don't want to exaggerate this too much. There are many other things we spend time on, like bug triage and so forth. But I think that test regression root cause analysis is probably the biggest and least predictable time sink, and it's the main reason why our releases tend to be a bit late.

@zegl

This comment has been minimized.

Show comment
Hide comment
@zegl

zegl Aug 6, 2017

Contributor

I started to sketch on a solution for a 2-track release cycle as discussed above (one for standard library + tooling, and another for everything else).

My initial idea was that we'd be able to create two branches at the start of each cycle (February / August 1), and release one of them at the same time as the release freeze of the other one begins (May / November 1).

I realise that this is definitely not optimal, as all of the changes would probably end up in one of the branches, and steal to much attention from the other, and possibly slow overall development.

If the fast-track release (the one with stdlib + tooling) is released at the same time as of the others release freeze, there would be no possibility to get feedback from the fast-track release into the normal-track release, as the release has been frozen.

My conclusion is that working on two releases at the same time is likely not worth the effort.

I'll leave the issue open, as someone else might have a better idea for how release performance can be improved.

Contributor

zegl commented Aug 6, 2017

I started to sketch on a solution for a 2-track release cycle as discussed above (one for standard library + tooling, and another for everything else).

My initial idea was that we'd be able to create two branches at the start of each cycle (February / August 1), and release one of them at the same time as the release freeze of the other one begins (May / November 1).

I realise that this is definitely not optimal, as all of the changes would probably end up in one of the branches, and steal to much attention from the other, and possibly slow overall development.

If the fast-track release (the one with stdlib + tooling) is released at the same time as of the others release freeze, there would be no possibility to get feedback from the fast-track release into the normal-track release, as the release has been frozen.

My conclusion is that working on two releases at the same time is likely not worth the effort.

I'll leave the issue open, as someone else might have a better idea for how release performance can be improved.

@josharian

This comment has been minimized.

Show comment
Hide comment
@josharian

josharian Aug 6, 2017

Contributor

There are at least two problems at play here:

(1) the speed with which bug fixes and new APIs are made available
(2) the contributor experience

The standard library is pretty stable at this point, and the tool chain needs long bake periods, and even standard library changes can be pretty heavy-hitting (e.g. the many subtle repercussions of the monotonic time change in 1.9), so I'm not convinced that reason (1) is compelling.

As for reason (2), there are other ways to go about it. For example, we could allow changes to be reviewed to +2 and queued for submission during the freeze; see this related issue and change comment. This should offer a better new/passing-through contributor experience. We would maintain the well-established cultural expectation that Go team members and the more active contributors will be more freeze-aware, and refrain from treating it as an open dev period.

In any case, I think sharpening the problem statement here (it is 1? 2? something else?) may help with finding the right solution.

Contributor

josharian commented Aug 6, 2017

There are at least two problems at play here:

(1) the speed with which bug fixes and new APIs are made available
(2) the contributor experience

The standard library is pretty stable at this point, and the tool chain needs long bake periods, and even standard library changes can be pretty heavy-hitting (e.g. the many subtle repercussions of the monotonic time change in 1.9), so I'm not convinced that reason (1) is compelling.

As for reason (2), there are other ways to go about it. For example, we could allow changes to be reviewed to +2 and queued for submission during the freeze; see this related issue and change comment. This should offer a better new/passing-through contributor experience. We would maintain the well-established cultural expectation that Go team members and the more active contributors will be more freeze-aware, and refrain from treating it as an open dev period.

In any case, I think sharpening the problem statement here (it is 1? 2? something else?) may help with finding the right solution.

@EddieRingle

This comment has been minimized.

Show comment
Hide comment
@EddieRingle

EddieRingle Aug 6, 2017

I was able to watch @dsnet's talk yesterday, and I have to disagree with the conclusions the Go team seems to be reaching here. I'd like to refer to Greg KH's talk about Linux Kernel Development: https://www.youtube.com/watch?v=vyenmLqJQjs

In both the case of Go and Linux, the projects have massive, high-quality test suites that are run routinely to check for regressions. Linux uses this to support its (relatively) fast release cycle, but Go does not.

In @dsnet's presentation, he shows how there's a decent influx of patches at the opening of a new Go development cycle, but a much larger one as the cycle nears the end of the 3 months period. I would argue that it is the long period between releases that encourages this behavior. If releases happened more frequently, people might not feel as rushed to submit their changes, which in turn (I believe) would reduce the number of bugs and regressions introduced.

EddieRingle commented Aug 6, 2017

I was able to watch @dsnet's talk yesterday, and I have to disagree with the conclusions the Go team seems to be reaching here. I'd like to refer to Greg KH's talk about Linux Kernel Development: https://www.youtube.com/watch?v=vyenmLqJQjs

In both the case of Go and Linux, the projects have massive, high-quality test suites that are run routinely to check for regressions. Linux uses this to support its (relatively) fast release cycle, but Go does not.

In @dsnet's presentation, he shows how there's a decent influx of patches at the opening of a new Go development cycle, but a much larger one as the cycle nears the end of the 3 months period. I would argue that it is the long period between releases that encourages this behavior. If releases happened more frequently, people might not feel as rushed to submit their changes, which in turn (I believe) would reduce the number of bugs and regressions introduced.

@EddieRingle

This comment has been minimized.

Show comment
Hide comment
@EddieRingle

EddieRingle Aug 7, 2017

I know he's on vacation (or at least he claimed he was, but I still see him on the mailing lists), but I'd love to get @bradfitz's feedback as well when he gets the chance.

EddieRingle commented Aug 7, 2017

I know he's on vacation (or at least he claimed he was, but I still see him on the mailing lists), but I'd love to get @bradfitz's feedback as well when he gets the chance.

@rsc

This comment has been minimized.

Show comment
Hide comment
@rsc

rsc Aug 7, 2017

Contributor

The Go blog post Contributors Summit mentions under "The standard library" that the standard library suffers from the 6 month release schedule, because new fixes and features might have to wait up to 9 months (from freeze to next release).

If this is the problem you want to solve, then halving the release cycle still means waiting 4.5 months for a critical fix. If 9 months is not OK, I don't see why 4.5 months is OK. That is, I don't think halving the release cycle really solves this problem.

But halving the release cycle does create various problems, chief among them that we don't have time to do that and maintain quality.

So this seems like a non-starter for me.

If the goal is to solve the "timely fixes to key libraries" problem, then I think instead we should be talking about how to make it easier to build a release toolchain with some set of patches.

Contributor

rsc commented Aug 7, 2017

The Go blog post Contributors Summit mentions under "The standard library" that the standard library suffers from the 6 month release schedule, because new fixes and features might have to wait up to 9 months (from freeze to next release).

If this is the problem you want to solve, then halving the release cycle still means waiting 4.5 months for a critical fix. If 9 months is not OK, I don't see why 4.5 months is OK. That is, I don't think halving the release cycle really solves this problem.

But halving the release cycle does create various problems, chief among them that we don't have time to do that and maintain quality.

So this seems like a non-starter for me.

If the goal is to solve the "timely fixes to key libraries" problem, then I think instead we should be talking about how to make it easier to build a release toolchain with some set of patches.

@EddieRingle

This comment has been minimized.

Show comment
Hide comment
@EddieRingle

EddieRingle Aug 7, 2017

But halving the release cycle does create various problems, chief among them that we don't have time to do that and maintain quality.

I think this is the problem I'm trying to get at: "Why are releases so painful for the Go team that they choose not to release more frequently than every 6 months?"

I don't see why Go (especially with Google's backing and infrastructure) isn't able to do this when other projects like the Linux kernel, Rust, and GitLab are. They're able to iterate and receive feedback quickly, with new releases every 4-8 weeks (disregarding even more frequent releases such as nightly builds). If the toolchain seriously needs to stop and "bake" for many months at a time to check for regressions, then perhaps there's an even deeper underlying issue (in Go itself or in the development process) that's making those regressions/bugs more likely to appear or harder to diagnose and fix.

EddieRingle commented Aug 7, 2017

But halving the release cycle does create various problems, chief among them that we don't have time to do that and maintain quality.

I think this is the problem I'm trying to get at: "Why are releases so painful for the Go team that they choose not to release more frequently than every 6 months?"

I don't see why Go (especially with Google's backing and infrastructure) isn't able to do this when other projects like the Linux kernel, Rust, and GitLab are. They're able to iterate and receive feedback quickly, with new releases every 4-8 weeks (disregarding even more frequent releases such as nightly builds). If the toolchain seriously needs to stop and "bake" for many months at a time to check for regressions, then perhaps there's an even deeper underlying issue (in Go itself or in the development process) that's making those regressions/bugs more likely to appear or harder to diagnose and fix.

@bcmills

This comment has been minimized.

Show comment
Hide comment
@bcmills

bcmills Aug 7, 2017

Member

I don't see why Go (especially with Google's backing and infrastructure) isn't able to do this when other projects like the Linux kernel, Rust, and GitLab are.

One key difference between Go and the Linux kernel and Rust, at least, is that neither of the latter include a garbage collector. GC bugs in particular tend to be difficult to detect (because tests don't often trigger many GC cycles) and diagnose (because symptoms of GC bugs often don't occur until an access much later in the program).

Member

bcmills commented Aug 7, 2017

I don't see why Go (especially with Google's backing and infrastructure) isn't able to do this when other projects like the Linux kernel, Rust, and GitLab are.

One key difference between Go and the Linux kernel and Rust, at least, is that neither of the latter include a garbage collector. GC bugs in particular tend to be difficult to detect (because tests don't often trigger many GC cycles) and diagnose (because symptoms of GC bugs often don't occur until an access much later in the program).

@ianlancetaylor

This comment has been minimized.

Show comment
Hide comment
@ianlancetaylor

ianlancetaylor Aug 7, 2017

Contributor

In addition to what @bcmills said, I think the deeper underlying issue is the integrated compiler and runtime that Go uses, plus the range of targets that we try to support. Or maybe we just suck, it's hard to tell.

It's tempting to think that Google's backing and infrastructure would let the Go team do anything, but the Go team is small and not in any sense a company-wide priority.

In any case, faster releases are not good in and of themselves. After all, more releases create more of a burden to people trying to keep up, and/or they create more of a support burden to the team supporting more older releases (e.g., see https://golang.org/security and consider how we would have to change it if we did faster releases). So: what problem are we trying to solve?

Contributor

ianlancetaylor commented Aug 7, 2017

In addition to what @bcmills said, I think the deeper underlying issue is the integrated compiler and runtime that Go uses, plus the range of targets that we try to support. Or maybe we just suck, it's hard to tell.

It's tempting to think that Google's backing and infrastructure would let the Go team do anything, but the Go team is small and not in any sense a company-wide priority.

In any case, faster releases are not good in and of themselves. After all, more releases create more of a burden to people trying to keep up, and/or they create more of a support burden to the team supporting more older releases (e.g., see https://golang.org/security and consider how we would have to change it if we did faster releases). So: what problem are we trying to solve?

@ALTree

This comment has been minimized.

Show comment
Hide comment
@ALTree

ALTree Aug 7, 2017

Member

Saying "but look at project X, they release every 6 weeks" is not terribly useful.

You mentioned the Linux kernel and gitlab, but they are products so different from a programming language that I'm not sure their release frequency is relevant, at all.

Regarding Rust, yes, they release every 6 weeks, but considering that

  1. The rust standard library is very small compared to the Go (or Python or Java) one. This is
    by design (they went for small std + good packaging support from the start and putting most libraries outside the core project);

  2. They use LLVM, while Go has its own backend (not exactly the easiest part of a toolchain to improve, maintain and debug);

  3. As noted above, no runtime or GC means hard-to-debug runtime issues are arguably not a problem

It's not that surprising that they can release so often; and the project (the language, the toolchain and the standard library) is so different from Go that any comparison between release frequencies is not really fair.

Rust is not the only other programming language out there. What about the other languages? How often they release a new major (in the 1.X sense) version? I suspect you'll find that a timeframe of several months between releases (like 6, for Go, or even more) it's quite common.

Member

ALTree commented Aug 7, 2017

Saying "but look at project X, they release every 6 weeks" is not terribly useful.

You mentioned the Linux kernel and gitlab, but they are products so different from a programming language that I'm not sure their release frequency is relevant, at all.

Regarding Rust, yes, they release every 6 weeks, but considering that

  1. The rust standard library is very small compared to the Go (or Python or Java) one. This is
    by design (they went for small std + good packaging support from the start and putting most libraries outside the core project);

  2. They use LLVM, while Go has its own backend (not exactly the easiest part of a toolchain to improve, maintain and debug);

  3. As noted above, no runtime or GC means hard-to-debug runtime issues are arguably not a problem

It's not that surprising that they can release so often; and the project (the language, the toolchain and the standard library) is so different from Go that any comparison between release frequencies is not really fair.

Rust is not the only other programming language out there. What about the other languages? How often they release a new major (in the 1.X sense) version? I suspect you'll find that a timeframe of several months between releases (like 6, for Go, or even more) it's quite common.

@rsc

This comment has been minimized.

Show comment
Hide comment
@rsc

rsc Aug 11, 2017

Contributor

First, please note that the blog post records the discussion, not promises to take various actions. I am personally quite skeptical of moving anything important out of the standard library.

I don't fully understand what problem you want to solve in this issue. The original post makes clear that it wants to reduce the time between a fix being made and the fix being in a release. But as I noted above, if 9 months is too long, 4½ months is probably still too long. An easy way to make local patches to a release seems like it would fix the "time to fix available" more directly. I talked to Ian, and he suggested that maybe companies are more willing to run releases than releases+patches, which does seem like a valid point. But it's still the case that 4½ months is a long time.

Regardless of the reason, the thought experiment of what it would mean to shorten the cycle is probably worth doing. We talked about this at the proposal meeting earlier this week and came up with the following considerations and implications if we wanted to shorten to (for concreteness) a 4-month cycle.

(1) The current cycle is tailored to the calendar so that no particular phase of the year is consumed by too many holidays. We'd have to look carefully at how to fit additional cycles in.

(2) We recently extended release support to two cycles (12 months) from the date of release. If we shorten the cycle I expect that for all the same reasons we'd need to extend support to keep it at 12 months, which would mean more work supporting old releases.

(3) We haven't done a great job motivating the freeze and explaining how important it is to maintaining release quality. @dsnet's talk at Gophercon is probably a good starting point, and we should turn it into a blog post.

(4) It's not clear we have the bandwidth to run more release cycles. I'd feel a lot better about shortening the current release cycles if we consistently had the release ready early. As it is, we've recently been consistently a few weeks late. If we're going to shorten the cycles, that few weeks becomes a larger fraction of the next cycle. When we introduced the current 3+3 cycles, we talked about moving to 4+2 if we found ourselves finishing the freeze work early. But I think that's only ever happened once in four years.

Overall it's far from clear to me that this is a net win for anyone: it doesn't really make emergency fixes timely, and it introduces all these other problems that we'd need to solve.

In short, if you want more frequent releases, help us hit our current release dates. If we can do that consistently, then I think it might make sense to talk about shortening the cycle.

Contributor

rsc commented Aug 11, 2017

First, please note that the blog post records the discussion, not promises to take various actions. I am personally quite skeptical of moving anything important out of the standard library.

I don't fully understand what problem you want to solve in this issue. The original post makes clear that it wants to reduce the time between a fix being made and the fix being in a release. But as I noted above, if 9 months is too long, 4½ months is probably still too long. An easy way to make local patches to a release seems like it would fix the "time to fix available" more directly. I talked to Ian, and he suggested that maybe companies are more willing to run releases than releases+patches, which does seem like a valid point. But it's still the case that 4½ months is a long time.

Regardless of the reason, the thought experiment of what it would mean to shorten the cycle is probably worth doing. We talked about this at the proposal meeting earlier this week and came up with the following considerations and implications if we wanted to shorten to (for concreteness) a 4-month cycle.

(1) The current cycle is tailored to the calendar so that no particular phase of the year is consumed by too many holidays. We'd have to look carefully at how to fit additional cycles in.

(2) We recently extended release support to two cycles (12 months) from the date of release. If we shorten the cycle I expect that for all the same reasons we'd need to extend support to keep it at 12 months, which would mean more work supporting old releases.

(3) We haven't done a great job motivating the freeze and explaining how important it is to maintaining release quality. @dsnet's talk at Gophercon is probably a good starting point, and we should turn it into a blog post.

(4) It's not clear we have the bandwidth to run more release cycles. I'd feel a lot better about shortening the current release cycles if we consistently had the release ready early. As it is, we've recently been consistently a few weeks late. If we're going to shorten the cycles, that few weeks becomes a larger fraction of the next cycle. When we introduced the current 3+3 cycles, we talked about moving to 4+2 if we found ourselves finishing the freeze work early. But I think that's only ever happened once in four years.

Overall it's far from clear to me that this is a net win for anyone: it doesn't really make emergency fixes timely, and it introduces all these other problems that we'd need to solve.

In short, if you want more frequent releases, help us hit our current release dates. If we can do that consistently, then I think it might make sense to talk about shortening the cycle.

@EddieRingle

This comment has been minimized.

Show comment
Hide comment
@EddieRingle

EddieRingle Aug 11, 2017

@rsc A few things that come to mind after reading your reply:

Regarding release quality, it was mentioned earlier that GC changes are an example of potential sources of regressions and bugs. I wonder if a three-tier branching model would be beneficial, something like:

  • master - Active development; as it is now when not under a feature freeze.
  • staging/1.10 - Changes for a given cycle merged here for integration testing. "High-risk" changes like those that touch the GC runtime could bake here a little longer, whereas "low-risk" changes could progress to the next branch more quickly. It might be decided that a certain change isn't stable by the time the scheduled release date rolls around, so it could be quickly reverted here (or simply not merged into the next branch).
  • release/1.10 - Validated changes end up here in tagged releases as part of the 1.10 train.

(4) It's not clear we have the bandwidth to run more release cycles. I'd feel a lot better about shortening the current release cycles if we consistently had the release ready early. As it is, we've recently been consistently a few weeks late. If we're going to shorten the cycles, that few weeks becomes a larger fraction of the next cycle. When we introduced the current 3+3 cycles, we talked about moving to 4+2 if we found ourselves finishing the freeze work early. But I think that's only ever happened once in four years.

What about cutting the merge window down as much as possible? For example, 2 week merge window, three months of testing. Any changes that don't get in during those 2 weeks simply roll to the next window.

EddieRingle commented Aug 11, 2017

@rsc A few things that come to mind after reading your reply:

Regarding release quality, it was mentioned earlier that GC changes are an example of potential sources of regressions and bugs. I wonder if a three-tier branching model would be beneficial, something like:

  • master - Active development; as it is now when not under a feature freeze.
  • staging/1.10 - Changes for a given cycle merged here for integration testing. "High-risk" changes like those that touch the GC runtime could bake here a little longer, whereas "low-risk" changes could progress to the next branch more quickly. It might be decided that a certain change isn't stable by the time the scheduled release date rolls around, so it could be quickly reverted here (or simply not merged into the next branch).
  • release/1.10 - Validated changes end up here in tagged releases as part of the 1.10 train.

(4) It's not clear we have the bandwidth to run more release cycles. I'd feel a lot better about shortening the current release cycles if we consistently had the release ready early. As it is, we've recently been consistently a few weeks late. If we're going to shorten the cycles, that few weeks becomes a larger fraction of the next cycle. When we introduced the current 3+3 cycles, we talked about moving to 4+2 if we found ourselves finishing the freeze work early. But I think that's only ever happened once in four years.

What about cutting the merge window down as much as possible? For example, 2 week merge window, three months of testing. Any changes that don't get in during those 2 weeks simply roll to the next window.

@josharian

This comment has been minimized.

Show comment
Hide comment
@josharian

josharian Aug 11, 2017

Contributor

@EddieRingle

  1. It's frequently not obvious what changes are high risk. A last minute 1.9 bug fix that was hard to find but easy to fix came from what I would have considered a low- to moderate risk change.

  2. What is this "merge window" of which you speak? It feels like maybe you aren't very familiar with the details of how things currently work and why, which is a prerequisite for proposing changes.

  3. Avoiding lots of branches is a conscious decision.

  4. It's easy to concoct proposals. But a better place to start would be to do what several of us have asked for: Precisely state what the problem you want to solve is.

Contributor

josharian commented Aug 11, 2017

@EddieRingle

  1. It's frequently not obvious what changes are high risk. A last minute 1.9 bug fix that was hard to find but easy to fix came from what I would have considered a low- to moderate risk change.

  2. What is this "merge window" of which you speak? It feels like maybe you aren't very familiar with the details of how things currently work and why, which is a prerequisite for proposing changes.

  3. Avoiding lots of branches is a conscious decision.

  4. It's easy to concoct proposals. But a better place to start would be to do what several of us have asked for: Precisely state what the problem you want to solve is.

@rsc

This comment has been minimized.

Show comment
Hide comment
@rsc

rsc Oct 9, 2017

Contributor

Closing for now, at least until we're hitting our current release targets.

Contributor

rsc commented Oct 9, 2017

Closing for now, at least until we're hitting our current release targets.

@rsc rsc closed this Oct 9, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment