Brainstorming on faster releases

[bgrant0607]

What if the biweekly Kubernetes releases were usable and stable?

What would be the benefits and challenges?

Some benefits are described here:

http://spf13.com/post/release-early-release-often-to-minimize-risk

Some issues / questions to consider:

• Changes to development processes needed to ensure stability and that only functional features / APIs are exposed
  • Feature branches -- discuss in Brainstorming on Feature Branches #566
  • Feature gating
• Release process changes
  • LTS releases
  • Versioning scheme
  • Release notes
  • Feature/effort-tracking process
• Documentation process and how to support documentation for multiple releases
• Redefining supported version skew, and making it work
• What testing would be necessary to ensure stability and backward compatibility
• Further automation of upgrade tests (e.g., Refactor select e2e tests and incorporate them into automated upgrade testing kubernetes#35078)
• What upgrade procedures/tools would be tested?
• Expression of which releases work for a given feature, test, Helm chart, etc.
• Impact on conformance certification approach -- Refresh our approach to conformance testing and create a distribution certification program #432
• Reducing disruption caused by cluster upgrades
• Impact on marketing approach
  ...

This issue is intended to be a home for discussion on that topic.

cc @idvoretskyi @thockin @jagosan @pwittrock

[calebamiles]

cc: @philips

[pwittrock]

I tend to think of more frequent releases as 2 distinct problems, each of which are really challenging and will require a lot of effort.

Supporting the ability to perform a release at a moments notice

Target: developers
Benefit: stable and orderly release process

• Decouple code from set of enabled features
  • Flag guard features until fully certified as ready to enable
• Push on green
  • 100% confidence in test coverage ability to verify release
  • Dashboards with clear signal - ok / not ok
• Continuous stability
  • Non-flaky tests
  • Immediately rollback PRs that break tests

Supporting publishing frequent releases

Target: End users
Benefit: Features available sooner, less dramatic changes per-release

• Support for upgrade / downgrade process that fits users needs
  • Stepwise upgrades / downgrades no longer required (can skip minor versions)
  • Reduced service disruption in upgrade / downgrade process
  • Reduced ops time in upgrade / downgrade process
• Clear client / server version compatibility
  • Support version skew for N releases
  • Clearly define which versions of client tools work with versions of server

[thockin]

The idea of skipping releases has been much on my mind. Very many things we want to do involve deprecating an old behavior, and as engineers we want to clean up later. That requires some overlap, which we have specced in the deprecation policy doc. The ability to do this depends on the user upgrading in an orderly way and not skipping releases. To allow skipping we basically can never get rid of anything. That doesn't seem tenable either. We get a lot of things wrong. Additionally, any-to-any seems like a test impossibility. Am I over-reacting? How do we give users sane upgrades without cutting off our own feet?

…

On May 31, 2017 8:14 AM, "Phillip Wittrock" ***@***.***> wrote: I tend to think of more frequent releases as 2 distinct problems, each of which are really challenging and will require a lot of effort. Supporting the ability to perform a release at a moments notice Target: developers Benefit: stable and orderly release process - Decouple code from set of enabled features - Flag guard features until fully certified as ready to enable - Push on green - 100% confidence in test coverage ability to verify release - Dashboards with clear signal - ok / not ok - Continuous stability - Non-flaky tests - Immediately rollback PRs that break tests Supporting publishing frequent releases Target: End users Benefit: Features available sooner, less dramatic changes per-release - Support for upgrade / downgrade process that fits users needs - Stepwise upgrades / downgrades no longer required (can skip minor versions) - Reduced service disruption in upgrade / downgrade process - Reduced ops time in upgrade / downgrade process - Clear client / server version compatibility - Support version skew for N releases - Clearly define which versions of client tools work with versions of server — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#567 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFVgVL8PotdPqkA1rDVQT1s1_eef2g7Aks5r_YPZgaJpZM4NE4MT> .

[jagosan]

What if we put some effort into automating the process of rolling forward? That is, the user would choose to jump from 1.4 to 1.7, and the system would incrementally move through interim releases in an automated process? Agree that allowing users to skip removes technical flexibility and adds tremendously (I would argue unacceptably) to the upgrade test complexity. On Jun 10, 2017 11:14 PM, "Tim Hockin" <notifications@github.com> wrote: The idea of skipping releases has been much on my mind. Very many things we want to do involve deprecating an old behavior, and as engineers we want to clean up later. That requires some overlap, which we have specced in the deprecation policy doc. The ability to do this depends on the user upgrading in an orderly way and not skipping releases. To allow skipping we basically can never get rid of anything. That doesn't seem tenable either. We get a lot of things wrong. Additionally, any-to-any seems like a test impossibility. Am I over-reacting? How do we give users sane upgrades without cutting off our own feet?

On May 31, 2017 8:14 AM, "Phillip Wittrock" ***@***.***> wrote: I tend to think of more frequent releases as 2 distinct problems, each of which are really challenging and will require a lot of effort. Supporting the ability to perform a release at a moments notice Target: developers Benefit: stable and orderly release process - Decouple code from set of enabled features - Flag guard features until fully certified as ready to enable - Push on green - 100% confidence in test coverage ability to verify release - Dashboards with clear signal - ok / not ok - Continuous stability - Non-flaky tests - Immediately rollback PRs that break tests Supporting publishing frequent releases Target: End users Benefit: Features available sooner, less dramatic changes per-release - Support for upgrade / downgrade process that fits users needs - Stepwise upgrades / downgrades no longer required (can skip minor versions) - Reduced service disruption in upgrade / downgrade process - Reduced ops time in upgrade / downgrade process - Clear client / server version compatibility - Support version skew for N releases - Clearly define which versions of client tools work with versions of server — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#567 (comment) , or mute the thread <https://github.com/notifications/unsubscribe-

auth/AFVgVL8PotdPqkA1rDVQT1s1_eef2g7Aks5r_YPZgaJpZM4NE4MT>

.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#567 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAI0YP4vF4YJxqykLvzRSTS-qDwz7P4uks5sC1ufgaJpZM4NE4MT> .

[idvoretskyi]

@jagosan

That is, the user would choose to jump from 1.4 to 1.7, and the system
would incrementally move through interim releases in an automated process?

Hmm, can you describe this idea with more details?

[bgrant0607]

For the record, release history.

1.0 was 7/21/2015

Subsequent releases:
http://blog.kubernetes.io/2015/11/Kubernetes-1-1-Performance-upgrades-improved-tooling-and-a-growing-community.html
http://blog.kubernetes.io/2016/03/Kubernetes-1.2-even-more-performance-upgrades-plus-easier-application-deployment-and-management-.html
http://blog.kubernetes.io/2016/07/kubernetes-1.3-bridging-cloud-native-and-enterprise-workloads.html
http://blog.kubernetes.io/2016/09/kubernetes-1.4-making-it-easy-to-run-on-kuberentes-anywhere.html
http://blog.kubernetes.io/2016/12/kubernetes-1.5-supporting-production-workloads.html
http://blog.kubernetes.io/2017/03/kubernetes-1.6-multi-user-multi-workloads-at-scale.html
http://blog.kubernetes.io/2017/06/kubernetes-1.7-security-hardening-stateful-application-extensibility-updates.html

[bgrant0607]

cc @monopole

[G-Harmon]

/cc me.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[idvoretskyi]

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[mrbobbytables]

Is this still relevant? It's been open and frozen for some time now.
I'm going to mark it as rotten for now, if it should be left open please update or re-freeze it 👍

/remove-lifecycle frozen
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.