Communicate etcd2 support deprecation timeline

[jpbetz]

In the Kubernetes 1.9 release notes, it was announced that etcd2 support is deprecated and will be removed in "Kubernetes 1.13 or 1.14".

Unfortunately, not all cluster operators read the release notes in their entirety. Let's kick the rhetoric up a notch:

• Send out a email to kubernetes-announce explaining the change.
• Add a matrix to the kubernetes documentation with the supported etcd version of each kubernetes version.
• Update the description of the --storage-backend flag in kube-apiserver to clearly state that etcd2 support is deprecated and being removed in 1.13.
• Log warning in kube-apiserver logs when using etcd2 that it is deprecated.

[dims]

@jpbetz : why add a new option just to remove it in a bit? can't we just get them to use --storage-backend=etcd2 if they really wish to? will this new option do something different?

[jpbetz]

@dims To aggressively guide the k8s user base off of etcd2 so we don't get to 1.13 (or 1.14) and still have a large user base still using etcd2.

The idea is to introduce the flag as a means of communicating to kubernetes cluster operator that they really should upgrade to etcd 3 when they upgrade to kubernetes 1.10, and that if they decide to stay on etcd2, that it's deprecated and going away in a few releases. It's a little friendlier than just dropping etcd2 support in 1.10 without sufficient warning, but also high friction enough to force cluster operators to seriously consider upgrading from etcd2 to etcd3 as part of their kubernetes 1.10 upgrade. Even if they don't read release notes, documentation or kubernetes-announce emails, they'll see this.

We could use the --storage-backend flag if we removed the etcd2 option and replaced it with a deprecated-etcd2 option. This way the operator is still forced to make a config change and, in doing so, acknowledge the deprecation of etcd2. Does that seem cleaner? I've updated the issue description to match.

[dims]

+1 @jpbetz i like it. Thanks!

[liggitt]

Replace the --storage-backend=etcd2 flag option with --storage-backend=deprecated-etcd2 in the kube-apiserver flags, and add a note in the flag description clearly stating that etcd2 support is deprecated and will be removed in 1.13.

We should not do this. Changing the flag of a deprecated option defeats the point of the deprecation period. It is a breaking change (an upgraded cluster would not come back up) prior to the end of the deprecation period.

Agree on raising the visibility of the announcement, but not to changing the required flag value.

[jpbetz]

@liggitt Hm. I do agree that changing the flag violates backward compatibility. This is tough because etcd2 in k8s 1.8+ is effectively unmaintained. How about we just update the --storage-backend description to clearly state that etcd2 is deprecated for now and see how that goes? I'll update this issues description to match.

[liggitt]

updating the description and printing a warning when starting in that mode both seem reasonable

[dims]

thanks @jpbetz @liggitt it's even better than before :)

[Deshke]

if etcd2 support is dropped, a migration guide from 2 to 3 should be in place (and working)

[liggitt]

if etcd2 support is dropped, a migration guide from 2 to 3 should be in place (and working)

It is the responsibility of deployments/installers that set up etcd to manage etcd2->etcd3 migrations. GCE/GKE has done single member migrations and Openshift has done multi-member migrations. I agree we should gather documentation for both scenarios, but automation/testing/support of a particular migration belongs to the tooling that set up the etcd cluster.

[jpbetz]

+1 to what @liggitt said.

The responsibilities are:

• etcd - Documentation of how to perform etcd upgrades, what versions of etcd are supported/ actively maintained. This exists and is in good shape
• kubernetes distros (GKE, Openshift, AKS, EKS,...) - Documentation of how to do etcd upgrades via their tools/infrastructure
• Kubernetes: Document which versions of etcd are supported by which versions of k8s, test these version pairing work together correctly

[Deshke]

@liggitt was more of a wave to kops

[jberkus]

If you plan to do this for the 1.10 release, please add a kind/ and priority/ labels to this issue. Otherwise, it will be automatically removed from 1.10 milestone as of Code Slush Tuesday.

[jberkus]

Are we announcing deprecation in 1.10? Please create a PR for the docs/release notes. Otherwise we'll postpone this to 1.11.

[jberkus]

@jpbetz @liggitt

[jpbetz]

@jberkus The official announcement for etcd2 deprecation was part of 1.9. This issue is just about turning up the volume in how we communicate that etcd2 support will be removed, I'll move this to 1.11 since I don't think we're going to get it all done for 1.10.

[k8s-github-robot]

[MILESTONENOTIFIER] Milestone Removed From Issue

@jpbetz

Important: This issue was missing labels required for the v1.11 milestone for more than 3 days:

kind: Must specify exactly one of kind/bug, kind/cleanup or kind/feature.
priority: Must specify exactly one of priority/critical-urgent, priority/important-longterm or priority/important-soon.

Help

• Additional instructions
• Commands for setting labels

[wojtek-t]

Apparently node-e2e tests are still running etcd v2 underneath:
https://github.com/kubernetes/kubernetes/blob/master/test/e2e_node/services/apiserver.go#L45

Fixing that should be prioritized.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[jpbetz]

/remove-lifecycle rotten

[jpbetz]

/remove-lifecycle stale

[tao12345666333]

Please see kubernetes/enhancements#622.

[robinsondan87]

Sorry if this is the incorrect thread but will dropping support for etcd2 also include dropping support for the v2 API as v2 can be run whilst on the etcd 3.x releases and is by default if the ENV isn't specified.

[liggitt]

will dropping support for etcd2 also include dropping support for the v2 API as v2 can be run whilst on the etcd 3.x releases and is by default if the ENV isn't specified.

Yes, support for the v2 API is what is being dropped.

[dannyk81]

@liggitt @robinsondan87 I'd like to cross-link this issue with flannel-io/flannel#554.

My understanding is that starting K8s 1.13, the api-server won't support etcd v2 API as a storage backend, however flannel (details in the linked issue above) also uses etcd with v2 API to store the subnet/config information.

There seems to be some confusion, whether flannel will be able to operate in K8s 1.13 given this deprecation and unless I'm mistaken - it should not be affected, since etcd still supports that v2 API and flannel is interfacing with etcd directly.

Is there anything we're missing? and if that's not the case, the flannel/canal users community should be aware that they need to look for an alternative starting K8s 1.13.

Hope we can get some clarification here 😄

[dims]

@dannyk81 that's my understanding as well.

kubernetes 1.13 change log notes that etcd 3.2.24 is the recommended version:
https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.13.md#external-dependencies

3.2.24 etcd still supports the v2 API and flannel can use it directly. Note that as long as flannel is not trying to use the data stored by kubernetes api server (using v3 API), we should be good.

[liggitt]

etcd2 support was dropped in 1.13

/close

[k8s-ci-robot]

@liggitt: Closing this issue.

In response to this:

etcd2 support was dropped in 1.13

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.