Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kube-apiserver 1.13.x refuses to work when first etcd-server is not available. #72102

Closed
Cytrian opened this issue Dec 17, 2018 · 52 comments · Fixed by etcd-io/etcd#10476, etcd-io/etcd#10911 or #81434

Comments

@Cytrian
Copy link

commented Dec 17, 2018

How to reproduce the problem:
Set up a new demo cluster with kubeadm 1.13.1.
Create default configurationwith kubeadm config print init-defaults
Initialize cluster as usual with kubeadm init

Change the --etcd-servers list in kube-apiserver manifest to --etcd-servers=https://127.0.0.2:2379,https://127.0.0.1:2379, so that the first etcd node is unavailable ("connection refused").

The kube-apiserver is then not able to connect to etcd any more.

Last message: Unable to create storage backend: config (\u0026{ /registry [https://127.0.0.2:2379 https://127.0.0.1:2379] /etc/kubernetes/pki/apiserver-etcd-client.key /etc/kubernetes/pki/apiserver-etcd-client.crt /etc/kubernetes/pki/etcd/ca.crt true 0xc000381dd0 \u003cnil\u003e 5m0s 1m0s}), err (dial tcp 127.0.0.2:2379: connect: connection refused)\n","stream":"stderr","time":"2018-12-17T12:13:19.608822816Z"}

kube-apiserver does not start.

If I upgrade etcd to version 3.3.10, it reports an error remote error: tls: bad certificate", ServerName ""

Environment:

  • Kubernetes version 1.13.1
  • kubeadm in Vagrant box

I also experience this bug in an environment with a real etcd cluster.

/kind bug

@Cytrian

This comment has been minimized.

Copy link
Author

commented Dec 17, 2018

/sig api-machinery

@yue9944882

This comment has been minimized.

Copy link
Member

commented Dec 17, 2018

/remove-sig api-machinery
/sig cluster-lifecycle

@yue9944882

This comment has been minimized.

Copy link
Member

commented Dec 17, 2018

/sig api-machinery

apologies, just had another look and it's indeed an api-machinery issue.

// Endpoints defines a set of URLs (schemes, hosts and ports only)
// that can be used to communicate with a logical etcd cluster. For
// example, a three-node cluster could be provided like so:
//
// Endpoints: []string{
// "http://node1.example.com:2379",
// "http://node2.example.com:2379",
// "http://node3.example.com:2379",
// }
//
// If multiple endpoints are provided, the Client will attempt to
// use them all in the event that one or more of them are unusable.
//
// If Client.Sync is ever called, the Client may cache an alternate
// set of endpoints to continue operation.

we are passing the server list straight into etcd v3 client which return the error u reported. not sure if it's designed

@JishanXing

This comment has been minimized.

Copy link
Contributor

commented Dec 20, 2018

This is an etcdv3 client issue. See etcd-io/etcd#9949

@fedebongio

This comment has been minimized.

Copy link
Contributor

commented Dec 20, 2018

/cc @jpbetz

@timothysc

This comment has been minimized.

Copy link
Member

commented Feb 1, 2019

/assign @timothysc @detiber

So live updating a static pod manifest is typically not recommended, was this triggered via some other operation or were you editing your static manifests?

@Cytrian

This comment has been minimized.

Copy link
Author

commented Feb 1, 2019

No pod manifest involved here. Just a group of etcd and a kube-apiserver. The issue appeared when we rebooted the first etcd node.

@alexbrand

This comment has been minimized.

Copy link
Member

commented Feb 4, 2019

I was able to repro this issue with the repro steps provided by @Cytrian. I also reproduced this issue with a real etcd cluster.

As @JishanXing previously mentioned, the problem is caused by a bug in the etcd v3 client library (or perhaps the grpc library). The vault project is also running into this: hashicorp/vault#4349

The problem seems to be that the etcd library uses the first node’s address as the ServerName for TLS. This means that all attempts to connect to any server other than the first will fail with a certificate validation error (i.e. cert has ${nameOfNode2} in SANs, but the client is expecting ${nameOfNode1}).

An important thing to highlight is that when the first etcd server goes down, it also takes the Kubernetes API servers down, because they fail to connect to the remaining etcd servers.

With that said, this all depends on what your etcd server certificates look like:

  • If you follow the kubeadm instructions to stand up a 3 node etcd cluster, you get a set of certificates that include the first node’s name and IP in the SANs (because all certs are generated on the first etcd node). Thus, you should not run into this issue.
  • If you have used another process to generate certificates for etcd, and the certs do not include the first node’s name and IP in the SANs, you will most likely run into this issue when the first etcd node goes down.

To reproduce the issue with a real etcd cluster:

  1. Create a 3 node etcd cluster with TLS enabled. Each certificate should only contain the name/IP of the node that will be serving it.
  2. Start an API server that points to the etcd cluster.
  3. Stop the first etcd node.
  4. API server crashes and fails to come back up

Versions:

  • kubeadm version: v1.13.2
  • kubernetes api server version: v1.13.2
  • etcd image: k8s.gcr.io/etcd:3.2.24

API server crash log: https://gist.github.com/alexbrand/ba86f506e4278ed2ada4504ab44b525b

I was unable to reproduce this issue with API server v1.12.5 (n.b. this was somewhat of a non-scientific test => tested by updating the image field of the API server static pod produced by kubeadm v1.13.2)

@timothysc

This comment has been minimized.

Copy link
Member

commented Feb 4, 2019

@timothysc

This comment has been minimized.

Copy link
Member

commented Feb 4, 2019

@liggitt ^ FYI.

@neolit123

This comment has been minimized.

Copy link
Member

commented Feb 6, 2019

thank you for the investigation @alexbrand

@gyuho

This comment has been minimized.

Copy link
Member

commented Aug 1, 2019

@dims Sure. The earliest we can do is August 13, 2019.

  1. We need gRPC 1.23 to include upstream bug fixes https://github.com/grpc/grpc-go/milestone/21
  2. We need a day or two to rewrite 3.3 client balancer

Today, we do etcd 3.4 release code freeze, which means we will start running functional (failure injection) tests + kubemark to test new etcd client changes including etcd-io/etcd#10911.

If anything changes, we will post updates here.

@dims

This comment has been minimized.

Copy link
Member

commented Aug 1, 2019

Thanks a ton @gyuho !

@DaniDD

This comment has been minimized.

Copy link

commented Aug 2, 2019

Hi, is this the same issue:
I have 3 control planes. On every control plane are one etcd-server. The apiserver can only connect to one etcd-server. If I enter more then one etcd server in the kube-apiserver.conf I have Warnings in the log:
clientconn.go:1251] grpc: addrConn.createTransport failed to connect to {172.22.75.8:2379 0 }. Err :connection error: desc = "transport: authentication handshake f
ailed: x509: certificate is valid for 172.22.75.8, 127.0.0.1, ::1, 172.22.75.8, not 172.22.75.7". Reconnecting...

The certificates are correct. I have this check with curl -v --cert apiserver-etcd-client.crt --key apiserver-etcd-client.key --cacert etcd/ca.crt ...

@figo

This comment has been minimized.

Copy link
Member

commented Aug 2, 2019

@dims Sure. The earliest we can do is August 13, 2019.

  1. We need gRPC 1.23 to include upstream bug fixes https://github.com/grpc/grpc-go/milestone/21
  2. We need a day or two to rewrite 3.3 client balancer

Today, we do etcd 3.4 release code freeze, which means we will start running functional (failure injection) tests + kubemark to test new etcd client changes including etcd-io/etcd#10911.

If anything changes, we will post updates here.

@gyuho a quick clarification, is August 13, 2019 the earliest day the fix going to land on 3.3? thx

@gyuho

This comment has been minimized.

Copy link
Member

commented Aug 3, 2019

August 13, 2019 the earliest day the fix going to land on 3.3

Yes

@gyuho

This comment has been minimized.

Copy link
Member

commented Aug 13, 2019

Update: https://github.com/grpc/grpc-go/releases/tag/v1.23.0 is out. Bumping up gRPC etcd-io/etcd#11029 in etcd master branch, in addition to Go runtime upgrade https://groups.google.com/forum/#!topic/golang-announce/65QixT3tcmg. Once tests look good, we will start working on 3.3 backports.

@gyuho

This comment has been minimized.

Copy link
Member

commented Aug 14, 2019

@dims @jpbetz https://github.com/etcd-io/etcd/releases/tag/v3.3.14-beta.0 has been released with all the fixes. Please try. Once tests look good in the next few days, I will release v3.3.14.

Update: https://github.com/etcd-io/etcd/releases/tag/v3.3.14 https://github.com/etcd-io/etcd/releases/tag/v3.3.15 has been released.

@dims

This comment has been minimized.

Copy link
Member

commented Aug 14, 2019

@gyuho i just kicked off a WIP on the k/k CI - #81434

@gyuho

This comment has been minimized.

Copy link
Member

commented Aug 15, 2019

Some tests in k8s may break. We need pass grpc.WithBlock with the new etcd client. Please take a look at #81435.

@ylhyh

This comment has been minimized.

Copy link

commented Aug 21, 2019

Is there a hotfix for v1.15?

@dims

This comment has been minimized.

Copy link
Member

commented Aug 21, 2019

@igcherkaev

This comment has been minimized.

Copy link

commented Aug 29, 2019

The issue is still there with Kubernetes 1.14.6 and etcd 3.3.15. Any changes to kubernetes libs or code needed to tackle this issue?

@dims

This comment has been minimized.

Copy link
Member

commented Aug 29, 2019

@igcherkaev

This comment has been minimized.

Copy link

commented Aug 30, 2019

@dims hmm, if there are changes required in k8s source code, why has this issue got closed? I'll try to build a custom image and test, though. Thanks.

@liggitt

This comment has been minimized.

Copy link
Member

commented Aug 30, 2019

this will be resolved in 1.16 in #81434

the changes required (upgrading etcd client libraries) were too invasive to backport to patch releases

@igcherkaev

This comment has been minimized.

Copy link

commented Aug 30, 2019

@liggitt that leaves people on 1.13-1.15 without proper H/A? I think this issue deserves to be fixed in the three supported releases of Kubernetes. The hotfix mentioned here looks simple enough to be added, but you are saying the proper fix will require much more. So this is all sorta obscure and confusing to the community, IMO.

Not that I am complaining, don't get me wrong, it's just I think everyone would welcome some clarity on this issue. Maybe document it somewhere and provide some workarounds for the people who are still on v1.13.-1.15? Because right now, bring down the first etcd member and oops, api is not working, cluster is not working.

@liggitt

This comment has been minimized.

Copy link
Member

commented Aug 30, 2019

agree on documenting the issue at the very least, and recommending any possible workarounds

@jpbetz @gyuho, any feedback on ^?

@gyuho

This comment has been minimized.

Copy link
Member

commented Aug 30, 2019

@liggitt @igcherkaev I will work on the documentation.

@igcherkaev

This comment has been minimized.

Copy link

commented Aug 30, 2019

@dims just built custom 1.14 image from the release-1.14 branch and patched that credentials.go file and it's so much better now when I bring down the first etcd node down. Now I am confused, if the fix is so simple, why does it have to wait until 1.16?

@gyuho

This comment has been minimized.

Copy link
Member

commented Aug 30, 2019

@igcherkaev The fix requires upgrading google.golang.org/grpc from v1.7.5 to v1.23.0, which is quite a big change :0

@gyuho

This comment has been minimized.

Copy link
Member

commented Aug 30, 2019

I am adding "Known issue" section to etcd docs here kubernetes/website#16156.

@dims

This comment has been minimized.

Copy link
Member

commented Aug 30, 2019

@igcherkaev see what @gyuho said :)

@javendo

This comment has been minimized.

Copy link

commented Sep 10, 2019

Some days ago I had opened this issue #81837, but reading this one I think it is related. Can anyone take a look at my issue and see it they are related and if so I can close it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.