kube-apiserver and TLS etc: kubectl reports unhealthy cluster #29330

xgerman · 2016-07-20T22:36:44Z

I am running v1.3.0-beta.1 and have set up etc, k8 with TLS. Here is the relevant part for the kube-apiserver:
--etcd_servers=https://192.168.200.2:2379
--etcd-cafile=/srv/kubernetes/ca.crt
--etcd-certfile=/srv/kubernetes/kubecfg.crt
--etcd-keyfile=/srv/kubernetes/kubecfg.key \

I tried it running without those parameters and kube-apiserver didn't start but failed with (from the logs):
etcd cluster is unavailable or misconfigured

So it can do something with those certs.

Now, when I run
vagrant@k8-master:~$ kubectl get cs
NAME STATUS MESSAGE ERROR
scheduler Healthy ok
controller-manager Healthy ok
etcd-0 Unhealthy Get https://192.168.200.2:2379/health: remote error: bad certificate

But when I run etcdctl using the SAM certificates:
vagrant@k8-master:~$ sudo etcdctl --debug --endpoints https://192.168.200.2:2379 --ca-file /srv/kubernetes/ca.crt --key-file /srv/kubernetes/kubecfg.key --cert-file /srv/kubernetes/kubecfg.crt cluster-health
Cluster-Endpoints: https://192.168.200.2:2379
cURL Command: curl -X GET https://192.168.200.2:2379/v2/members
member ce2a822cea30bfca is healthy: got healthy result from https://192.168.200.2:2379
cluster is healthy

Furthermore:
sudo etcdctl --debug --endpoints https://192.168.200.2:2379 --ca-file /srv/kubernetes/ca.crt --key-file /srv/kubernetes/kubecfg.key --cert-file /srv/kubernetes/kubecfg.crt ls registry
start to sync cluster using endpoints(https://192.168.200.2:2379)
cURL Command: curl -X GET https://192.168.200.2:2379/v2/members
got endpoints(https://192.168.200.2:2379) after sync
Cluster-Endpoints: https://192.168.200.2:2379
cURL Command: curl -X GET https://192.168.200.2:2379/v2/keys/registry?quorum=false&recursive=false&sorted=false
/registry/ranges
/registry/namespaces
/registry/events
/registry/services
/registry/serviceaccounts
/registry/secrets
/registry/deployments
/registry/replicasets
/registry/pods

I suspect that some code path in kube-apiserver isn't aware of the certificates?

elcct · 2016-08-24T10:50:57Z

I have the same issue.
I am running Kubernetes 1.3.5 with Etcd 2.3.7 on Ubuntu 16.04.

root@gc01:/opt/kubernetes/certs# $GOPATH/bin/etcdctl -ca-file=ca.pem -cert-file=client.pem -key-file=client-key.pem -endpoints=https://gc01.xxxxx:2379 ls registry
/registry/ranges
/registry/namespaces
/registry/services
/registry/serviceaccounts

root@gc01:/opt/kubernetes/certs# $GOPATH/bin/etcdctl -ca-file=ca.pem -cert-file=client.pem -key-file=client-key.pem -endpoints=https://gc01.xxxxx:2379 cluster-health
member 12fabe43e0b7020b is healthy: got healthy result from https://gc01.xxxxx:2379
member 3e9e6038432c0da7 is healthy: got healthy result from https://gc03.xxxxx:2379
member 5223f27d945df07a is healthy: got healthy result from https://gc05.xxxxx:2379
member 6c3dd0c34e4e7bee is healthy: got healthy result from https://gc02.xxxxx:2379
member e32849a38113a4f2 is healthy: got healthy result from https://gc04.xxxxx:2379
cluster is healthy

root@gc01:/opt/kubernetes/certs# /opt/kubernetes/bin/kubectl get cs
NAME                 STATUS      MESSAGE                                                                         ERROR
controller-manager   Healthy     ok
scheduler            Healthy     ok
etcd-0               Unhealthy   Get https://gc01.xxxxx:2379/health: remote error: bad certificate
etcd-3               Unhealthy   Get https://gc04.xxxxx:2379/health: remote error: bad certificate
etcd-4               Unhealthy   Get https://gc05.xxxxx:2379/health: remote error: bad certificate
etcd-2               Unhealthy   Get https://gc03.xxxxx:2379/health: remote error: bad certificate
etcd-1               Unhealthy   Get https://gc02.xxxxx:2379/health: remote error: bad certificate

My etcd part of apiserver configuration is the same is in the post above:

  --etcd-cafile=/opt/kubernetes/certs/ca.pem \
  --etcd-certfile=/opt/kubernetes/certs/client.pem \
  --etcd-keyfile=/opt/kubernetes/certs/client-key.pem \
  --etcd-servers=https://gc01.xxxxx:2379,https://gc02.xxxxx:2379,https://gc03.xxxxx:2379,https://gc04.xxxxx:2379,https://gc05.xxxxx:2379 \

What to do?

elcct · 2016-08-25T09:56:43Z

It seems like this is the same problem: #27343 (comment)

a9b3 · 2016-10-08T23:54:50Z

@elcct Hello did you ever find a solution to this?

elcct · 2016-11-07T16:13:55Z

@esayemm sadly not. Also tried Kubernetes 1.4.5 - doesn't work :(

sergeyfd · 2017-01-13T15:15:22Z

Still same issue in 1.5.1

upolymorph · 2017-01-19T07:03:38Z

Yes I have the same issue too with 1.5.1 and checked that curl with correct --cacert --cert --key option is able to get from etcd url/heath: {"health": "true"}

yawboateng · 2017-01-27T05:29:46Z

seeing the same issue in 1.5.2

elcct · 2017-01-27T12:33:11Z

It seems like the fix is in the making, there is a pull request:

#39716

strugglingyouth · 2017-03-15T02:33:20Z

Yes I have the same issue too .

etcdctl version: 3.1.3
API version: 3.1
Kubernetes v1.6.0-beta.1 (master and node)

config apiserver as follow:

--etcd-cafile='/var/run/kubernetes/ca.pem' --etcd-certfile='/var/run/kubernetes/client.pem' --etcd-keyfile='/var/run/kubernetes/client-key.pem' --client-ca-file='/var/run/kubernetes/ca.pem'

create deployment and other is right, etcd is ok.

$ kubectl get cs
NAME STATUS MESSAGE ERROR
controller-manager Healthy ok
scheduler Healthy ok
etcd-0 Unhealthy Get https://xxxx:2379/health: remote error: tls: bad certificate

javapapo · 2017-03-24T10:55:59Z

+1

Kubernetes : 1.5.3
Cloud Provider: AWS
Installed with : kube-aws

etcd-2               Unhealthy   Get https://xxxx.compute.internal:2379/health: remote error: tls: bad certificate
etcd-1               Unhealthy   Get https://xxxxxx.compute.internal:2379/health: remote error: tls: bad certificate
etcd-0               Unhealthy   Get https://xxxxxx.compute.internal:2379/health: remote error: tls: bad certificate

kgrvamsi · 2017-05-13T08:15:52Z

Does Kubectl have a option to provide the etcd key when trying to query the component status?
something like

kubectl --etcd-key key.pem get cs

k8s-github-robot · 2017-05-31T22:49:16Z

@xgerman There are no sig labels on this issue. Please add a sig label by:
(1) mentioning a sig: @kubernetes/sig-<team-name>-misc
(2) specifying the label manually: /sig <label>

Note: method (1) will trigger a notification to the team. You can find the team list here.

xgerman · 2017-06-06T18:57:21Z

@kubernetes/sig-ui

x1957 · 2017-06-16T01:44:20Z

[root@c3-cloudml-srv-ct01 etcd]# kubectl version
Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.4", GitCommit:"d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae", GitTreeState:"clean", BuildDate:"2017-05-19T18:44:27Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.4", GitCommit:"d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae", GitTreeState:"clean", BuildDate:"2017-05-19T18:33:17Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}

[root@c3-cloudml-srv-ct01 etcd]# etcdctl --ca-file=/root/ssl/etcd/ca.pem --cert-file=/root/ssl/etcd/etcd.pem --key-file=/root/ssl/etcd/etcd-key.pem --endpoints=https://xxxx:2379,https://xxxx:2379,https://xxxx:2379 cluster-health
2017-06-16 09:41:36.277313 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
2017-06-16 09:41:36.278113 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
member 269592983f1bbc6c is healthy: got healthy result from https://xxxx:2379
member 961ff05c1312685d is healthy: got healthy result from https://xxxx:2379
member d64dfe4c6ed3bddd is healthy: got healthy result from https://xxxx:2379
cluster is healthy

[root@c3-cloudml-srv-ct01 etcd]# kubectl get cs
NAME                 STATUS      MESSAGE                                                                    ERROR
controller-manager   Healthy     ok                                                                         
scheduler            Healthy     ok                                                                         
etcd-0               Unhealthy   Get https://xxxx:2379/health: remote error: tls: bad certificate   
etcd-2               Unhealthy   Get https://xxxx:2379/health: remote error: tls: bad certificate   
etcd-1               Unhealthy   Get https://xxxx:2379/health: remote error: tls: bad certificate

antoineco · 2017-06-21T18:39:10Z

Fixed in #39716. Will be in Kubernetes 1.7.

- Start distinguishing `master_host` into FreeIPA and k8s API servers - Why: - Dogtag CA and API server are memory-heavy; put on different machines - Eventual API server redundancy - Redo groups and hosts.yaml file - Replace `master_host` etc. with `freeipa_master_host` - Parallel IPA requests breaking - Multiple, parallel requests to IPA server result in "Unauthorized" errors - No clean way to serialize; separate IPA operations into plays and use `serial: 1` - IPA cert operations: put into a proper role - Other operations: handle individually - Update to k8s version 1.7.0 - Earlier versions have trouble with TLS to etcd - kubernetes/kubernetes#29330 - Update dns and dashboard addons and manifests - Generalize use of `etcd_cluster_token` -> `cluster_id` - README notes - Download kubeadm

* Updated to work with the bumped stdlib * Updated to use the new "proper" certs from the latest simp-beaker-helpers * Tests will fail on checking 'componentstatus' using 'kubectl' due to a known bug in Kubernetes < 1.7 per kubernetes/kubernetes#29330

* Adds parameters and code to manage ports related to kubernetes with simp/iptables * Tests will fail on checking 'componentstatus' using 'kubectl' due to a known bug in Kubernetes < 1.7 per kubernetes/kubernetes#29330 SIMP-4158 #close SIMP-4187 #close

avalonzst · 2020-01-13T03:12:35Z

problem still occurs for kubenetes1.6.0 with etcd 3.3.8
[root@umsk8s-master kubernetes]# etcdctl --version
etcdctl version: 3.3.8
API version: 2
[root@umsk8s-master kubernetes]# kube-apiserver --version
Kubernetes v1.6.0
[root@umsk8s-master kubernetes]# kubectl get cs
NAME STATUS MESSAGE ERROR
controller-manager Healthy ok
scheduler Healthy ok
etcd-0 Unhealthy Get https://172.30.251.200:20079/health: remote error: tls: bad certificate
etcd-1 Unhealthy Get https://172.30.251.201:20179/health: remote error: tls: bad certificate
etcd-4 Unhealthy Get https://172.30.251.204:20479/health: remote error: tls: bad certificate
etcd-2 Unhealthy Get https://172.30.251.202:20279/health: remote error: tls: bad certificate
etcd-3 Unhealthy Get https://172.30.251.203:20379/health: remote error: tls: bad certificate
[root@umsk8s-master kubernetes]#

apelisse added area/apiserver team/ux labels Jul 21, 2016

pwittrock added team/control-plane and removed team/ux labels Jul 21, 2016

sandys mentioned this issue Dec 27, 2016

Full TLS for cluster components kubernetes-sigs/kubespray#822

Closed

maltekrupa mentioned this issue Apr 15, 2017

etcd-01 Unhealthy Get https://10.240.0.11:2379/health: remote error: tls: bad certificate kelseyhightower/kubernetes-the-hard-way#156

Closed

k8s-github-robot added the needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. label May 31, 2017

0xmichalis closed this as completed Jun 24, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

kube-apiserver and TLS etc: kubectl reports unhealthy cluster #29330

kube-apiserver and TLS etc: kubectl reports unhealthy cluster #29330

xgerman commented Jul 20, 2016

elcct commented Aug 24, 2016

elcct commented Aug 25, 2016

a9b3 commented Oct 8, 2016

elcct commented Nov 7, 2016

sergeyfd commented Jan 13, 2017

upolymorph commented Jan 19, 2017

yawboateng commented Jan 27, 2017

elcct commented Jan 27, 2017

strugglingyouth commented Mar 15, 2017

javapapo commented Mar 24, 2017 •

edited

kgrvamsi commented May 13, 2017

k8s-github-robot commented May 31, 2017

xgerman commented Jun 6, 2017

x1957 commented Jun 16, 2017

antoineco commented Jun 21, 2017

avalonzst commented Jan 13, 2020

kube-apiserver and TLS etc: kubectl reports unhealthy cluster #29330

kube-apiserver and TLS etc: kubectl reports unhealthy cluster #29330

Comments

xgerman commented Jul 20, 2016

elcct commented Aug 24, 2016

elcct commented Aug 25, 2016

a9b3 commented Oct 8, 2016

elcct commented Nov 7, 2016

sergeyfd commented Jan 13, 2017

upolymorph commented Jan 19, 2017

yawboateng commented Jan 27, 2017

elcct commented Jan 27, 2017

strugglingyouth commented Mar 15, 2017

javapapo commented Mar 24, 2017 • edited

kgrvamsi commented May 13, 2017

k8s-github-robot commented May 31, 2017

xgerman commented Jun 6, 2017

x1957 commented Jun 16, 2017

antoineco commented Jun 21, 2017

avalonzst commented Jan 13, 2020

javapapo commented Mar 24, 2017 •

edited