Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kubeadm generate kubelet serving certs with master(s) CA #1223

Open
raravena80 opened this issue Nov 9, 2018 · 17 comments
Open

kubeadm generate kubelet serving certs with master(s) CA #1223

raravena80 opened this issue Nov 9, 2018 · 17 comments

Comments

@raravena80
Copy link

@raravena80 raravena80 commented Nov 9, 2018

Is this a BUG REPORT or FEATURE REQUEST?

/kind bug

Opening the kubeadm side for this issue on the metrics-server

Versions

$ kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.1", GitCommit:"4ed3216f3ec431b140b1d899130a69fc671678f4", GitTreeState:"clean", BuildDate:"2018-10-05T16:43:08Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}

Environment:

  • Kubernetes version (use kubectl version):
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.1", GitCommit:"4ed3216f3ec431b140b1d899130a69fc671678f4", GitTreeState:"clean", BuildDate:"2018-10-05T16:46:06Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.1", GitCommit:"4ed3216f3ec431b140b1d899130a69fc671678f4", GitTreeState:"clean", BuildDate:"2018-10-05T16:36:14Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}
  • Cloud provider or hardware configuration:
    Any
  • OS (e.g. from /etc/os-release):
    Any
  • Kernel (e.g. uname -a):
$ uname -a
Linux ip-172-31-1-118 4.15.0-1023-aws #23-Ubuntu SMP Mon Sep 24 16:31:06 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
  • Others:

What happened?

kubeadm creates certs under /var/lib/kubelet/pki/kubelet.* signed with a different CA from the one under /etc/kubernetes/pki/ca.pem

What you expected to happen?

As a result some apps like the metrics-server cannot collect stats from a secured kubelet because the kubelet has certs signed by a different ca from the K8s master(s)

Error sample:

E1108 23:49:32.090084       1 manager.go:102] unable to fully collect metrics: [unable to fully scrape metrics from source kubelet_summary:ip-x-x-x-x: unable to fetch metrics from Kubelet ip-x-x-x-x (ip-x-x-x-x): Get https://ip-x-x-x-x:10250/stats/summary/: x509: certificate signed by unknown authority, unable to fully scrape metrics from source kubelet_summary:ip-x-x-x-x: unable to fetch metrics from Kubelet ip-x-x-x-x (ip-x-x-x-x): Get https://ip-x-x-x-x:10250/stats/summary/: x509: certificate is valid for x.x.x.x not ip-x-x-x-x]

How to reproduce it (as minimally and precisely as possible)?

Install the metrics-server on run:

$ kubectl -n kube-system logs

Anything else we need to know?

Some more background here

There also steps in there that I followed to fix the issue.

@fabriziopandini

This comment has been minimized.

Copy link
Member

@fabriziopandini fabriziopandini commented Nov 9, 2018

@raravena80 I'm not aware of any certificate created by kubeadm under /var/lib/kubelet/pki/.. could you kindly provide more info? e.g kubeadm config files, steps to create the cluster

@raravena80

This comment has been minimized.

Copy link
Author

@raravena80 raravena80 commented Nov 12, 2018

@fabriziopandini I'm not entirely sure if the certs are created by the kubeadm, but the general procedure is described here.

This is what the content of the directory looks like:

root@ip-172-31-1-118:/var/lib/kubelet/pki# pwd
/var/lib/kubelet/pki
root@ip-172-31-1-118:/var/lib/kubelet/pki# ls -al
total 24
drwxr-xr-x 2 root root 4096 Jul 23 21:10 .
drwxr-xr-x 7 root root 4096 Nov 12 04:52 ..
-rw------- 1 root root 2810 Jul 23 21:09 kubelet-client-2018-07-23-21-09-53.pem
-rw------- 1 root root 1159 Jul 23 21:10 kubelet-client-2018-07-23-21-10-43.pem
lrwxrwxrwx 1 root root   59 Jul 23 21:10 kubelet-client-current.pem -> /var/lib/kubelet/pki/kubelet-client-2018-07-23-21-10-43.pem
-rw-r--r-- 1 root root 1501 Nov  8 23:53 kubelet.crt
-rw------- 1 root root 1679 Nov  8 23:53 kubelet.key
root@ip-172-31-1-118:/var/lib/kubelet/pki#

Are they kubelet.crt and kubelet.key files created by the kubelet the first time it loads?

@fabriziopandini

This comment has been minimized.

Copy link
Member

@fabriziopandini fabriziopandini commented Nov 12, 2018

@raravena80 thanks for the clarification
Probably I don't have the full context here so I leave room to others for answering.

Only one side note (might be it can help)
Kubeadm already creates a certificate named apiserver-kubelet-client for letting the api server talking securely with the kubelets; it is signed by ca and bound to necessary RBAC rules.

@timothysc

This comment has been minimized.

Copy link
Member

@timothysc timothysc commented Jan 7, 2019

/assign @liztio

@timothysc timothysc added this to the v1.14 milestone Jan 7, 2019
@anitgandhi

This comment has been minimized.

Copy link

@anitgandhi anitgandhi commented Jan 15, 2019

I think this is to pre-generate kubelet's server certs. I tinkered with trying to use the Kubelet flags for TLS server bootstrap and rotate server certs, unfortunately I could not get Kubelet to request a server cert for itself using the bootstrap token. Kubelet ends up falling back to its default behavior for server certs, which is to generate a self-signed one.

To the best of my knowledge, right now the only way around that is to generate Kubelet's server certs out-of-band and place them at a deterministic path and kubelet (configured by kubeadm) will pick it up, and set some kubelet flags accordingly; reference: https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet-tls-bootstrapping/#client-and-serving-certificates

The apiserver-kubelet-client is the client cert that API server will present to a kubelet, but kubelet is configured to trust clients that are signed by the k8s CA:

# cat /var/lib/kubelet/config.yaml 
address: 0.0.0.0
apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 2m0s
    enabled: true
  x509:
    clientCAFile: /etc/kubernetes/pki/ca.crt

It's the kubelet's identity as a server being presented that needs to be signed by the k8s CA, which comes around to the original question.

There's also some relevant discussion at the end of this thread: #118

I think kubeadm may have to add a CSR approver for server cert requests with a valid bootstrap token, just like it does for client cert requests?

@kfox1111

This comment has been minimized.

Copy link

@kfox1111 kfox1111 commented Feb 6, 2019

What about having the kubelet upload its self signed ca to a configmap somewhere? the nodeadmission plugin could restrict it to just its own configmap. metrics-server can use it to contact the node.

@stgleb

This comment has been minimized.

Copy link

@stgleb stgleb commented Feb 13, 2019

Any ideas about that?

@alexbrand

This comment has been minimized.

Copy link
Member

@alexbrand alexbrand commented Mar 13, 2019

If possible, I think we should use the TLS bootstrapping facilities built into the kubelet for requesting/rotating serving certificates.

@stgleb

This comment has been minimized.

Copy link

@stgleb stgleb commented Mar 13, 2019

@alexbrand I agree on that

@brokenjacobs

This comment has been minimized.

Copy link

@brokenjacobs brokenjacobs commented Apr 13, 2019

kubelet TLS bootstrapping only generates client certificates for whatever reason:
--bootstrap-kubeconfig string
Path to a kubeconfig file that will be used to get client certificate for kubelet. If the file specified by --kubeconfig does not exist, the bootstrap kubeconfig is used to request a client certificate from the API server. On success, a kubeconfig file referencing the generated client certificate and key is written to the path specified by --kubeconfig. The client certificate and key file will be stored in the directory pointed by --cert-dir.

And kubeadm already does this. Perhaps this is a kubelet feature request?

@neolit123 neolit123 added sig/auth and removed sig/auth labels Apr 13, 2019
@neolit123

This comment has been minimized.

Copy link
Member

@neolit123 neolit123 commented Apr 13, 2019

lets summarize the issue:

as outlined by @anitgandhi:
#1223 (comment)

the problem with kubeadm here is that we are not passing a couple of flags to the kubelet:

--tls-cert-file=<some-path>/kubelet.crt
--tls-private-key-file=<some-path>/kubelet.key

without these flags the kubelet defaults to self-signing it's serving certificate when it first runs, which can be verified with:

sudo openssl verify -verbose -CAfile /var/lib/kubelet/pki/kubelet.crt /var/lib/kubelet/pki/kubelet.crt

with a self-signed certificate instead of certificate signed by the cluster CA (/etc/kubernetes/ca.crt), deployments like the metrics server cannot scrape the kubelet, because the self-signed cert SAN will only include DNS:hostname.

possible solutions:
A) implement singing of a new kubelet.crt/key pair, ideally under /var/lib/kubelet/pki and setting the extra kubelet flags --tls-cert-file, --tls-private-key-file.

B) document means to enable this on demand similarly to how @raravena80 did it here: https://stackoverflow.com/questions/53212149/x509-certificate-signed-by-unknown-authority-kubeadm/53218524#53218524
except possibly using Kubernetes CSRs / kubeadm commands.

C) as commented by @alexbrand

If possible, I think we should use the TLS bootstrapping facilities built into the kubelet for requesting/rotating serving certificates.

D) ?

@kubernetes/sig-cluster-lifecycle
to me this seems in the space between bug/feature.

also see:
https://github.com/kubernetes/community/pull/602/files

@anitgandhi

This comment has been minimized.

Copy link

@anitgandhi anitgandhi commented Apr 15, 2019

I think something between options B+C should be done since a lot of the bootstrap token client cert/CSR logic kubelet+kubeadm would have common logic to this.

@timothysc timothysc modified the milestones: v1.15, Next May 10, 2019
@DanyC97

This comment has been minimized.

Copy link

@DanyC97 DanyC97 commented Jun 4, 2019

lets summarize the issue:

as outlined by @anitgandhi:
#1223 (comment)

the problem with kubeadm here is that we are not passing a couple of flags to the kubelet:

--tls-cert-file=<some-path>/kubelet.crt
--tls-private-key-file=<some-path>/kubelet.key

without these flags the kubelet defaults to self-signing it's serving certificate when it first runs, which can be verified with:

sudo openssl verify -verbose -CAfile /var/lib/kubelet/pki/kubelet.crt /var/lib/kubelet/pki/kubelet.crt

with a self-signed certificate instead of certificate signed by the cluster CA (/etc/kubernetes/ca.crt), deployments like the metrics server cannot scrape the kubelet, because the self-signed cert SAN will only include DNS:hostname.

possible solutions:
A) implement singing of a new kubelet.crt/key pair, ideally under /var/lib/kubelet/pki and setting the extra kubelet flags --tls-cert-file, --tls-private-key-file.

B) document means to enable this on demand similarly to how @raravena80 did it here: https://stackoverflow.com/questions/53212149/x509-certificate-signed-by-unknown-authority-kubeadm/53218524#53218524
except possibly using Kubernetes CSRs / kubeadm commands.

C) as commented by @alexbrand

If possible, I think we should use the TLS bootstrapping facilities built into the kubelet for requesting/rotating serving certificates.

D) ?

@kubernetes/sig-cluster-lifecycle
to me this seems in the space between bug/feature.

also see:
https://github.com/kubernetes/community/pull/602/files

great summary @neolit123 . Do you know if this will slip onto next cycle or work in progress as we speak ? Asking mainly because of the metrics-server which imo every deployment wants to have it ;)

@neolit123

This comment has been minimized.

Copy link
Member

@neolit123 neolit123 commented Jun 4, 2019

@randomvariable mentioned that there is another workaround for that.
from the discussions thus far, we are hesitant about signing the kubelet-serving cert with the cluster CA. this topic needs further discussion.

@astrieanna

This comment has been minimized.

Copy link

@astrieanna astrieanna commented Jun 4, 2019

/remove-help

because the solution to implement has not been chosen yet.

@NeilW

This comment has been minimized.

Copy link

@NeilW NeilW commented Nov 4, 2019

Any movement on this? I'm running up against it to support autoscaling features within a kubeadm deployed cluster.

Current workaround is to turn off CA checking of the kubelet certificate.

helm install --set 'args={--kubelet-insecure-tls}' --namespace kube-system metrics stable/metrics-serve
@neolit123

This comment has been minimized.

Copy link
Member

@neolit123 neolit123 commented Nov 4, 2019

not really, it's blocked on design proposals.
there are a number of workarounds but the work of documenting those stalled:
#1602

--kubelet-insecure-tls

this might not be ideal for all users.

@neolit123 neolit123 changed the title kubeadm generate kubelet certs with master(s) CA kubeadm generate kubelet serving certs with master(s) CA Nov 13, 2019
@neolit123 neolit123 modified the milestones: v1.17, Next Nov 13, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.