Improve documentation on how authentication is handled in Kubernetes

[devurandom]

This is a follow up from #10265 (comment)

There I was asking about documentation on serviceaccounts and certificates. I have already read http://docs.k8s.io/design/service_accounts.md (and a few others), but still there are lots of questions:

• Is the CA certificate in --root-ca-file= authenticated by the key in --service-account-key-file=?
• If so, I cannot create the --service-account-key-file= on the host automatically via a systemd.service unit, unless I also automatically create the --root-ca-file= via a systemd.service, correct?
• Is the missing ca.crt secret in my default serviceaccount the same as the one I would hand to the kube-controller-manager via --root-ca-file=?
• Will I have to distribute these files to the client I run kubectl on?
• Will the kube-controller-manager and/or the kubelets automatically create certificates for themselves signed by --root-ca-file=?
• If so, how do I control which host is allowed to get such a certificate, and which will not be allowed to join the cluster?
• How is the certificate for connection with a client kubectl created? Do I need to also automatically create a client certificate using a systemd.service and then fetch the file using e.g. ssh?
• If so, I assume the automatic generation of the key as laid out in the examples is not actually recommended, as it is utterly cumbersome?
• I read that on GCE you use Salt to distribute the (manually created) root certificates. Is there any convenient way to do this on a bare-metal CoreOS cluster? Writing a properly indented write_files directive for cloud-config appears unflexible and cumbersome on its own.
• Is all authentication between the Kubernetes master (kube-apiserver, kube-controller-manager, kube-scheduler) and the kubelets and kubectls handled via HTTPS and client certificates?

The following question was already answered by @satnam6502 in #10265 (comment), but I mention it here for completeness (in case someone wants to write down docs or a FAQ):

• What is the token of the serviceaccount being used for?

One of the uses of the token is to make it more convenient to issue things like curl commands to the API server or application endpoints (e.g. Elasticsearch and KIbana logging endpoints). See https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/getting-started-guides/logging-elasticsearch.md

[devurandom]

@antoineco explained (#10265 (comment)) the certificate part a little bit more and wrote about what to do when creating a cluster:

1. The easiest way to generate your certificates is to use this script. Example:

./make-ca-cert.sh <master_ip> IP:<master_ip>,IP:10.0.0.1,DNS:kubernetes,DNS:kubernetes.default,DNS:kubernetes.default.svc,DNS:kubernetes.default.svc.cluster.local

2. Make sure you start your api server and controller manager (on the master) with the following flags:

kube-apiserver
  --client-ca-file=/srv/kubernetes/ca.crt
  --tls-cert-file=/srv/kubernetes/server.cert
  --tls-private-key-file=/srv/kubernetes/server.key
  --token-auth-file=/srv/kubernetes/known_tokens.csv
  --admission_control=NamespaceLifecycle,NamespaceExists,LimitRanger,SecurityContextDeny,ServiceAccount,ResourceQuota

kube-controller-manager
  --root-ca-file=/srv/kubernetes/ca.crt
  --service-account-private-key-file=/srv/kubernetes/server.key

3. Clear your old default secrets, they all be automatically regenerated:

$ kubectl get secrets  --all-namespaces
NAMESPACE     NAME                  TYPE                                  DATA
default       default-token-6oahq   kubernetes.io/service-account-token   2
kube-system   default-token-ur28n   kubernetes.io/service-account-token   2

$ kubectl delete secret --namespace=default default-token-6oahq
$ kubectl delete secret --namespace=kube-system  default-token-ur28n

Note that everything I described above is automated by the cluster/kube-up.sh bootstrap script.

[antoineco]

Is the CA certificate in --root-ca-file= authenticated by the key in --service-account-key-file=?

• First a CA is created, the result is a cert/key pair (ca.crt/ca.key). You can use easyrsa to generate your PKI or OpenSSL if you feel comfortable with it.
• Then a certificate is requested and signed using this CA (server.cert/server.key), it will be used
  1. by the api server to enable HTTPS and verify service account tokens
  2. by the controller manager to sign service account tokens, so that pods can authenticate against the API using these tokens
• Another certificate is requested and signed (kubecfg.crt/kubecfg.key) using the same CA, you can use it to authenticate your clients for example.

If so, I cannot create the --service-account-key-file= on the host automatically via a systemd.service unit, unless I also automatically create the --root-ca-file= via a systemd.service, correct?

By using the script mentioned in the previous comment, you can easily generate everything in one shot. Our unit looks like this:

[Unit]
Description=Generate Kubernetes API Server certificates
ConditionPathExists=!/srv/kubernetes/.certs.lock
Requires=network-online.target
After=network-online.target

[Service]
ExecStartPre=-/usr/sbin/groupadd -r kube-cert
ExecStartPre=/usr/bin/wget -q -N -P /opt/bin https://raw.githubusercontent.com/GoogleCloudPlatform/kubernetes/v0.21.1/cluster/saltbase/salt/generate-cert/make-ca-cert.sh
ExecStartPre=/usr/bin/chmod u=rwx,go= /opt/bin/make-ca-cert.sh
ExecStart=/opt/bin/make-ca-cert.sh _use_aws_external_ip_ IP:10.10.0.1,DNS:kubernetes,DNS:kubernetes.default,DNS:kubernetes.default.svc,DNS:kubernetes.default.svc.freeletics.local
Type=oneshot
RemainAfterExit=true

Then make your apiserver and controller manager units depend on it. Example:

[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
Wants=generate-k8s-certs.service
After=generate-k8s-certs.service

[...]

Is the missing ca.crt secret in my default serviceaccount the same as the one I would hand to the kube-controller-manager via --root-ca-file=?

Yes

Will I have to distribute these files to the client I run kubectl on?

Use the kubercfg.crt/key pair for that purpose, or generate one pair per client to make it easier to revoke. In your PKI folder:

./easyrsa build-client-full kubectl-username nopass

Will the kube-controller-manager and/or the kubelets automatically create certificates for themselves signed by --root-ca-file=?

No, you have to find a way to distribute them. Since you're running on CoreOS, why don't you use ectd? Your certs are nothing but plain text files after all.

You can take a look at what's inside the cluster/ folder: on GCE the ca cert and server cert/key pair are distributed via salt, while other providers like AWS still use good old tokens but this is being worked out by the Kubernetes team if I'm not mistaken.

If so, how do I control which host is allowed to get such a certificate, and which will not be allowed to join the cluster?

I think you have to figure it out for yourself, this is outside the Kubernetes topic. Don't distribute certs to all your servers. Keep control over your server fleet. etc.

Personally we distribute certs to our CoreOS boxes using a central storage (S3 bucket on AWS) and leverage IAM roles (again some AWS mechanisms) to decide which instance can access what. Just a suggestion.

How is the certificate for connection with a client kubectl created? Do I need to also automatically create a client certificate using a systemd.service and then fetch the file using e.g. ssh?

Already answered. Use a CA to generate and sign certificates. Systemd might not be appropriate for that, unless you know in advance which certs you want to generate, or use a single client cert that you will use on every client.

If so, I assume the automatic generation of the key as laid out in the examples is not actually recommended, as it is utterly cumbersome?

Something else you have to figure out for yourself, only you can decide what you do with your CA.

I read that on GCE you use Salt to distribute the (manually created) root certificates. Is there any convenient way to do this on a bare-metal CoreOS cluster? Writing a properly indented write_files directive for cloud-config appears unflexible and cumbersome on its own.

write_files is ugly :) But works fine for testing. If you can store certs somewhere you can easily write a script which puts the content of the certs inside a kubeconfig file. I mentioned it before but here is a great example

Is all authentication between the Kubernetes master (kube-apiserver, kube-controller-manager, kube-scheduler) and the kubelets and kubectls handled via HTTPS and client certificates?

Yes, certificate or token (hint: use certs if possible)

[WIZARD-CXY]

good stuff

[devurandom]

Ok, let's assume I have setup CA and client certificates on a separate machine and now want to place them onto the appropriate nodes in my CoreOS cluster.

No, you have to find a way to distribute them. Since you're running on CoreOS, why don't you use ectd? Your certs are nothing but plain text files after all.

I thought about this, too. But etcd has no concept of authorisation (according to their website "Access control lists (ACLs) will be added to etcd in the near future."), so for security reasons I cannot place the certificate keys into etcd and then make each node fetch its own. It would work to distribute the CA cert, though.

Personally we distribute certs to our CoreOS boxes using a central storage (S3 bucket on AWS) and leverage IAM roles (again some AWS mechanisms) to decide which instance can access what. Just a suggestion.

I thought about writing global Fleet services to download each node's certs and keys using rsync (mainly because that is easily available on the CoreOS host), which I would secure using the hosts allow rsyncd.conf directive on the rsync host.

If you can store certs somewhere you can easily write a script which puts the content of the certs inside a kubeconfig file.

From the kubeconfig docs it seems that kubeconfig is for client-side configuration only. It does not replace write_files in the cloud-config, as you suggested…?

[antoineco]

A kubeconfig file should be used with every "client" as long as authentication is involved, this includes kubelet and kube-proxy. So you'll somehow have to fetch/generate these config files on your nodes, unless you disabled SSL within your cluster.

[antoineco]

@devurandom I though this might interest you https://coreos.com/blog/introducing-etcd-2.1/

[devurandom]

Thanks! Now it should be easy to distribute the certs/keys using etcd and confd.

[fzu-huang]

@devurandom I run ./make-ca-cert.sh script but nothing happened.. /srv/kubernetes/ is a empty directory. do you know why

[dalanlan]

@fzu-huang
I believe It's a network issue(GFW). You can run this to confirm

curl -L -O https://storage.googleapis.com/kubernetes-release/easy-rsa/easy-rsa.tar.gz 

[WIZARD-CXY]

better run without > /dev/null 2>&1 , just let the stdout and stderr print for your debug

[fzu-huang]

@dalanlan
you are right : (
Is there any other way to create crt and key?

[dalanlan]

Not have been there.
Finding your own way to access that might be an easy solution. VPN spreads anywhere after all.

[Icedroid]

ubuntu 14.04 can't not generate cert:

writing new private key to '/tmp/kubernetes_cacert.Q23Fzv/easy-rsa-master/easyrsa3/pki/private/ca.key'
-----
Error Loading request extension section req_extra
140657255773856:error:22075075:X509 V3 routines:v2i_GENERAL_NAME_ex:unsupported option:v3_alt.c:557:name=ip
140657255773856:error:22098080:X509 V3 routines:X509V3_EXT_nconf:error in extension:v3_conf.c:93:name=subjectAltName, value=IP:192.168.1.12,ip:192.168.1.12,IP:10.0.0.1,DNS:kubernetes,DNS:kubernetes.default,DNS:kubernetes.default.svc,DNS:kubernetes.default.svc.cluster.local

Easy-RSA error:

Failed to generate request

[devurandom]

@Icedroid You need to use current master of OpenVPN/easy-rsa. v3.0.0-rc2 contains a bug which makes it impossible to specify subject-alternate-names.

[antoineco]

You need to write IP in capital letters.

[mbforbes]

+cc @roberthbailey

@devurandom would you be willing to send a PR to help update our docs for this?

[Icedroid]

@antoineco I already used IP capital:
sudo ./make-ca-cert.sh 192.168.1.12 IP:192.168.1.12,IP:10.0.0.1,DNS:kubernetes,DNS:kubernetes.default,DNS:kubernetes.default.svc,DNS:kubernetes.default.svc.cluster.local

After I run sudo apt-get update && sudo apt-get -y upgrade, the Easy-RSA error gone.
So I think @devurandom is right

[devurandom]

@mbforbes Very happily, but first I need to get it fully running myself. At the moment my installation still seems to be a bit fragile.

[devurandom]

@mbforbes:

1. Is it correct that the stock dns addon will not work in an authenticated k8s environment? I got lots of "bad certificate" messages in the kube-apiserver logs (apparently from kube2sky), until I provided the container with an own kubeconfig file with a proper certificate.
2. The from scratch guide says about using "Different credentials for every kubelet, etc." that "We are working on this but all the pieces are not ready yet.". What exactly is missing? As you use a CA architecture, I would assume that it just works. When I juggled the certificates in the wrong way, and thus had them intermixed, I didn't notice trouble related to this either...

[roberthbailey]

/cc @mikedanese (for your first question)

For the second question, it should work with different credentials per kubelet, but we don't do this today using any of the automated cluster creation scripts. The part that isn't ready yet is the authorization framework in the apiserver that can differentiate clients from each other. Right now having any valid credentials gives you complete access to the cluster so there isn't a lot of value in having different credentials per kubelet.

[mikedanese]

In answer to one:

By default, kube2sky will use the inClusterConfig system to configure the secure connection. This requires kubernetes version v1.0.3 or higher and a couple extra kubernetes configuration steps. Firstly, for your apiserver you must create a SSL certificate pair with a SAN that includes the ClusterIP of the kubernetes service. Look here for an example of how to properly generate certs. Secondly, you need to pass the ca.crt that you generated to the --root-ca-file option of the controller-manager. This will distribute the root CA to /var/run/secrets/kubernetes.io/serviceaccount/ of all pods. If you are using ABAC authorization (as opposed to AllowAll which is the default), you will also need to give the system:serviceaccount:<namespace-of-kube2sky(likely kube-system)>:default readonly access to the cluster (look here for more info).

You can also run kube2sky over http if you have an insecure cluster setup.

[devurandom]

@mikedanese An addition to your instructions: If one had ever used kube-controller-manager without the --root-ca-file option, one has to clear all secrets listed by kubectl --namespace={default,kube-system} get secrets. They will be immediately recreated in their proper form.

[jmccarty3]

I think one thing that should be documented better is that without the --admission-control=ServiceAccount being added to the api server, the service token will not be mounted despite it being generated. This was a pretty large source of confusion for me setting up my own cluster without the set up scripts.

[jonmoter]

This is obviously an old issue, but adding some comments about our own experience.

I'm building out a highly available Kubernetes cluster in both AWS and on bare metal servers. Thus, multiple API servers. We're using Hashicorp Vault and their PKI backend to generate client and server certs for the different components.

The way we had it configured, each API host had its own server cert and private key, but they were all signed by the same CA. Then we created client certs signed by the same CA, and distributed those to the Node hosts, and other clients that wanted to connect to the hosts via kubectl. So far so good.

However, when running something like SkyDNS, it was failing to authenticate with the API servers using a ServiceAccount. After some debugging, I think the issue is that since the private keys are different on each API host, one of them is generating the ServiceAccount tokens that are put into Secrets. But if a request goes to the other API host, it fails to validate the secret.

So the solution seems to be that the public/private keypair you use for generating and validating the ServiceAccount tokens need to be the same across the API hosts. (Please correct me if I'm wrong about this.)

This makes sense now that we've figured this out, but it took a lot of head-desking to figure out what was going on here.

[erictune]

What you said makes sense. I hope your head feels better!

[bkleef]

@jonmoter issue could also be ECDSA signature algorithm over supported RSA (#28180 and #9927)

[gvenka008c]

I have 3 masters running. How do i generate certs for 3 masters?

I tried on one of the master and copied the files to other 2 servers, but kube-controller-manager and kube-apiserver failed to startup

./make-ca-cert.sh master1_ip IP:master2_ip ,IP:master3_ip,DNS:kubernetes,DNS:kubernetes.default,DNS:kubernetes.default.svc,DNS:kubernetes.default.svc.cluster.local

/etc/kubernenets/apiserver config

# Add your own!
KUBE_API_ARGS="--client-ca-file=/srv/kubernetes/ca.crt --tls-cert-file=/srv/kubernetes/server.cert --tls-private-key-file=/srv/kubernetes/server.key"

/etc/kubernetes/controller-manager

#KUBE_CONTROLLER_MANAGER_ARGS="--root-ca-file=/srv/kubernetes/ca.crt --service-account-private-key-file=/srv/kubernetes/server.key"

/srv/kubernetes files
 kubernetes]# ls -ltr
total 28
-rw-rw----. 1 root root 1216 Mar 21 15:12 ca.crt
-rw-rw----. 1 root root 1704 Mar 21 15:12 server.key
-rw-rw----. 1 root root 4870 Mar 21 15:12 server.cert
-rw-------. 1 root root 1704 Mar 21 15:12 kubecfg.key
-rw-------. 1 root root 4466 Mar 21 15:12 kubecfg.crt

[antoineco]

@gvenka008c that's a question for Stackoverflow. Make sure you reference the certificate and not only the key. Add logs showing the error if this doesn't fix it (on Stackoverflow please)

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

[hanikesn]

/remove-lifecycle stale
TBH I think official documentation in that regard is still lacking.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[hanikesn]

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.