How to configure my own CA for k3s ?

[parekhha]

Hi,
I am trying to configure my own ca with k3s. My expectation is, I give my cacert and cakey to k3s and k3s automatically generate all required cert from that and also rotate those certificate if expired.

Is it possible ?

[brandond]

Can you discus your use case a bit? Other than it being fun to experiment with, I'm not sure what the point would be. As far as I know, even the commercial cloud providers just use unique self-signed CAs for each cluster.

k3s actually follows best practices and uses three separate CAs. There aren't any specific arguments or options to load existing CA certs, but you can pre-create the files and put them in the correct location and k3s will use them instead of creating new ones.

Here's an example:

mkdir -p /var/lib/rancher/k3s/server/tls
cd /var/lib/rancher/k3s/server/tls
openssl genrsa -out client-ca.key 2048
openssl genrsa -out server-ca.key 2048
openssl genrsa -out request-header-ca.key 2048
openssl req -x509 -new -nodes -key client-ca.key -sha256 -days 3560 -out client-ca.crt -addext keyUsage=critical,digitalSignature,keyEncipherment,keyCertSign -subj '/CN=k3s-client-ca'
openssl req -x509 -new -nodes -key server-ca.key -sha256 -days 3560 -out server-ca.crt -addext keyUsage=critical,digitalSignature,keyEncipherment,keyCertSign -subj '/CN=k3s-server-ca'
openssl req -x509 -new -nodes -key request-header-ca.key -sha256 -days 3560 -out request-header-ca.crt -addext keyUsage=critical,digitalSignature,keyEncipherment,keyCertSign -subj '/CN=k3s-request-header-ca'

After doing this you can install k3s as usual using install.sh and it should use the openssl-generated certs and keys.

[parekhha]

@brandond
Thanks. It worked. But when I rotate these ca certs all certificates are getting re-generated except dynamic-cert.json. Due to this, I am getting unknown cert error while using kubectl. Steps I followed

1. Stop K3s service
2. Remove all certs from /var/lib/rancher/k3s/server/tls
3. Recreate all three CAs.
4. Start k3s service.

I see all new certs are signed by new ca but dynamic-cert.json is having old certificate. Is it bug or I am missing something ?

[brandond]

The k3s-serving cert isn't handled the same as others. It doesn't get rotated when it expires, either - there's an open issue about it. I'm not sure what the best way is to get it updated on an existing cluster. Maybe try deleting it and the linked secret?

[parekhha]

@brandond
Yeah, Following steps worked

1. First delete secret and stop k3s before it recreate secret
2. Remove all certs from /var/lib/rancher/k3s/server/tls
3. Recreate all three CAs.
4. Start k3s service.
   Thanks !!!

[vntbbb]

@brandond
Yeah, Following steps worked

1. First delete secret and stop k3s before it recreate secret
2. Remove all certs from /var/lib/rancher/k3s/server/tls
3. Recreate all three CAs.
4. Start k3s service.
   Thanks !!!

Hi, parekhha, I do like you,but get error.
my shell:
#!/usr/bin/bash

kubectl -n kube-system delete secret k3s-serving && sleep 2
systemctl stop k3s && sleep 4

#default openssl.cnf: /etc/pki/tls/openssl.cnf
cd tls && rm ./* -rf
openssl genrsa -out client-ca.key 2048
openssl genrsa -out server-ca.key 2048
openssl genrsa -out request-header-ca.key 2048
#echo "gen client-ca.crt"
#openssl req -x509 -new -nodes -key client-ca.key -sha256 -days 3560 -out client-ca.crt -extensions v3_req -subj '/CN=k3s-client-ca'
#echo "gen server-ca.crt"
#openssl req -x509 -new -nodes -key server-ca.key -sha256 -days 3560 -out server-ca.crt -extensions v3_req -subj '/CN=k4s-server-ca'
#echo "gen request-header-ca.crt"
#openssl req -x509 -new -nodes -key request-header-ca.key -sha256 -days 3560 -out request-header-ca.crt -extensions v3_req -subj '/CN=k3s-request-header-ca'

echo "restart k3s ..."
systemctl start k3s

echo "over"
then get errors:
[root@localhost]# systemctl status k3s
● k3s.service - Lightweight Kubernetes
Loaded: loaded (/etc/systemd/system/k3s.service; enabled; vendor preset: disabled)
Active: active (running) since Fri 2020-08-14 05:22:01 EDT; 6min ago
...
Aug 14 05:28:00 localhost k3s[60167]: W0814 05:28:00.703863 60167 handler_proxy.go:97] no RequestInfo found in the context
Aug 14 05:28:00 localhost k3s[60167]: E0814 05:28:00.703923 60167 controller.go:114] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: faile...navailable
Aug 14 05:28:00 localhost k3s[60167]: , Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
Aug 14 05:28:00 localhost k3s[60167]: I0814 05:28:00.703932 60167 controller.go:127] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.i...d Requeue.
Aug 14 05:28:00 localhost k3s[60167]: I0814 05:28:00.743580 60167 log.go:172] http: TLS handshake error from 10.42.1.11:52766: remote error: tls: bad certificate
Aug 14 05:28:01 localhost k3s[60167]: I0814 05:28:01.638100 60167 log.go:172] http: TLS handshake error from 10.42.1.11:52770: remote error: tls: bad certificate
Aug 14 05:28:01 localhost k3s[60167]: I0814 05:28:01.643213 60167 log.go:172] http: TLS handshake error from 10.42.1.11:52772: remote error: tls: bad certificate
Aug 14 05:28:01 localhost k3s[60167]: I0814 05:28:01.644060 60167 log.go:172] http: TLS handshake error from 10.42.1.9:33172: remote error: tls: bad certificate
Aug 14 05:28:01 localhost k3s[60167]: I0814 05:28:01.646780 60167 log.go:172] http: TLS handshake error from 10.42.1.9:33174: remote error: tls: bad certificate
Aug 14 05:28:01 localhost k3s[60167]: I0814 05:28:01.744863 60167 log.go:172] http: TLS handshake error from 10.42.1.11:52780: remote error: tls: bad certificate
Hint: Some lines were ellipsized, use -l to show in full.

[brandond]

It sounds like you have some pods still running with old certificates. If you're going to try to use custom CA certs, you're best off doing it before starting k3s for the first time.

[vntbbb]

It sounds like you have some pods still running with old certificates. If you're going to try to use custom CA certs, you're best off doing it before starting k3s for the first time.

Yeah, I create my production cluster by k3s install.sh. Instead of restarting our cluster once a year, we want to generate a long time certificates(e.g., a 5-year serving-kube-apiserver.crt). Is there any tool to generate new certs and apply them to my cluster?

thanks.

[brandond]

There is not currently any automated way to do this. Most folks would tell you that you should probably be rebooting to apply kernel and package updates at least once a year. For k3s at least, restarting the service does not affect running pods, so you could just schedule a weekly restart of k3s and not worry about it.

[vntbbb]

There is not currently any automated way to do this. Most folks would tell you that you should probably be rebooting to apply kernel and package updates at least once a year. For k3s at least, restarting the service does not affect running pods, so you could just schedule a weekly restart of k3s and not worry about it.

ok, thank you!

[ericis]

Possibly related k3d discussion: k3d-io/k3d#687

[TonyShanc]

Hi, I use vntbbb 's script above to delete all certs after deleting secret and restart k3s, then recreate three CA certs and start k3s but old CA certs coverd my own CA certs. Is there any update has changed certs creation logic?
os: unbuntu 20.04
k3s version: v1.23.9+k3s1 (f45cf32)

[ShylajaDevadiga]

Re-opening for validation

[ShylajaDevadiga]

Validated on commit id 2156015 from master branch

• Created a 3 node cluster with etcd backend
• Validated user-provided CA certificate using certs.sh from the PR

$ curl -fL https://get.k3s.io | INSTALL_K3S_COMMIT=21560155217bdfc12c5105d7cb79fad61b2985aa INSTALL_K3S_SKIP_START=true sh -s - server --token <TOKEN> --cluster-init
$ wget https://raw.githubusercontent.com/brandond/k3s/00a76d82a2d64bc18f589bd9a3bec371c634ee6b/contrib/util/certs.sh


$ chmod u+x certs.sh 
$ sudo ./certs.sh 
$ sudo systemctl enable k3s
$ sudo systemctl start k3s

$ kubectl get nodes
NAME              STATUS   ROLES                       AGE    VERSION
ip-172-31-5-188   Ready    control-plane,etcd,master   15m    v1.26.1+k3s-21560155
ip-172-31-6-21    Ready    control-plane,etcd,master   3m6s   v1.26.1+k3s-21560155
ip-172-31-7-172   Ready    control-plane,etcd,master   12m    v1.26.1+k3s-21560155

$ kubectl get pods -A
NAMESPACE     NAME                                      READY   STATUS      RESTARTS   AGE
kube-system   coredns-5c6b6c5476-gqpd2                  1/1     Running     0          15m
kube-system   helm-install-traefik-crd-nwc9l            0/1     Completed   0          15m
kube-system   helm-install-traefik-lpstb                0/1     Completed   1          15m
kube-system   local-path-provisioner-5d56847996-649z2   1/1     Running     0          15m
kube-system   metrics-server-7b67f64457-m8x9l           1/1     Running     0          15m
kube-system   svclb-traefik-016ce390-22c9c              2/2     Running     0          12m
kube-system   svclb-traefik-016ce390-5px5h              2/2     Running     0          15m
kube-system   svclb-traefik-016ce390-djj42              2/2     Running     0          3m8s
kube-system   traefik-56b8c5fb5c-5n7zc                  1/1     Running     0          15m

[dene14]

Just to save other people's time posting here full script that will generate both server side (3 CAs for k8s) and client side (kubectl, argocd, etc) certificates:

openssl genrsa -out client-ca.key 2048
openssl genrsa -out server-ca.key 2048
openssl genrsa -out request-header-ca.key 2048
openssl req -x509 -new -nodes -key client-ca.key -sha256 -days 3651 -out client-ca.crt -addext keyUsage=critical,digitalSignature,keyEncipherment,keyCertSign -subj '/CN=k3s-client-ca'
openssl req -x509 -new -nodes -key server-ca.key -sha256 -days 3651 -out server-ca.crt -addext keyUsage=critical,digitalSignature,keyEncipherment,keyCertSign -subj '/CN=k3s-server-ca'
openssl req -x509 -new -nodes -key request-header-ca.key -sha256 -days 3651 -out request-header-ca.crt -addext keyUsage=critical,digitalSignature,keyEncipherment,keyCertSign -subj '/CN=k3s-request-header-ca'

# Generating client key/cert
openssl req -newkey rsa:2048 -nodes -keyout client.key -out client.csr -subj "/CN=system:admin/O=system:masters" 
openssl x509 -req -in client.csr -CA client-ca.crt -CAkey client-ca.key -CAcreateserial -out client.crt -days 3650 -extensions v3_ext

echo """
# These keychains to be used on K8S control plane (3 independent CAs)
serverTlsSecret:
  clientCaCrt: `cat client-ca.crt | base64 -w0`
  clientCaKey: `cat client-ca.key | base64 -w0`
  requestHeaderCaCrt: `cat request-header-ca.crt | base64 -w0`
  requestHeaderCaKey: `cat request-header-ca.key | base64 -w0`
  serverCaCrt: `cat server-ca.crt | base64 -w0`
  serverCaKey: `cat server-ca.key | base64 -w0`

"""

echo """
# These keychains to be used on clients like kubectl or argocd
clientTlsSecret:
  certificate-authority-data: `cat server-ca.crt | base64 -w0`
  client-certificate-data: `cat client.crt client-ca.crt | base64 -w0`
  client-key-data: `cat client.key | base64 -w0`
"""
rm -f client-ca.key server-ca.key request-header-ca.key client-ca.crt server-ca.crt request-header-ca.crt client.csr client.key client.crt

[brandond]

@dene14 there is actually a page for this in the docs, along with a script to do it for you.

https://docs.k3s.io/cli/certificate#using-custom-ca-certificates