Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[release-1.24] Support rotating the default self-signed certs #7083

Closed
brandond opened this issue Mar 13, 2023 · 2 comments
Closed

[release-1.24] Support rotating the default self-signed certs #7083

brandond opened this issue Mar 13, 2023 · 2 comments
Assignees
@brandond @ShylajaDevadiga
Milestone
v1.24.12+k3s1

Comments

@brandond
Copy link
Contributor

@brandond brandond modified the milestones: v1.24.13+k3s1, v1.24.12+k3s1 Mar 13, 2023
@brandond brandond self-assigned this Mar 13, 2023
@brandond brandond mentioned this issue Mar 13, 2023
Merged
@ShylajaDevadiga
Copy link
Contributor

Validated as per docs on release-1.24 branch using commit id 7fee87d

1. Rotate client and server certificates manually, using k3s certificate rotate subcommand:

# Stop K3s
systemctl stop k3s
# Rotate certificates
k3s certificate rotate
# Start K3s
systemctl start k3s

Validated crt is rotated and backed up in /var/lib/rancher/k3s/server/tls-xxx dir
Validated permission on crt and key files to be 644 and 600 or more restrictive respectively

$ sudo find /var/lib/rancher/k3s/server/tls -name "*.key" | sudo xargs stat -c '%n %a'
/var/lib/rancher/k3s/server/tls/service.current.key 600
/var/lib/rancher/k3s/server/tls/client-k3s-controller.key 600
/var/lib/rancher/k3s/server/tls/serving-kubelet.key 600
/var/lib/rancher/k3s/server/tls/client-scheduler.key 600
/var/lib/rancher/k3s/server/tls/client-kubelet.key 600
/var/lib/rancher/k3s/server/tls/temporary-certs/apiserver-loopback-client__.key 644
/var/lib/rancher/k3s/server/tls/service.key 600
/var/lib/rancher/k3s/server/tls/client-ca.key 600
/var/lib/rancher/k3s/server/tls/client-controller.key 600
/var/lib/rancher/k3s/server/tls/client-admin.key 600
/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key 600
/var/lib/rancher/k3s/server/tls/client-k3s-cloud-controller.key 600
/var/lib/rancher/k3s/server/tls/request-header-ca.key 600
/var/lib/rancher/k3s/server/tls/client-kube-proxy.key 600
/var/lib/rancher/k3s/server/tls/server-ca.key 600
/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key 600
/var/lib/rancher/k3s/server/tls/etcd/peer-server-client.key 600
/var/lib/rancher/k3s/server/tls/etcd/client.key 600
/var/lib/rancher/k3s/server/tls/etcd/peer-ca.key 600
/var/lib/rancher/k3s/server/tls/etcd/server-ca.key 600
/var/lib/rancher/k3s/server/tls/etcd/server-client.key 600
/var/lib/rancher/k3s/server/tls/client-auth-proxy.key 600

$ sudo find /var/lib/rancher/k3s/server/tls -name "*.crt" | sudo xargs stat -c '%n %a'
/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt 644
/var/lib/rancher/k3s/server/tls/client-k3s-cloud-controller.crt 644
/var/lib/rancher/k3s/server/tls/client-controller.crt 644
/var/lib/rancher/k3s/server/tls/temporary-certs/apiserver-loopback-client__.crt 644
/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt 644
/var/lib/rancher/k3s/server/tls/client-admin.crt 644
/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt 644
/var/lib/rancher/k3s/server/tls/client-scheduler.crt 644
/var/lib/rancher/k3s/server/tls/request-header-ca.crt 644
/var/lib/rancher/k3s/server/tls/client-kube-proxy.crt 644
/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt 644
/var/lib/rancher/k3s/server/tls/client-k3s-controller.crt 644
/var/lib/rancher/k3s/server/tls/client-ca.crt 644
/var/lib/rancher/k3s/server/tls/etcd/peer-ca.crt 644
/var/lib/rancher/k3s/server/tls/etcd/client.crt 644
/var/lib/rancher/k3s/server/tls/etcd/peer-server-client.crt 644
/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt 644
/var/lib/rancher/k3s/server/tls/etcd/server-client.crt 644
/var/lib/rancher/k3s/server/tls/server-ca.crt 644
/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt 644

2. Rotate Individual certificate

$ sudo systemctl stop k3s
$ sudo k3s certificate rotate --service api-server
INFO[0000] Rotating certificates for api-server service 
INFO[0000] Successfully backed up certificates for all services to path /var/lib/rancher/k3s/server/tls-1678915558, please restart k3s server or agent to rotate certificates 
$ sudo systemctl start k3s

$ sudo ls -l /var/lib/rancher/k3s/server/tls|grep api
-rw-r--r-- 1 root root 1181 Mar 15 21:26 client-kube-apiserver.crt
-rw------- 1 root root  227 Mar 15 21:26 client-kube-apiserver.key
-rw-r--r-- 1 root root 1376 Mar 15 21:26 serving-kube-apiserver.crt
-rw------- 1 root root  227 Mar 15 21:26 serving-kube-apiserver.key

3. Using Custom CA Certificates

# mkdir -p /var/lib/rancher/k3s/server/tls
# curl -sL https://github.com/k3s-io/k3s/raw/master/contrib/util/generate-custom-ca-certs.sh | bash -
# curl -fL https://get.k3s.io | INSTALL_K3S_COMMIT=7fee87d9765d01f1e478b06553b0b100410de289 sh -s - server --cluster-init

# kubectl get nodes
NAME              STATUS   ROLES                       AGE   VERSION
ip-172-31-6-40    Ready    control-plane,etcd,master   14m   v1.24.11+k3s-7fee87d9
ip-172-31-8-113   Ready    <none>                      11m   v1.24.11+k3s-7fee87d9

# kubectl get pods -A
NAMESPACE     NAME                                      READY   STATUS      RESTARTS   AGE
kube-system   coredns-7b5bbc6644-bp8xb                  1/1     Running     0          14m
kube-system   helm-install-traefik-crd-xxgpb            0/1     Completed   0          14m
kube-system   helm-install-traefik-jbclx                0/1     Completed   1          14m
kube-system   local-path-provisioner-687d6d7765-27zkc   1/1     Running     0          14m
kube-system   metrics-server-667586758d-jfzw9           1/1     Running     0          14m
kube-system   svclb-traefik-53419730-2rgmc              2/2     Running     0          14m
kube-system   svclb-traefik-53419730-2vcs4              2/2     Running     0          11m
kube-system   traefik-64b96ccbcd-bg779                  1/1     Running     0          14m

@ShylajaDevadiga
Copy link
Contributor

4. Rotating Custom CA Certificates

  • 3 server 1 agent cluster that has been started with custom CA certificates.
  • Rotate the CA certificates and keys non-disruptively, using same root CA
mkdir -p /var/lib/rancher/k3s/server/tls
curl -sL https://github.com/k3s-io/k3s/raw/master/contrib/util/generate-custom-ca-certs.sh | bash -
curl -fL https://get.k3s.io/ | INSTALL_K3S_COMMIT=7fee87d9765d01f1e478b06553b0b100410de289 sh -s - server --cluster-init
sudo mkdir -p /opt/k3s/server/tls
cp /var/lib/rancher/k3s/server/tls/root-ca.pem /opt/k3s/server/tls/
cp /var/lib/rancher/k3s/server/tls/intermediate-ca.pem /opt/k3s/server/tls/
cp /var/lib/rancher/k3s/server/tls/intermediate-ca.key /opt/k3s/server/tls/
cp /var/lib/rancher/k3s/server/tls/service.key /opt/k3s/server/tls/
curl -sL https://github.com/k3s-io/k3s/raw/master/contrib/util/generate-custom-ca-certs.sh | DATA_DIR=/opt/k3s bash
k3s certificate rotate-ca --path=/opt/k3s/server

Restarted k3s on all nodes, first servers and then agent

$ kubectl get nodes
NAME               STATUS   ROLES                       AGE    VERSION
ip-172-31-14-213   Ready    <none>                      168m   v1.24.11+k3s-7fee87d9
ip-172-31-14-24    Ready    control-plane,etcd,master   175m   v1.24.11+k3s-7fee87d9
ip-172-31-2-199    Ready    control-plane,etcd,master   174m   v1.24.11+k3s-7fee87d9
ip-172-31-6-16     Ready    control-plane,etcd,master   177m   v1.24.11+k3s-7fee87d9
$ kubectl get pods -A
NAMESPACE     NAME                                      READY   STATUS      RESTARTS   AGE
kube-system   coredns-7b5bbc6644-vnprp                  1/1     Running     0          176m
kube-system   helm-install-traefik-crd-h8w7l            0/1     Completed   0          176m
kube-system   helm-install-traefik-f84pf                0/1     Completed   1          176m
kube-system   local-path-provisioner-687d6d7765-td4hk   1/1     Running     0          176m
kube-system   metrics-server-667586758d-86c6v           1/1     Running     0          176m
kube-system   svclb-traefik-a49ce526-8cbk7              2/2     Running     0          175m
kube-system   svclb-traefik-a49ce526-kw5sk              2/2     Running     0          176m
kube-system   svclb-traefik-a49ce526-mcd4z              2/2     Running     0          174m
kube-system   svclb-traefik-a49ce526-zppmf              2/2     Running     0          168m
kube-system   traefik-64b96ccbcd-b2rsg                  1/1     Running     0          176m

5. Rotating Self-Signed CA Certificates

  • Install k3s on all nodes.
curl -fL https://get.k3s.io | INSTALL_K3S_COMMIT=7fee87d9765d01f1e478b06553b0b100410de289 sh -s - server --cluster-init --token <TOKEN>
  • Run the script to generate updated self-signed certificates
curl -sL https://github.com/k3s-io/k3s/raw/master/contrib/util/rotate-default-ca-certs.sh | bash -
$ sudo ls -l /var/lib/rancher/k3s/server/tls
total 136
-rw-r--r-- 1 root root 1169 Mar 21 20:26 client-admin.crt
-rw------- 1 root root  227 Mar 21 20:26 client-admin.key
-rw-r--r-- 1 root root 1178 Mar 21 20:26 client-auth-proxy.crt
-rw------- 1 root root  227 Mar 21 20:26 client-auth-proxy.key
-rw-r--r-- 1 root root  566 Mar 21 20:26 client-ca.crt
-rw------- 1 root root  227 Mar 21 20:26 client-ca.key
-rw-r--r-- 1 root root  566 Mar 21 20:26 client-ca.nochain.crt
-rw-r--r-- 1 root root 1161 Mar 21 20:26 client-controller.crt
-rw------- 1 root root  227 Mar 21 20:26 client-controller.key
-rw-r--r-- 1 root root 1157 Mar 21 20:26 client-k3s-cloud-controller.crt
-rw------- 1 root root  227 Mar 21 20:26 client-k3s-cloud-controller.key
-rw-r--r-- 1 root root 1149 Mar 21 20:26 client-k3s-controller.crt
-rw------- 1 root root  227 Mar 21 20:26 client-k3s-controller.key
-rw-r--r-- 1 root root 1173 Mar 21 20:26 client-kube-apiserver.crt
-rw------- 1 root root  227 Mar 21 20:26 client-kube-apiserver.key
-rw-r--r-- 1 root root 1145 Mar 21 20:26 client-kube-proxy.crt
-rw------- 1 root root  227 Mar 21 20:26 client-kube-proxy.key
-rw------- 1 root root  227 Mar 21 20:26 client-kubelet.key
-rw-r--r-- 1 root root 1149 Mar 21 20:26 client-scheduler.crt
-rw------- 1 root root  227 Mar 21 20:26 client-scheduler.key
-rw-r--r-- 1 root root 4773 Mar 21 20:27 dynamic-cert.json
drwxr-xr-x 2 root root 4096 Mar 21 20:26 etcd
-rw-r--r-- 1 root root  591 Mar 21 20:26 request-header-ca.crt
-rw------- 1 root root  227 Mar 21 20:26 request-header-ca.key
-rw-r--r-- 1 root root  570 Mar 21 20:26 server-ca.crt
-rw------- 1 root root  227 Mar 21 20:26 server-ca.key
-rw-r--r-- 1 root root  570 Mar 21 20:26 server-ca.nochain.crt
-rw------- 1 root root 1679 Mar 21 20:26 service.current.key
-rw------- 1 root root 1679 Mar 21 20:26 service.key
-rw-r--r-- 1 root root 1376 Mar 21 20:26 serving-kube-apiserver.crt
-rw------- 1 root root  227 Mar 21 20:26 serving-kube-apiserver.key
-rw------- 1 root root  227 Mar 21 20:26 serving-kubelet.key
drwx------ 2 root root 4096 Mar 21 20:26 temporary-certs
  • Load the updated certs into the datastore
k3s certificate rotate-ca --path=/var/lib/rancher/k3s/server/rotate-ca
  • Restart k3s on all nodes
sudo systemctl restart k3s
$ sudo ls -l /var/lib/rancher/k3s/server/tls
total 136
-rw-r--r-- 1 root root 3075 Mar 21 20:34 client-admin.crt
-rw------- 1 root root  227 Mar 21 20:34 client-admin.key
-rw-r--r-- 1 root root 3146 Mar 21 20:34 client-auth-proxy.crt
-rw------- 1 root root  227 Mar 21 20:34 client-auth-proxy.key
-rw-r--r-- 1 root root 2468 Mar 21 20:30 client-ca.crt
-rw------- 1 root root  454 Mar 21 20:30 client-ca.key
-rw-r--r-- 1 root root  627 Mar 21 20:34 client-ca.nochain.crt
-rw-r--r-- 1 root root 3063 Mar 21 20:34 client-controller.crt
-rw------- 1 root root  227 Mar 21 20:34 client-controller.key
-rw-r--r-- 1 root root 3059 Mar 21 20:34 client-k3s-cloud-controller.crt
-rw------- 1 root root  227 Mar 21 20:34 client-k3s-cloud-controller.key
-rw-r--r-- 1 root root 3051 Mar 21 20:34 client-k3s-controller.crt
-rw------- 1 root root  227 Mar 21 20:34 client-k3s-controller.key
-rw-r--r-- 1 root root 3079 Mar 21 20:34 client-kube-apiserver.crt
-rw------- 1 root root  227 Mar 21 20:34 client-kube-apiserver.key
-rw-r--r-- 1 root root 3042 Mar 21 20:34 client-kube-proxy.crt
-rw------- 1 root root  227 Mar 21 20:34 client-kube-proxy.key
-rw------- 1 root root  227 Mar 21 20:26 client-kubelet.key
-rw-r--r-- 1 root root 3051 Mar 21 20:34 client-scheduler.crt
-rw------- 1 root root  227 Mar 21 20:34 client-scheduler.key
-rw-r--r-- 1 root root 4773 Mar 21 20:27 dynamic-cert.json
drwxr-xr-x 2 root root 4096 Mar 21 20:26 etcd
-rw-r--r-- 1 root root 2555 Mar 21 20:30 request-header-ca.crt
-rw------- 1 root root  454 Mar 21 20:30 request-header-ca.key
-rw-r--r-- 1 root root 2472 Mar 21 20:30 server-ca.crt
-rw------- 1 root root  454 Mar 21 20:30 server-ca.key
-rw-r--r-- 1 root root  627 Mar 21 20:34 server-ca.nochain.crt
-rw------- 1 root root 1675 Mar 21 20:34 service.current.key
-rw------- 1 root root 3354 Mar 21 20:30 service.key
-rw-r--r-- 1 root root 3278 Mar 21 20:34 serving-kube-apiserver.crt
-rw------- 1 root root  227 Mar 21 20:34 serving-kube-apiserver.key
-rw------- 1 root root  227 Mar 21 20:26 serving-kubelet.key
drwx------ 2 root root 4096 Mar 21 20:26 temporary-certs

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@brandond brandond

@ShylajaDevadiga ShylajaDevadiga

Labels
None yet
Projects
K3s Development
Archived in project
Milestone
v1.24.12+k3s1
Development

No branches or pull requests
2 participants
@brandond @ShylajaDevadiga