kubelet-rubber-stamp is simple CSR auto approver operator to help bootstrapping kubelet serving certificates easily.
The logic used follows the same logic used when auto-approving kubelet client certificates in kubelet TLS bootstrap phase.
So basically the flow is:
- kubelet gets the client cert (see TLS bootstrap)
- Kubelet creates a CSR
- kubelet-rubber-stamp reacts to the creation of a CSR
- validates that it's a valid request for kubelet serving certificate
- validates that the requestor (the kubelet/node) has sufficient authorization
- approve the CSR
- Kubelet fetches the certificate
- Kubelet auto-rotates certs, goto 2 :)
To enable kubelet serving certificate bootstrapping you need to set
serverTLSBootstrap: true in kubelet configuration file. That will enable kubelet to start the automated bootsrapping process.
Once kubelet, api-server and kubelet-rubber-stamp have all played nice, kubelet should have gotten hold of the automatically bootsrapped serving certificate.
Kubelet stores the certificates it uses on disk, under
root@cluster-worker-4:~# ls -lah /var/lib/kubelet/pki/kubelet-server-* -rw------- 1 root root 1.2K Mar 6 09:46 /var/lib/kubelet/pki/kubelet-server-2019-03-06-09-46-48.pem lrwxrwxrwx 1 root root 59 Mar 6 09:46 /var/lib/kubelet/pki/kubelet-server-current.pem -> /var/lib/kubelet/pki/kubelet-server-2019-03-06-09-46-48.pem
You can see the certificate details e.g. with:
root@cluster-worker-4:~# openssl x509 -in /var/lib/kubelet/pki/kubelet-server-current.pem -text -noout
Check that the output details match the expected ones, especially that the certificate is signed by your Kubernetes CA.
Another way of validating is to peek into the kubelet API to see which certificate it offers:
root@cluster-worker-4:~# openssl s_client -showcerts -connect localhost:10250 </dev/null ...
Check that the certificate kubelet offers in it's API is signed by control plane.
deploy directory has the needed skeleton YAMLs for deploying to a cluster. They should though be taken more as reference, so adapt to your cluster setup.
Image is available on
quay.io/kontena/kubelet-rubber-stamp-<arch>:<tag>. We bake both
arm64 images automatically. We've also automated several different tags:
latest- Always up-to-date with
masterbranch. Might not be 100% stable.
x.y.z- Actual releases from git tags. Creates also semver "aliases", so releasing e.g.
0.1tag point to latest
Use the usual troubleshooting steps as for any app running on Kubernetes:
- Is the pod running?
kubectl get pod --all-namespaces
- Check the pod/container logs For this you probably need to ssh into the node as if kubelet does not have proper TLS cert, API server will not allow to proxy the logs.
- Make sure your RBAC rules allow kubelet to create proper CSRs
- Check kubelet logs for any errors related to TLS bootstrapping
- Check that the CSRs in the API look correct.
kubectl get csr ...
Bug reports and pull requests are welcome on GitHub at https://github.com/kontena/kubelet-rubber-stamp
Copyright (c) 2019 Kontena, Inc.
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.