-
Notifications
You must be signed in to change notification settings - Fork 4.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
/srv/kubernetes/kubelet-server.crt expired, did not auto-renew #15970
Comments
No, kops expects you to update nodes at least every 455 days. |
OK, good to at least know that's expected behavior, thanks. |
@johngmyers rotateCertificates is not used for this kind of use-case? |
Hey @darintay @johngmyers, following the above: my company has inherited k8s clusters made and maintained by kOps, and one of them (fortunately, the non-production one) started having this issue of nodes going offline due to the expiration of the kubelet-server.crt, as it seems. I’m wondering what’s the best approach here? I noticed you mentioned that a manual kops update cluster is needed every at least 455 days, but, what should I do if this cluster wasn’t updated in time, and some of its nodes already have their certificates expired?
We're pretty stuck here and we have more clusters (production) that we afraid that would have the same issue, and we won't know how to handle it properly. Thank you in advance! |
@aviadhaham This may be too late, but I'll comment here in case anyone else runs into this issue. The If you're attempting to recover without also running an update to limit the number of moving parts during recovery, you'll need the Fixing the control plane first before moving on to nodes is a good idea, so an example of doing that first: Of course, updating more often in the future to avoid this should be a priority--but this will get things running again so you can focus on updating. |
/kind bug
1. What
kops
version are you running? The commandkops version
, will displaythis information.
1.25.4
2. What Kubernetes version are you running?
kubectl version
will print theversion if a cluster is running or provide the Kubernetes version specified as
a
kops
flag.1.23.5
3. What cloud provider are you using?
AWS
4. What commands did you run? What is the simplest way to reproduce this issue?
A node that was running for 400+ days had its kubelet-server.crt certificate expire, which broke all pods on the node.
Is something supposed to be auto-renewing this certificate? I couldn't see anything in the kubelet logs about it. I know that kubelets have certificate rotation (https://kubernetes.io/docs/tasks/tls/certificate-rotation/) but I don't know if that is supposed to be covering this file, or if it's something on the kops side.
Not sure if there's an easy way to test/reproduce this due to the duration of these certs.
(I know ideally I'd be doing control plane upgrades frequently enough that this doesn't matter, but would like to sort this out for if/when that doesn't happen again in future)
The text was updated successfully, but these errors were encountered: