Certificate gets to invalid and no way to recover if k3s gets started at a future time

[ckyoog]

Environmental Info:
K3s Version: v1.23.10+k3s1
Linux system: Alpine-3.15.4
Linux kernel: 5.15.55-0-virt

Describe the bug:
Certificate gets to invalid and no way to recover if k3s gets started at a future time. This issue is different from #5163, which is about the certificate is used after the valid time range. My issue is about the certificate is used before the valid time range. They are opposite. The workaround mentioned in #5163 and the fix #5896 don't work for this one.

Steps To Reproduce:
My steps

• start a linux system, somehow the system time is a future time
• and the system has no network settings (means no access to the internet),
• install and start k3s
• restart ntpd, the right system time is set, and it is earlier than the time when k3s got started,
• then this error messages can be seen everywhere, in /var/log/k3s.log, when run "kubectl", ...
  Unable to connect to the server: x509: certificate has expired or is not yet valid: current time 2022-09-16T18:32:28Z is before 2022-09-16T18:41:21Z

For you, to easily reproduce

• stop ntpd
• run date <a future time>
• install and start k3s
• start ntpd
• then there you go

Expected behavior:
Unless I set the system time to the future time again, k3s is not usable any more. Restarting can't fix it. k3s can rotate the expired certs to the future, it also needs to rotate the future certs to now, doesn't it?

[brandond]

Duplicate of #5163 (comment)

You should ensure that time does not go backwards while Kubernetes is running, in particular backwards to before the service was started.

[ckyoog]

Thank you for responding. I of course don't want to set the system time backwards. It is because the system time was wrong when k3s was started, then ntpd corrected the time, made it go backwards.

And it is no way for me to start k3s after ntpd corrects the system time. Also, in my opinion, k3s should not stop working because of these certificates stuff, otherwise you guys wouldn't have bothered to add the "auto-renew" feature.

[brandond]

Right, but as noted in the comments on that other issue, Kubernetes does not handle time going backwards well if at all. Renewal covers extending certificates that have or are about to expire, it does not handle certificates that aren't valid yet because they were issued in the future.

If this is a common problem for you, you might consider adding a dependency on your time sync service of choice to the k3s service.

[ckyoog]

Ok..., is there any workarounds to make k3s re-generate those invalid certificates?

I tried deleting the whole dir /var/lib/rancher/k3s/server/tls, and also deleting the secret k3s-serving (using etcd-ctrl, kubectl is no more usable at this moment), then start k3s again, but it didn't work. Is there any thing I could do manually to get k3s started again?

Getting it work again is so important and basic, isn't? Like 5163 (comment) mentioned, k3s doesn't work once the timezone changed. And like my case, k3s doesn't work once the system time goes backwards. I don't think these are what you guys expected, right? (just FYI, in my case, k3s was not as panic as the 5163 (comment).)

[JasperWey]

face same issue, set time backwards, and k3s crash with x509 error

[brandond]

Don't set the system time backwards to before K3s was originally started. Everything, including the cluster CA certificates, will be invalid. This is not something we are planning on supporting.