-
Notifications
You must be signed in to change notification settings - Fork 2.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
x509: certificate has expired or is not yet valid #5163
Comments
Generally it's best to stop k3s before jumping the time forward; there are bits of Kubernetes that don't react well to time skips like that. We've tested certificate renewal on startup extensively in the past and I would be disappointed if it regressed; can you confirm that this still is an issue if K3s is stopped when you move the time forward? |
@brandond I changed the os time to
|
You appear to still have K3s running while moving the time forward? Stop it before changing the time. |
@brandond Hi, I followed your suggestion to re-test and still have the same problem, the steps are as follows:
Then, query the
|
I am facing same issue in v1.20.0-rc5+k3s1 |
I also found the same problem on k3s version 1.22.6 |
I am facing the same issue in v1.23.8+k3s, I did some digging and these clues might be useful. Normally, K3s use CertificateRenewDays to determine whether a renewed certificate is required. After a new certificate is generated, it will be stored in memory, file, and k8s secret. I added a line to the log which shows that the new certificate has been generated. Compared to fingerprint, I'm sure this is the new certificate.
However, when the secret was finally updated, the new certificate was not written. Fingerprints are still old:
@brandond |
Right, that change in behavior is discussed in the docstring for the function.
Checking the fingerprint annotation should not be necessary since the second (newer) secret is always returned by the merge function if the new secret contains all the CNs from the current secret, as is the case when updating a certificate to extend its duration. |
I just moved to k3s and did a fresh install. I'm facing the cert issue as above on v1.23.8+k3s2. |
@brandond Thanks for your reply, you are right. There is really no need to check the fingerprint. I have some new discoveries. If the
Maybe |
@niusmallnan how are you are you reproducing this? Are you changing the system time or something else to simulate an expired cert? For me, simply restarting the process will reliably renew the certificate for me when it has less than 90 days of validity remaining. |
@brandond I reproduce this way #5163 (comment) |
Hmm, interesting. Clearly we need a bit more testing in this area.
|
Ah, I see. The locally cached secret is used initially, but when the apiserver comes up, the datastore secret is merged into it, and replaces it - which is what we want in general, to ensure that the datastore certificate is used in favor of the one from the local node. However, we shouldn't do this if the one from the datastore is expired. |
I use this workaround to get the
|
This comment was marked as off-topic.
This comment was marked as off-topic.
@LarsBingBong no, what you're reporting does not have anything to do with the dynamic certificate expiring. Please open a new issue. |
i'm seeing what i -think- might be a related problem, running on a celeron N5095 under truenas scale (it's plausible this is part of truenas, but, it occurs when starting the k3s came up and generated the certificate and was working okay, then i corrected my timezone settings and restarted and now the certificate is not yet valid, causing k3s to panic on boot. so there's definitely no regenerating certificates happening and i cant use kubectl to clear out existing secrets. the panic i'm seeing is:
|
That particular crash is deep inside the Kubernetes code; it looks like they don't expect that sort of error. I will say that our cert renewal code checks for certificates to be expired, it does not check to see if they are from the future. I am not sure I'm interested in fixing that particular case; don't set your system clock backwards. |
thanks for the response! i waited it out so am back running now ^_^
it's probably not something that comes up a lot in the world of real computers, but it is reasonably common for embedded devices not to have RTCs so time travel is part of the experience. most often this is in the -forward- direction which should already be handled but it can happen backwards if you're unlucky, and you certainly can't expect the system time to be correct at boot.
well i didn't -intend- to set the system clock backwards, but my timezone was wrong and i live in the future (GMT+12) 😅 |
Hi, I'm facing the same issue now:
I have run the cluster normally almost one year. K3S version:v1.21.3+k3s1 |
@xzycn please see the steps at #5163 (comment) |
@brandond level=warning msg="Unable to watch for tunnel endpoints: the server has asked for the client to provide credentials (get endpoints) After I restart k3s-agent on the worker node, the errors disappear. Yes, I should repeat the action on every node. Is there any official docs about the certificates in k3s cluster? |
Another question: Thank you in advance :) |
Problem exists on 1.23.9+k3s1 as well. The workaround works:
I see a potential fix (#5951) has been merged to 1.23. When do you think a release with this fix in place will be made please? Also is there a way I can install the master 1.23.x version so I can test and report back it's fixed on a VM? |
Validated on version
Environment DetailsInfrastructure
Node(s) CPU architecture, OS, and Version:
Cluster Configuration:
Testing Steps
Validation Results:
|
I have tested 1.23.10-rc1+k3s1 and the problem has been resolved. Thanks. |
In my case I found the following.
--> contrary to my belief that certificate is the (generated) one from |
@marcelstoer you're not looking at the server certificate for the apiserver. As suggested by the filename, that is a client certificate. The server cert and key used by the listener are stored in the secret and cached locally in the json file. Can you confirm what version of K3s you're using? |
Thanks for the clarification Brad. I observed this on ancient v1.17.15+k3s1 installation. |
Ah yes, that is ancient, and I would expect that version to still be affected by this issue. It has long been fixed on newer releases. |
So far I've been reluctant to upgrade this installation as I'm really not an k3s expert (or k8s for that matter). However, according to https://docs.k3s.io/upgrades/manual#manually-upgrade-k3s-using-the-binary it should be as easy as replacing the k3s binary. What can go wrong... |
Step through each Kubernetes minor one at a time, and be ready to upgrade any user manifests that use old API versions. |
Environmental Info:
K3s Version: v1.22.6+k3s1 and v1.23.4+k3s1
Node(s) CPU architecture, OS, and Version: ubuntu 1804
Cluster Configuration: 1 server
Describe the bug:
Cannot rotate
k3s-serving
certificate after restarting k3sSteps To Reproduce:
curl -sfL https://get.k3s.io | sh -
Fri Feb 25 02:37:07 UTC 2022
Expected behavior:
Restart k3s to automatically rotate certificates
Actual behavior:
Restart k3s, k3s-serving does not automatically rotate.
But I can manually rotate the k3s-serving certificate by:
Additional context / logs:
Backporting
The text was updated successfully, but these errors were encountered: