"crypto/rsa: verification error" after restarting k3s server

[iameli]

Describe the bug
I started up a k3s cluster on a Raspberry Pi 3+. Completely standard install with and everything seemed to be working well. After I restarted the server, it seems to be having problems with its TLS certification. All kubectl commands are failing with this command:

Unable to connect to the server: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "k3s-ca")

This shows up in journalctl as well, these same lines over and over:

Apr 29 03:19:44 deerling k3s[533]: time="2019-04-29T03:19:44.829455265+01:00" level=error msg="server https://localhost:6443/cacerts is not trusted: Get https://localhost:6443/cacerts: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"k3s-ca\")"

Speculation: the Pi doesn't have an onboard clock. Could the CA have gotten an incorrect time when it was generated, and now that the time updated with NTP it's no longer valid?

To Reproduce
Steps to reproduce the behavior:

1. curl -sfL https://get.k3s.io | sh - on a Pi 3 B+
2. I installed a DaemonSet? I can't imagine it caused the problem, but here's the pokemon-themed manifest.
3. Unplugged the pi and plugged it back in.

I imagine it'd be useful for me to include the CA or HTTPS cert but I'm not actually sure how to access them - will post here if someone can point me in the right direction.

[iameli]

Update: it works if I change its server to server: https://10.9.168.90:6443 in the kubeconfig file, instead of server: https://localhost:6443. Unsure what that means. (That's its LAN IP FWIW.)

[ibuildthecloud]

We may have a general issue with k3s in that we really shouldn't start the k3s server until your time is set. I think we could do something like "if the time is before 1980 wait".

[tfiduccia]

Version - v0.6.0-rc3
Verified fixed

[deminngi]

@galal-hussein

K3s - v0.6.1

After reboot I get

Unable to connect to the server: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "k3s-ca")

What I tested so far

• changed localhost to host ip address in /etc/rancher/k3s/k3s.yaml doesn't help
• restart k3s service with systemctl after reboot doesn't help.

please advice.

[hlugt]

Same issue here on K3S - v0.6.1 arm64 debian. It also happens with a systemctl restart or a start after a stop...
First time I ran into it was after upgrade form 0.5.0 to 0.6.1. (But did not check behaviour with 0.5.0)
Fixed it by recreating the cluster....
Looks like I need to do it like that again?

Retried some times after complete cleanup: same behaviour with or without boot: every k3s server restart gives me this issue...

(running on pine64 rockpro in this case)

[QuentinFAIDIDE]

You can run your k3s kubectl commands with the --insecure-skip-tls-verify flag, it will skip this cert error, but does not feel like a very safe fix.

[deminngi]

Thx @QuentinFAIDIDE
I upgraded to 0.7.0-rc1 and still no verification error anymore.

[NicklasWallgren]

Are we forced to upgrade from 0.6.1 to 0.7.0, or are there any other solutions to this problem?

[hlugt]

Thx @QuentinFAIDIDE
I upgraded to 0.7.0-rc1 and still no verification error anymore.

Also no error on a systemctl restart? On my arm64 rockpro64 I need to reboot to get the server ready for connections again. Same on 0.7.0-rc3.
Can somebody offer advice on how to troubleshoot this (to be able to maybe help with some constructive input?)