Remote kubectl x509: certificate is valid for 127.0.0.1

[ferllings]

Version:
k3s version v1.17.2+k3s1 (cdab19b)
kubectl v1.17.2

Describe the bug
I just did a fresh k3s install, and it's working fine locally.
And I'm trying to connect the server remotely.
I did a copy of the k3s.yaml locally, updated the server ip.

But when I run kubectl get all I get the following error:
Unable to connect to the server: x509: certificate is valid for 10.43.0.1, 127.0.0.1, not [remote ip]

Thanks for your help.

[brandond]

Did you note the --tls-san option? You can use this to add additional hostnames that the cert is valid for.

   --tls-san value                            (listener) Add additional hostname or IP as a Subject Alternative Name in the TLS cert

https://rancher.com/docs/k3s/latest/en/installation/install-options/#registration-options-for-the-k3s-server

[ferllings]

Thanks,
Is that a new option? I never had to use that before.
Should I put it directly inside the systemd file? or somewhere else?

[brandond]

By default it seems to add a bunch of SANs, including one for the eth0 address. Did you pass it any odd arguments that might have changed that?

X509v3 Subject Alternative Name:
    DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc.cluster.local, DNS:localhost, IP Address:10.0.1.20, IP Address:10.43.0.1, IP Address:127.0.0.1

If you want to change it, I think you need to reinstall from scratch since it only generates the certificate on initial startup or when it's about to expire.

[ferllings]

Thanks, it's working now:
I reinstalled using
curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--tls-san x.x.x.x" sh -s -

To give a little more context, the server is Scaleway cloud instance.
The public IP is external, it doesn't show up on ifconfig. That's probably why it's not added during the default install.

But what I don't understand is that I have an older cloud instance, where I installed k3s few months ago, and did'nt had to do that.

Anyway. Thanks again.

[dirien]

Thanks a lot for the snippet! That helped a lot!

[keyvaann]

It would be good to have the option to change the SANs without reinstalling the cluster

[ukd1]

It would be good to have the option to change the SANs without reinstalling the cluster

I'd love this as well

[Abhinav-26]

hey @brandond Is there any way to just add a new IP using flag --tls-san without reinstalling cluster again with the given flag?

[MichaelBui]

FYI, I've solved my issue without re-installing my cluster by edit the secret:

kubectl -n kube-system edit secrets/k3s-serving

You will see something like this:

kind: Secret
metadata:
  annotations:
    listener.cattle.io/cn-10.43.0.1: 10.43.0.1
    listener.cattle.io/cn-127.0.0.1: 127.0.0.1

If you want to add IP 1.2.3.4 as a SAN, just add 1 more line like this (noted that the IP appears twice):

kind: Secret
metadata:
  annotations:
    listener.cattle.io/cn-10.43.0.1: 10.43.0.1
    listener.cattle.io/cn-127.0.0.1: 127.0.0.1
    listener.cattle.io/cn-1.2.3.4: 1.2.3.4

Hope it helps!

[gesarki]

@MichaelBui How did you recreate the secret with that annotation applied?

After adding that annotation, I restarted k3s and checked the secret contents by base64 decoding it and then running it through openssl x509 -in certificate.crt -text -noout and it still didn't have the new SAN field.

[MichaelBui]

@gesarki I edit the secret directly. Even after I started (embedded k3s inside FreeNAS SCALE), the annotation is still there & I can connect using my Lens as normal

[reesericci]

Can I add multiple SANs using this option?

[Epic55]

@MichaelBui Hi, i did like u wrote, but it didn't help. I added my external IP in secrets, i restarted k3s, but i can't connect to k3s with kubectl from local station, the same error.
I found 1 article on the internet, there is info to regenerate apiserver cert. Do i need to regenerate apiserver cert? cause there is no an external ip address in /var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt

[sxyandapp]

我尝试了各种办法（在不重新安装k3s的情况下），最终都没有成功。最终我在k3s中禁用调metrics-server，然后手动使用 metrics-server-components.yaml 文件安装成功，需要注意的是，需要在.spec.template.spec.containers.args 中添加--kubelet-insecure-tls 参数

[vast0906]

FYI, I've solved my issue without re-installing my cluster by edit /etc/systemd/system/k3s.service

ExecStart=/usr/local/bin/k3s \
    server \
       '–advertise-address=106.xx.xx.xx' \
       '-tls-san=106.xx.xx.xx' \  ## this is important

and

kubectl -n kube-system delete secrets/k3s-serving
mv /var/lib/rancher/k3s/server/tls/dynamic-cert.json /tmp/dynamic-cert.json
systemctl restart k3s

k3s verson

 k3s --version
k3s version v1.22.7+k3s1 (8432d7f2)
go version go1.16.10

you can try it

[arthurnn]

after changing the tls-san I ran this on the server

curl -vk --resolve 172.31.13.97:6443:127.0.0.1  https://172.31.13.97:6443/ping

and the new certificate was right. source

[lakamsani]

On Rancher Desktop 1.7.0 on a Mac, can confirm this workaround with the modification of editing secrets/serving-cert instead of secrets/k3s-serving

[pnocera]

Also don't forget to copy the secrets/k3s-serving tls.crt base64 data into your local kubeconfig clusters->cluster->certificate-authority-data after the secret has been regenerated. ( it took me one hour to understand why it was still failing )

[MonolithicMonk]

I just want to add this here for some that my run into similar issue as me.

In my situation, there was already an annotation for the problematic ip address. I discovered this when I tried to add the annotation for it as suggested by @MichaelBui.

So what what worked for me was to simply restart the server using sudo systemctl restart k3s