k3s seems to ignore --data-dir option when rotating certs using script

[tomwiggers]

Environmental Info:
K3s Version:

~# kubectl get node
NAME        STATUS   ROLES                       AGE     VERSION
<server>    Ready    control-plane,etcd,master   7h10m   v1.26.3+k3s1
<server>    Ready    control-plane,etcd,master   6h18m   v1.26.3+k3s1
<server>    Ready    control-plane,etcd,master   6h26m   v1.26.3+k3s1
<agent>     Ready    <none>                      21d     v1.26.1+k3s1
<agent>     Ready    <none>                      6h14m   v1.26.3+k3s1

Node(s) CPU architecture, OS, and Version:
Linux macaw 5.10.0-21-arm64 #1 SMP Debian 5.10.162-1 (2023-01-21) aarch64 GNU/Linux

Cluster Configuration:
3 servers, 2 agents. All of them except one agent are arm64 (one agent is amd64).

Describe the bug:
I am facing some TLS issues in the cluster and would like try rotating the certificates in the cluster. I tried to run k3s -d /k3s certificate rotate. With -d as I have the k3s data on a seperate partition. I stopped k3s first. However it seems to ignore the option:

~# k3s -d /k3s certificate rotate
FATA[0000] open /var/lib/rancher/k3s/server/token: no such file or directory

One agent (the amd64 one), is a slightly older version, but this issue was noticed on one of the server nodes.

See also #7081.

Steps To Reproduce:

• Installed K3s: v1.26.3+k3s1 (01ea3ff2) with data directory on a different location than default.
• Stop k3s.
• Attempt to rotate certificates using the script: k3s -d <dir> certificate rotate.

Expected behavior:
Certificates rotated succesfully.

Actual behavior:
An error was shown stating it cannot find the token.

Additional context / logs:
None.

Thank you! I hope I did not do something wrong and it actually does work. I did check this page and there does not seem to be test case covering this, but I am not sure if that is needed.

[brandond]

You shouldn't mix subcommands and flags, the CLI parser gets confused. Can you try k3s certificate rotate --data-dir /k3s instead?

I will also note that certificate rotation does not in general solve any problems. It is just a manual way to trigger something that will happen automatically when certs are about to expire.

[tomwiggers]

Hi, k3s --help states the following command syntax:
k3s [global options] command [command options] [arguments...]

The --data-dir option is a global option. If I try to put the option at the end I get a different error:

~# k3s certicate rotate --data-dir /k3s
No help topic for 'certicate'

edit: thank you by the way for the additional note, will continue troubleshooting.

[tomwiggers]

This is quite interesting, I tried it again and now it worked. The exact same command.

~# k3s certificate rotate -d /k3s
INFO[0000] Server detected, rotating server certificates
INFO[0000] Rotating certificates for admin service
INFO[0000] Rotating certificates for etcd service
INFO[0000] Rotating certificates for api-server service
INFO[0000] Rotating certificates for controller-manager service
INFO[0000] Rotating certificates for cloud-controller service
INFO[0000] Rotating certificates for scheduler service
INFO[0000] Rotating certificates for k3s-server service
INFO[0000] Rotating dynamic listener certificate
INFO[0000] Rotating certificates for k3s-controller service
INFO[0000] Rotating certificates for auth-proxy service
INFO[0000] Rotating certificates for kubelet service
INFO[0000] Rotating certificates for kube-proxy service
INFO[0000] Successfully backed up certificates for all services to path /k3s/server/tls-1681933858, please restart k3s server or agent to rotate certificates

I am still confused though as to why the option needs to be at the back of the command.

[brandond]

~# k3s certicate rotate --data-dir /k3s
 No help topic for 'certicate'

That's not how you spell "certificate".

I am still confused though as to why the option needs to be at the back of the command.

Flags go after commands. k3s certificate rotate is the command, --data-dir=x is a flag. Mixing the flags in between parts of the commands confuses the CLI parser.