K3s' `kubectl` prefers `/etc/rancher/k3s/k3s.yaml` over `~/.kube/config`

[optix2000]

Version:
k3s version v1.17.3+k3s1 (5b17a17)

K3s arguments:
curl -sfL https://get.k3s.io | sh -s - --no-flannel --no-deploy traefik

Describe the bug
kubectl prefers reading /etc/rancher/k3s/k3s.yaml over ~/.kube/config when ~/.kube/config exists.

To Reproduce

1. Write a valid ~/.kube/config file. sudo kubectl config view --raw > ~/.kube/config
2. Try running a kubectl command. kubectl config view

Expected behavior
K3s' kubectl follows the load rules in kubectl config --help, with a fallback to /etc/rancher/k3s/k3s.yaml as an additional last rule.

 The loading order follows these rules:

  1.  If the --kubeconfig flag is set, then only that file is loaded. The flag may only be set once and no merging takes
place.
  2.  If $KUBECONFIG environment variable is set, then it is used as a list of paths (normal path delimiting rules for
your system). These paths are merged. When a value is modified, it is modified in the file that defines the stanza. When
a value is created, it is created in the first file that exists. If no files in the chain exist, then it creates the
last file in the list.
  3.  Otherwise, ${HOME}/.kube/config is used and no merging takes place.

Actual behavior

• kubectl tries to read /etc/rancher/k3s/k3s.yaml and fails.
  Yes I understand that I can work around this with --write-kubeconfig-mode, but having a proper ~/.kube/config file provides interop with existing K8s tools (ie helm) and lets me work on other k8s clusters using the same kubectl command.
  Yes I also understand I can also work around this issue by exporting my own $KUBECONFIG env var, but k3s' kubectl breaks assumptions of how the command works (see below).
• kubectl config --help says 3. Otherwise, ${HOME}/.kube/config is used and no merging takes place, which doesn't seem to be the case.

Additional context / logs

% ls -l ~/.kube/config
-rw-r--r-- 1 user user 1052 Mar 14 20:32 /home/user/.kube/config
% kubectl --v=9 config view
WARN[2020-03-14T20:48:00.146588365-07:00] Unable to read /etc/rancher/k3s/k3s.yaml, please start server with --write-kubeconfig-mode to modify kube config permissions
F0314 20:48:00.159402   23853 helpers.go:114] error: error loading config file "/etc/rancher/k3s/k3s.yaml": open /etc/rancher/k3s/k3s.yaml: permission denied

[proligde]

I agree, plus it behaves differently, when kubectl was installed via snap install kubectl --classic prior to installing k3s. Then kubectl keeps using ~/.kube/config so the behaviour depends on the installation order as well which I guess it shouldn't

[brandond]

@proligde yes, the k3s embedded version of kubectl has some slightly modified behavior in order to accommodate loading config from a nonstandard location. If you install another kubectl binary to a location that's before k3s in your $PATH, that kubectl will be run by your shell instead, and it will not look for the k3s kubeconfig.

[andrewvaughan]

This seems to be a problem with environment variables too. Setting KUBECONFIG does nothing:

andrew@raspi001:~$ echo $KUBECONFIG
/home/andrew/.kube/config

Although it seems very... insistent on running with the default:

andrew@raspi001:~$ k3s kubectl config
WARN[2020-09-21T22:07:12.259477330Z] Unable to read /etc/rancher/k3s/k3s.yaml, please start server with --write-kubeconfig-mode to modify kube config permissions 
Modify kubeconfig files using subcommands like "kubectl config set
current-context my-context"

...

In fact, even using the --kubeconfig option is failing for me:

andrew@raspi001:~$ k3s kubectl config set-cluster default --server=https://10.0.0.201:6443 --kubeconfig ~andrew/.kube/config
WARN[2020-09-21T22:09:56.631637527Z] Unable to read /etc/rancher/k3s/k3s.yaml, please start server with --write-kubeconfig-mode to modify kube config permissions 
error: open /etc/rancher/k3s/k3s.yaml.lock: permission denied

This whole thing seems very broken...

[brandond]

The behavior here probably varies a bit depending on which version of k3s you're using, as we've switched cli libraries a few times. You can always install your own if you don't like the one wrapped by k3s.

[anttilinno]

Reasoning is sound for x86. Pre-built binaries exist, are easily downloadable and installable. Maybe even in package repositories. But for the raspberry, there are no prebuilt binaries that can easily be downloaded and installed :) I guess I can always download the source code and compile it for the pi, but it is general PITA.

Addendum for those, who get here via search: Ok, I can download https://packages.debian.org/sid/armhf/kubernetes-client/download for Raspberry Pi OS. Will try that out, if it works without hiccups.

[cjellick]

I sorted by 👍 and saw that this was near the top of the list, so I spoke with @ibuildthecloud about it.

Easiest thing for me to convey his intent is to just paste the convo here:

Darren:

Yeah, I don't like that behavior
I don't like that we have a modified kubectl that looks for /etc/rancher/k3s/k3s.yaml. I just wanted it to work out of the box so I added that hack.
But I really wanted curl ... | sh - && kubectl get all to work

Me: why not jsut write to ~/.kube/config? im prob being simple there

Darren:

So the kubeconfig is generated on k3s start. So that's what makes the flow odd.
If we added a k3s init command it could help.
That would generate CA, certs, kubeconfig. We could then run that in the installer and copy the kubectl to ~/.kube/config

So, if/when we address this, the "success criteria" are:

1. Don't break upgrades
2. curl ... | sh - && kubectl get all continues to work
3. We ideally use a completey unmodified version of kubectl.
4. Works properly with HA setups where additional server nodes might join later.