prefer-bundled-bin flag cannot be set via config.yaml.d

[brandond]

Via @Oats87 on Slack

The --prefer-bundled-bin flag can only be set via the CLI or config.yaml. If placed in a yaml file in config.yaml.d it is not found.

This appears to be because the FindString code only processes the main config file, it does not iterate over files in config.yaml.d:

• k3s/cmd/k3s/main.go

  Line 121 in a92f163

  preferRes := configfilearg.MustFindString(args, "prefer-bundled-bin")

• k3s/pkg/configfilearg/parser.go

  Lines 99 to 102 in a92f163

  	func (p *Parser) FindString(args []string, target string) (string, error) {
  	configFile, isSet := p.findConfigFileFlag(args)
  	if configFile != "" {
  	bytes, err := readConfigFileData(configFile)

Compare this to the Parse code, which iterates over both the main config file, and the files in the .d directory:

• k3s/pkg/configfilearg/parser.go

  Lines 220 to 224 in a92f163

  	for _, file := range files {
  	bytes, err := readConfigFileData(file)
  	if err != nil {
  	return nil, err
  	}

[brandond]

cc @dereknola

[brandond]

Note that this should be validated by looking at the actual PATH env var for the k3s process. The issue is specifically in the function that sets up the order of that var; the function that sets the node-args annotation works properly.

# replace PID with the k3s process ID
cat /proc/PID/environ | xargs -n1 -0 echo | grep PATH 

[VestigeJ]

$ cat validation_template.md

##Environment Details
Initial version used for comparison VERSION=v1.27.1+k3s1
CommitID used to validate changes COMMIT=a736b4b1b932a2f513077959a931f8f2faae6329

Infrastructure

• Cloud

Node(s) CPU architecture, OS, and version:

Linux 5.14.21-150400.24.11-default x86_64 GNU/Linux 
PRETTY_NAME="SUSE Linux Enterprise Server 15 SP4"

Cluster Configuration:

NAME              STATUS   ROLES                       AGE    VERSION
ip-1-1-1-42       Ready    control-plane,etcd,master   124m   v1.27.1+k3s-a736b4b1 

Config.yaml:

$ get_figs

write-kubeconfig-mode: 644
debug: true
token: gurtyo
selinux: true
protect-kernel-defaults: true
cluster-init: true

$ sudo cat /etc/rancher/k3s/config.yaml.d/test.yaml

prefer-bundled-bin: true

Reproduction

$ curl https://get.k3s.io --output install-"k3s".sh
$ sudo chmod +x install-"k3s".sh
$ sudo groupadd --system etcd && sudo useradd -s /sbin/nologin --system -g etcd etcd
$ sudo modprobe ip_vs_rr
$ sudo modprobe ip_vs_wrr
$ sudo modprobe ip_vs_sh
$ sudo printf "on_oovm.panic_on_oom=0 \nvm.overcommit_memory=1 \nkernel.panic=10 \nkernel.panic_ps=1 \nkernel.panic_on_oops=1 \n" > ~/60-rke2-cis.conf or ~/90-kubelet.conf
$ sudo cp 60-rke2-cis.conf /etc/sysctl.d/ or 90-kubelet.conf
$ sudo systemctl restart systemd-sysctl
$ sudo INSTALL_K3S_VERSION=v1.27.1+k3s1 INSTALL_K3S_TYPE=server INSTALL_K3S_SKIP_START=true ./install-k3s.sh
$ sudo k3s server
$ ID=$(pgrep k3s-server)
$ sudo cat /proc/${ID}/environ | xargs -n1 -0 echo | grep PATH
$ get_report //generate this template

Results:

Observe the ordering of the path elements with the /usr/sbin/:/usr/bin:/usr/local/bin etc listed earlier and therefore higher priority than the /bin/aux
$ sudo cat /proc/${ID}/environ | xargs -n1 -0 echo | grep PATH

PATH=/var/lib/rancher/k3s/data/599b4f3127975cf230091dd1fbb1c60fb7fa3ce1782b287fd8269855a279a757/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/bin:/usr/local/sbin:/var/lib/rancher/k3s/data/599b4f3127975cf230091dd1fbb1c60fb7fa3ce1782b287fd8269855a279a757/bin/aux

Validation

Validation Steps

$ curl https://get.k3s.io --output install-"k3s".sh
$ sudo chmod +x install-"k3s".sh
$ sudo groupadd --system etcd && sudo useradd -s /sbin/nologin --system -g etcd etcd
$ sudo modprobe ip_vs_rr
$ sudo modprobe ip_vs_wrr
$ sudo modprobe ip_vs_sh
$ sudo printf "on_oovm.panic_on_oom=0 \nvm.overcommit_memory=1 \nkernel.panic=10 \nkernel.panic_ps=1 \nkernel.panic_on_oops=1 \n" > ~/60-rke2-cis.conf or ~/90-kubelet.conf
$ sudo cp 60-rke2-cis.conf /etc/sysctl.d/ or 90-kubelet.conf
$ sudo systemctl restart systemd-sysctl
$ sudo INSTALL_K3S_COMMIT=a736b4b1b932a2f513077959a931f8f2faae6329 INSTALL_K3S_TYPE=server INSTALL_K3S_SKIP_START=true ./install-k3s.sh
$ sudo k3s server
$ ID=$(pgrep k3s-server)
$ sudo cat /proc/${ID}/environ | xargs -n1 -0 echo | grep PATH
$ get_report //generate this template

Results:

$ sudo cat /proc/${ID}/environ | xargs -n1 -0 echo | grep PATH

PATH=/var/lib/rancher/k3s/data/7d4b5857c8071c3f5514e1640808d9006c9d57f1e1751174bc60c8f816e28d59/bin:/var/lib/rancher/k3s/data/7d4b5857c8071c3f5514e1640808d9006c9d57f1e1751174bc60c8f816e28d59/bin/aux:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/bin:/usr/local/sbin

Additional context / logs: