cni bin_dir not updated automatically after k3s update

[janekmichalik]

Environmental Info:
K3s Version:

k3s version v1.25.11+k3s1 (582f07cf)
go version go1.19.10

Node(s) CPU architecture, OS, and Version:

Linux mclx-36 5.4.0-81-generic #91-Ubuntu SMP Thu Jul 15 19:09:17 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration:

cluster with 1 node

Describe the bug:

If the config.toml.tmpl (under /var/lib/rancher/k3s/agent/etc/containerd) was created before manually, after k3s upgrade, the file still contains refs to old k3s (cni bin_dir).

version = 2

[plugins."io.containerd.internal.v1.opt"]
  path = "/var/lib/rancher/k3s/agent/containerd"
[plugins."io.containerd.grpc.v1.cri"]
  stream_server_address = "127.0.0.1"
  stream_server_port = "10010"
  enable_selinux = false
  enable_unprivileged_ports = true
  enable_unprivileged_icmp = true
  sandbox_image = "rancher/mirrored-pause:3.6"

[plugins."io.containerd.grpc.v1.cri".containerd]
  snapshotter = "overlayfs"
  disable_snapshot_annotations = true


[plugins."io.containerd.grpc.v1.cri".cni]
  bin_dir = "/var/lib/rancher/k3s/data/7c994f47fd344e1637da337b92c51433c255b387d207b30b3e0262779457afe4/bin"
  conf_dir = "/var/lib/rancher/k3s/agent/etc/cni/net.d"


[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
  runtime_type = "io.containerd.runc.v2"

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
  SystemdCgroup = true

and I need to manually replace current commit sha with the new one or current to make it work.
But I m afraid that if sth else will be changed as well, I won't include that change. And the replace solution seems to be a workaround.

Steps To Reproduce:

• install k3s 1.25.4
• create config.toml.tmpl (under /var/lib/rancher/k3s/agent/etc/containerd)
• restart k3s to load new config
• upgrade k3s to 1.25.11

Expected behavior:
k3s up and running

Actual behavior:
Pods stuck in CreateContainer phase with error:

Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "2ee8a11bbce2c5afe55dab4bba14909f741186a33531fb1bc9b2e81e432779b8": plugin type="bandwidth" failed (add): failed to find plugin "bandwidth" in path [/var/lib/rancher/k3s/data/7c994f47fd344e1637da337b92c51433c255b387d207b30b3e0262779457afe4/bin]

I m not sure if it a bug, but I could not find ask question section.

[dereknola]

When using a custom containerd configuration, it will be up to you to maintain it. K3s will not attempt to update or modify that file. In your setup, using the current/bin directory will go a long way to making upgrades easier. Outside of the occasional update to sandbox_image = "rancher/mirrored-pause:3.6" most of the other values will not change release to release.

[brandond]

#7991 will make this easier to a certain extent - users will be able to provide additional config sections without having to copy-paste and maintain the whole base config.

I'm going to close this out for now, as the behavior is (as @dereknola explained) expected when you provide your own config template.

[janekmichalik]

thx @dereknola and @brandond for feedback.