Minor issues with the k8s template

[afbjorklund]

Description

Trying out k8s 1.32 RC, found some issues:

• Flannel was a bit outdated (PR added)

#2991

• kubeadm config version was updated

[  191.575331] cloud-init[1077]: + kubeadm init --config kubeadm-config.yaml
[  191.605187] cloud-init[1077]: W1207 10:05:49.428211    3159 common.go:101] your configuration file uses a deprecated API spec: "kubeadm.k8s.io/v1beta3" (kind: "ClusterConfiguration"). Please use 'kubeadm config migrate --old-config old.yaml --new-config new.yaml', which will write the new, similar spec using a newer API version.

your configuration file uses a deprecated API spec: "kubeadm.k8s.io/v1beta3"

#2996

• sandbox image is not configured properly
• still using the old containerd config version (v2)

[  194.231682] cloud-init[1077]: [preflight] Running pre-flight checks
[  194.360972] cloud-init[1077]: [preflight] Pulling images required for setting up a Kubernetes cluster
[  194.361128] cloud-init[1077]: [preflight] This might take a minute or two, depending on the speed of your internet connection
[  194.361354] cloud-init[1077]: [preflight] You can also perform this action beforehand using 'kubeadm config images pull'
[  194.365795] cloud-init[1077]: W1207 10:05:52.188987    3159 checks.go:846] detected that the sandbox image "" of the container runtime is inconsistent with that used by kubeadm.It is recommended to use "registry.k8s.io/pause:3.10" as the CRI sandbox image.

detected that the sandbox image "" of the container runtime is inconsistent with that used by kubeadm

containerd/containerd#11114
kubernetes/website#48976

---

EDIT: It did work OK, just complained in the logs.

Switching k8s versions involves hacking the yaml.

[afbjorklund]

Actual the sandbox was configured, so the problem seems to be with containerd 2.0 CRI and kubernetes 1.31

crictl info

// SandboxImage returns the sandbox image used by the container runtime
func (runtime *CRIRuntime) SandboxImage() (string, error) {
        ctx, cancel := defaultContext()
        defer cancel()
        status, err := runtime.impl.Status(ctx, runtime.runtimeService, true)
        if err != nil {
                return "", errors.Wrap(err, "failed to get runtime status")
        }

        infoConfig, ok := status.GetInfo()["config"]
        if !ok {
                return "", errors.Errorf("no 'config' field in CRI info: %+v", status)
        }

        type config struct {
                SandboxImage string `json:"sandboxImage,omitempty"`
        }
        c := config{}

        if err := json.Unmarshal([]byte(infoConfig), &c); err != nil {
                return "", errors.Wrap(err, "failed to unmarshal CRI info config")
        }

        return c.SandboxImage, nil
}

Maybe they shouldn't have used the semi-internal "config", but a proper CRI method to communicate it?

containerd/containerd@3baf5ed

[afbjorklund]

The new config format (v3) is not documented upstream yet, they only do v2:

https://kubernetes.io/docs/setup/production-environment/container-runtimes/#containerd

https://github.com/containerd/containerd/blob/main/docs/cri/config.md

[plugins."io.containerd.grpc.v1.cri"]
  sandbox_image = "registry.k8s.io/pause:3.2"

  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
    SystemdCgroup = true

Nice touch of documenting an ancient version.

[plugins.'io.containerd.cri.v1.images'.pinned_images]
  sandbox = 'registry.k8s.io/pause:3.10'

[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc.options]
  SystemdCgroup = true

Since it has been moved to "images", it is no longer visible in the "runtime" status.

                configByt, err := json.Marshal(c.config)
...
                resp.Info = make(map[string]string)
                resp.Info["config"] = string(configByt)

[afbjorklund]

• Export compatibility config.sandboxImage in CRI containerd/containerd#11114

[afbjorklund]

Probably these things are never seen upstream, since kind disables the preflight <sigh>

[afbjorklund]

Issues have been opened in the upstream projects, so nothing more to do in Lima...

And it is only a couple of lines of log warnings now, who looks at those anyway right?

Configuration migrated from version 2, use containerd config migrate to avoid migration
WARN[0000] Ignoring unknown key in TOML for plugin error="strict mode: fields in the document are missing in the target struct" key=sandbox_image plugin=io.containerd.grpc.v1.cri

detected that the sandbox image "" of the container runtime is inconsistent with that used by kubeadm.It is recommended to use "registry.k8s.io/pause:3.10" as the CRI sandbox image.

---

It does work fine, and the configuration is the correct one (despite all the false complaints)

[  155.739859] cloud-init[1103]: [init] Using Kubernetes version: v1.32.0
[  155.740445] cloud-init[1103]: [preflight] Running pre-flight checks
[  155.819598] cloud-init[1103]: [preflight] Pulling images required for setting up a Kubernetes cluster
[  155.820668] cloud-init[1103]: [preflight] This might take a minute or two, depending on the speed of your internet connection
[  155.822049] cloud-init[1103]: [preflight] You can also perform this action beforehand using 'kubeadm config images pull'
[  155.823408] cloud-init[1103]: W1214 10:00:30.977242    3115 checks.go:846] detected that the sandbox image "" of the container runtime is inconsistent with that used by kubeadm.It is recommended to use "registry.k8s.io/pause:3.10" as the CRI sandbox image.

anders@lima-k8s:~$ sudo containerd config dump
...
    [plugins.'io.containerd.cri.v1.images'.pinned_images]
      sandbox = 'registry.k8s.io/pause:3.10'
...

CONTAINER ID    IMAGE                                              COMMAND                   CREATED          STATUS    PORTS    NAMES
7d4ce40061bc    registry.k8s.io/pause:3.10                         "/pause"                  5 minutes ago    Up                 k8s://kube-system/coredns-668d6bf9bc-bdpdb
cb9846518450    registry.k8s.io/pause:3.10                         "/pause"                  5 minutes ago    Up                 k8s://kube-system/coredns-668d6bf9bc-vthwq
2d5d56cf0a6d    registry.k8s.io/pause:3.10                         "/pause"                  6 minutes ago    Up                 k8s://kube-flannel/kube-flannel-ds-9m4gp
bbd8907d1685    registry.k8s.io/pause:3.10                         "/pause"                  6 minutes ago    Up                 k8s://kube-system/kube-proxy-sqc7t
ed7594cd9743    registry.k8s.io/pause:3.10                         "/pause"                  6 minutes ago    Up                 k8s://kube-system/kube-scheduler-lima-k8s
c7de7bee144f    registry.k8s.io/pause:3.10                         "/pause"                  6 minutes ago    Up                 k8s://kube-system/kube-controller-manager-lima-k8s
4b179d0c0b6c    registry.k8s.io/pause:3.10                         "/pause"                  6 minutes ago    Up                 k8s://kube-system/kube-apiserver-lima-k8s
833a01c3900a    registry.k8s.io/pause:3.10                         "/pause"                  6 minutes ago    Up                 k8s://kube-system/etcd-lima-k8s

There are other known issues at startup anyway, like the kubelet crash-looping (because it is started too soon, before being configured and before the cluster networking has been set up - which is supposed to pushed from CNI to CRI)

[afbjorklund]

This output seems redundant as well, ideally kubeadm should stay quiet - since we already did that step.

[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action beforehand using 'kubeadm config images pull'

Again this is something for upstream, one would add a exist-only step (depending on the pull policy)...

        if data.DryRun() {
                fmt.Println("[preflight] Would pull the required images (like 'kubeadm config images pull')")
                return nil
        }

        // new code goes here:
        // if preflight.RunImagesExistCheck(utilsexec.New(), data.Cfg(), data.IgnorePreflightErrors()) == nil {
        //        return nil
        //}

        fmt.Println("[preflight] Pulling images required for setting up a Kubernetes cluster")
        fmt.Println("[preflight] This might take a minute or two, depending on the speed of your internet connection")
        fmt.Println("[preflight] You can also perform this action beforehand using 'kubeadm config images pull'")
        return preflight.RunPullImagesCheck(utilsexec.New(), data.Cfg(), data.IgnorePreflightErrors())

[afbjorklund]

• kubeadm: check if all images already exist kubernetes/kubernetes#129474