crio 1.27 has default image registry.k8s.io/pause:3.6 #6985

Jeansen · 2023-05-30T08:27:11Z

What happened?

Creating a cluster wie kubeadm 1.27 and cri-o 1.27 shows:

W0530 09:51:11.146538   50207 checks.go:835] detected that the sandbox image "registry.k8s.io/pause:3.6" of the container runtime is inconsistent with that used by kubeadm. It is recommended that using "registry.k8s.io/pause:3.9" as the CRI sandbox image.

Although, kubeadm config images list gives:

 back to the nearest etcd version (3.5.7-0)
registry.k8s.io/kube-apiserver:v1.27.2
registry.k8s.io/kube-controller-manager:v1.27.2
registry.k8s.io/kube-scheduler:v1.27.2
registry.k8s.io/kube-proxy:v1.27.2
registry.k8s.io/pause:3.9
registry.k8s.io/etcd:3.5.7-0
registry.k8s.io/coredns/coredns:v1.10.1

cri-o overwrites the kubeadm image for pause with its default, which currently is 3.6. When I set the relevant config entry to in /etc/crio/crio.conf, e.g. pause_image = "registry.k8s.io/pause:3.9 and restart the crio service, kubeadm no longer complains.

What did you expect to happen?

No warning and the right pause image in use when creating a cluster with kubeadm in the same version as crio.

How can we reproduce it (as minimally and precisely as possible)?

Follow https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/.
In Essence, it would be enough to have von VM running Ubuntu or Debian with kubeadm and crio installed in version 1.27 and then running kubeadm init.

Anything else we need to know?

No response

CRI-O and Kubernetes version

crio version 1.27.0
Version:        1.27.0
GitCommit:      unknown
GitCommitDate:  unknown
GitTreeState:   clean
BuildDate:      2023-04-25T18:11:41Z
GoVersion:      go1.19
Compiler:       gc
Platform:       linux/amd64
Linkmode:       dynamic
BuildTags:
  apparmor
  seccomp
  containers_image_ostree_stub
  exclude_graphdriver_btrfs
  exclude_graphdriver_devicemapper
  containers_image_openpgp
LDFlags:          -s -w -X github.com/cri-o/cri-o/internal/pkg/criocli.DefaultsPath="" -X github.com/cri-o/cri-o/internal/version.buildDate=2023-04-25T18:11:41Z
SeccompEnabled:   true
AppArmorEnabled:  false

{
  "clientVersion": {
    "major": "1",
    "minor": "27",
    "gitVersion": "v1.27.2",
    "gitCommit": "7f6f68fdabc4df88cfea2dcf9a19b2b830f1e647",
    "gitTreeState": "clean",
    "buildDate": "2023-05-17T14:20:07Z",
    "goVersion": "go1.20.4",
    "compiler": "gc",
    "platform": "linux/amd64"
  },
  "kustomizeVersion": "v5.0.1",
  "serverVersion": {
    "major": "1",
    "minor": "27",
    "gitVersion": "v1.27.2",
    "gitCommit": "7f6f68fdabc4df88cfea2dcf9a19b2b830f1e647",
    "gitTreeState": "clean",
    "buildDate": "2023-05-17T14:13:28Z",
    "goVersion": "go1.20.4",
    "compiler": "gc",
    "platform": "linux/amd64"
  }
}

OS version

# On Linux:
$ cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

$ uname -a
Linux master0-k8s 5.10.0-20-amd64 #1 SMP Debian 5.10.158-2 (2022-12-13) x86_64 GNU/Linux

Additional environment details (AWS, VirtualBox, physical, etc.)

All my Nodes are created with Vagrant/Libvirt.

The text was updated successfully, but these errors were encountered:

sohankunkerkar · 2023-05-30T14:19:36Z

This change needs to get backported to 1.27, I will put up a PR shortly.

Jeansen · 2023-05-30T19:09:13Z

Great, thx. But how about the fact that crio overwrites kubernetes? Is this by design?

haircommander · 2023-05-30T19:18:18Z

currently crio and kubelet both have independent ideas of what pause image to use. Eventually, CRI implementation should specify the Pinned field of the ImageStatus response to tell kubelet to not garbage collect that image. Then, the kubelet can stop being in the business of knowing the pause image. CRI-O has recently added support for this, though I'm not sure if containerd has yet.

afbjorklund · 2023-05-31T05:28:10Z

There should be a warning in place now, if the container runtime has not been updated to use the kubeadm version...

Clean up dockershim flags in the kubelet kubernetes/kubernetes#106893

875c3a2

afbjorklund · 2023-05-31T05:29:16Z

Great, thx. But how about the fact that crio overwrites kubernetes? Is this by design?

It is at least the documented workaround, kubernetes requires the user to reconfigure their runtimes.

https://kubernetes.io/docs/setup/production-environment/container-runtimes/

Out of the box, they will probably use the wrong pause image - and might even use the wrong cgroups.

Jeansen · 2023-05-31T08:53:22Z

@afbjorklund Thank you for the insight but I cannot see that is is a requirement to overwrite the sandbox (pause) image, only an option that on CAN set.

Jeansen · 2023-05-31T08:55:00Z

There should be a warning in place now, if the container runtime has not been updated to use the kubeadm version...

Clean up dockershim flags in the kubelet kubernetes/kubernetes#106893

875c3a2

Yes, there is such a warning. That's how I initially came across the fact that an older pause image (3.6) had been used, instead of a more recent one (3.9).

afbjorklund · 2023-05-31T12:34:46Z

I backported the warning to cri-o and docker, since it was only implemented (and tested) for containerd

I think once the backport of the default image to cri-o 1.27 is done, it shouldn't need config (the default should be OK)

Jeansen · 2023-06-01T14:25:07Z

I suggest to keep this ticket open until I can verify it in a test.

github-actions · 2023-07-02T00:03:06Z

A friendly reminder that this issue had no activity for 30 days.

haircommander · 2023-07-03T18:31:47Z

@Jeansen what's the status of your check?

Jeansen · 2023-07-06T16:58:09Z

@haircommander Oh, sorry. It's vacation time in Germany ;-) I'll check soon and report here.

Jeansen · 2023-07-16T12:26:52Z

Installing from the repository I initially could not see any changes.

During kubeadm setup, I still get:
W0716 13:21:33.701119 4044 checks.go:835] detected that the sandbox image "registry.k8s.io/pause:3.6" of the container runtime is inconsistent with that used by kubeadm. It is recommended that using "registry.k8s.io/pause:3.9" as the CRI sandbox image.

Anyway, installing without any additional configuration settings afterwards shows me, that kubelet runs with version 3.9:

/usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --container-runtime-endpoint=unix:///var/run/crio/crio.sock --pod-infra-container-image=registry.k8s.io/pause:3.9

I assume, this is fine now?

haircommander · 2023-07-25T15:27:04Z

if you run crio config | grep pause I'm guessing there's a configuration file you have present that is setting 3.6, if you're using a binary whose internal version is 3.9

Jeansen · 2023-08-03T08:06:15Z

No, there is nothing set. Like I wrote, I use the default settings.

github-actions · 2023-09-03T00:02:45Z

A friendly reminder that this issue had no activity for 30 days.

shqear93 · 2023-09-23T09:26:03Z

same here

haircommander · 2023-09-26T20:32:47Z

Installing from the repository I initially could not see any changes.

During kubeadm setup, I still get: W0716 13:21:33.701119 4044 checks.go:835] detected that the sandbox image "registry.k8s.io/pause:3.6" of the container runtime is inconsistent with that used by kubeadm. It is recommended that using "registry.k8s.io/pause:3.9" as the CRI sandbox image.

Anyway, installing without any additional configuration settings afterwards shows me, that kubelet runs with version 3.9:

/usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --container-runtime-endpoint=unix:///var/run/crio/crio.sock --pod-infra-container-image=registry.k8s.io/pause:3.9

I assume, this is fine now?

this seems like there's some kubeadm field that's getting in the way. If your crio configuration says 3.9 then that's the one that's being used, regardless of what kubelet/kubeadm ask.

afbjorklund · 2023-09-27T06:46:02Z

If the crio config was indeed also changed to 3.9, then it should be reflected in crictl info... It probably needs a reload, though (but the upgrade should have done that)

github-actions · 2023-10-28T00:03:32Z

A friendly reminder that this issue had no activity for 30 days.

haircommander · 2023-12-04T20:38:20Z

I think this can be closed, CRI-O is using the pause image it's configured to. kubeadm/kubelet can only ask and CRI-O will ignore :)

Jeansen added the kind/bug Categorizes issue or PR as related to a bug. label May 30, 2023

sohankunkerkar self-assigned this May 30, 2023

sohankunkerkar mentioned this issue May 30, 2023

[1.27] *: update sandbox/pause image to 3.9 #6987

Merged

github-actions bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jul 2, 2023

haircommander removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jul 3, 2023

github-actions bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Sep 3, 2023

sohankunkerkar removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Sep 26, 2023

github-actions bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Oct 28, 2023

haircommander closed this as completed Dec 4, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

crio 1.27 has default image registry.k8s.io/pause:3.6 #6985

crio 1.27 has default image registry.k8s.io/pause:3.6 #6985

Jeansen commented May 30, 2023 •

edited

sohankunkerkar commented May 30, 2023 •

edited

Jeansen commented May 30, 2023

haircommander commented May 30, 2023

afbjorklund commented May 31, 2023 •

edited

afbjorklund commented May 31, 2023 •

edited

Jeansen commented May 31, 2023

Jeansen commented May 31, 2023 •

edited

afbjorklund commented May 31, 2023 •

edited

Jeansen commented Jun 1, 2023

github-actions bot commented Jul 2, 2023

haircommander commented Jul 3, 2023

Jeansen commented Jul 6, 2023

Jeansen commented Jul 16, 2023 •

edited

haircommander commented Jul 25, 2023

Jeansen commented Aug 3, 2023

github-actions bot commented Sep 3, 2023

shqear93 commented Sep 23, 2023

haircommander commented Sep 26, 2023

afbjorklund commented Sep 27, 2023

github-actions bot commented Oct 28, 2023

haircommander commented Dec 4, 2023

crio 1.27 has default image registry.k8s.io/pause:3.6 #6985

crio 1.27 has default image registry.k8s.io/pause:3.6 #6985

Comments

Jeansen commented May 30, 2023 • edited

What happened?

What did you expect to happen?

How can we reproduce it (as minimally and precisely as possible)?

Anything else we need to know?

CRI-O and Kubernetes version

OS version

Additional environment details (AWS, VirtualBox, physical, etc.)

sohankunkerkar commented May 30, 2023 • edited

Jeansen commented May 30, 2023

haircommander commented May 30, 2023

afbjorklund commented May 31, 2023 • edited

afbjorklund commented May 31, 2023 • edited

Jeansen commented May 31, 2023

Jeansen commented May 31, 2023 • edited

afbjorklund commented May 31, 2023 • edited

Jeansen commented Jun 1, 2023

github-actions bot commented Jul 2, 2023

haircommander commented Jul 3, 2023

Jeansen commented Jul 6, 2023

Jeansen commented Jul 16, 2023 • edited

haircommander commented Jul 25, 2023

Jeansen commented Aug 3, 2023

github-actions bot commented Sep 3, 2023

shqear93 commented Sep 23, 2023

haircommander commented Sep 26, 2023

afbjorklund commented Sep 27, 2023

github-actions bot commented Oct 28, 2023

haircommander commented Dec 4, 2023

Jeansen commented May 30, 2023 •

edited

sohankunkerkar commented May 30, 2023 •

edited

afbjorklund commented May 31, 2023 •

edited

afbjorklund commented May 31, 2023 •

edited

Jeansen commented May 31, 2023 •

edited

afbjorklund commented May 31, 2023 •

edited

Jeansen commented Jul 16, 2023 •

edited