kind v0.18 images do not have kindnetd and local-path provisioner preloaded properly

[anzdaddy]

What happened:

kind create cluster pulls image kindest/node:v1.26.3 for me, resulting in a broken kindnet pod (stuck in ErrImagePull/ImagePullBackOff):

❯ kind create cluster
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.26.3) 🖼
 ✓ Preparing nodes 📦
 ✓ Writing configuration 📜
 ✓ Starting control-plane 🕹️
 ✓ Installing CNI 🔌
 ✓ Installing StorageClass 💾
Set kubectl context to "kind-kind"
You can now use your cluster with:

kubectl cluster-info --context kind-kind

Have a nice day! 👋
❯ kubectl get pod -n kube-system | grep kindnet
❯ kubectl get pod -n kube-system | grep kindnet
kindnet-kd9gx                                0/1     ErrImagePull   0          1s
❯ kubectl get pod kindnet-kd9gx -o json -nkube-system | jq '.spec.containers[].image'
"docker.io/kindest/kindnetd:v20230330-48f316cd@sha256:c19d6362a6a928139820761475a38c24c0cf84d507b9ddf414a078cf627497af"
❯ docker run -it kind-control-plane crictl images
IMAGE                                                     TAG                 IMAGE ID            SIZE
docker.io/kindest/kindnetd:v20230330-48f316cd             <none>              43ef1c5209cd9       25.3MB
docker.io/kindest/local-path-provisioner:v0.0.23-kind.0   <none>              9eda906092e57       16.6MB
registry.k8s.io/coredns/coredns                           v1.9.3              b19406328e70d       13.4MB
registry.k8s.io/etcd                                      3.5.6-0             ef24580282403       80.5MB
registry.k8s.io/kube-apiserver                            v1.26.3             92e90fc362928       78MB
registry.k8s.io/kube-controller-manager                   v1.26.3             96fd77e7825a4       66.7MB
registry.k8s.io/kube-proxy                                v1.26.3             53df69d2174ba       63.4MB
registry.k8s.io/pause                                     3.7                 e5a475a038057       268kB
docker.io/kindest/local-path-helper:v20230330-48f316cd    <none>              e5f9a0a1ed364       2.92MB
registry.k8s.io/kube-scheduler                            v1.26.3             9c689f0fff925       56.3MB

What you expected to happen:

Using an older image, kind create cluster works as intended.

❯ kind create cluster --image kindest/node:v1.25.3
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.25.3) 🖼
 ✓ Preparing nodes 📦
 ✓ Writing configuration 📜
 ✓ Starting control-plane 🕹️
 ✓ Installing CNI 🔌
 ✓ Installing StorageClass 💾
Set kubectl context to "kind-kind"
You can now use your cluster with:

kubectl cluster-info --context kind-kind

Not sure what to do next? 😅  Check out https://kind.sigs.k8s.io/docs/user/quick-start/
❯ kubectl get pod -n kube-system | grep kindnet
❯ kubectl get pod -n kube-system | grep kindnet
❯ kubectl get pod -n kube-system | grep kindnet
kindnet-sdnzt                                0/1     ContainerCreating   0          0s
❯ kubectl get pod kindnet-sdnzt -o json -nkube-system | jq '.spec.containers[].image'
"docker.io/kindest/kindnetd:v20221004-44d545d1"
❯ docker run -it kind-control-plane crictl images
IMAGE                                      TAG                  IMAGE ID            SIZE
docker.io/kindest/kindnetd                 v20221004-44d545d1   7ba9b35cf55e6       23.7MB
docker.io/kindest/local-path-helper        v20220607-9a4d8d2a   304f67e47fc80       2.75MB
docker.io/kindest/local-path-provisioner   v0.0.22-kind.0       7902f9a1c54fa       15.6MB
registry.k8s.io/coredns/coredns            v1.9.3               b19406328e70d       13.4MB
registry.k8s.io/etcd                       3.5.4-0              8e041a3b0ba8b       81.1MB
registry.k8s.io/kube-apiserver             v1.25.3              cf534618e3365       74.2MB
registry.k8s.io/kube-controller-manager    v1.25.3              144f2d840093a       62.3MB
registry.k8s.io/kube-proxy                 v1.25.3              da3a3d1b12f76       59.6MB
registry.k8s.io/kube-scheduler             v1.25.3              44edac9952d58       50.6MB
registry.k8s.io/pause                      3.7                  e5a475a038057       268kB

How to reproduce it (as minimally and precisely as possible):

works fine:

❯ kind create cluster --image kindest/node:v1.25.3

broken kindnet:

❯ kind create cluster

Anything else we need to know?:

n/a

Environment:

• kind version: (use kind version): kind v0.18.0 go1.20.2 darwin/arm64

• Runtime info: (use docker info or podman info):

  (some content masked as …)

  ❯ docker info
  Client:
  Context:    desktop-linux
  Debug Mode: false
  Plugins:
    buildx: Docker Buildx (Docker Inc.)
      Version:  v0.10.3
      Path:     …/.docker/cli-plugins/docker-buildx
    compose: Docker Compose (Docker Inc.)
      Version:  v2.15.1
      Path:     …/.docker/cli-plugins/docker-compose
    dev: Docker Dev Environments (Docker Inc.)
      Version:  v0.1.0
      Path:     …/.docker/cli-plugins/docker-dev
    extension: Manages Docker extensions (Docker Inc.)
      Version:  v0.2.18
      Path:     …/.docker/cli-plugins/docker-extension
    sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
      Version:  0.6.0
      Path:     …/.docker/cli-plugins/docker-sbom
    scan: Docker Scan (Docker Inc.)
      Version:  v0.25.0
      Path:     …/.docker/cli-plugins/docker-scan
    scout: Command line tool for Docker Scout (Docker Inc.)
      Version:  v0.6.0
      Path:     …/.docker/cli-plugins/docker-scout
  
  Server:
  Containers: 3
    Running: 3
    Paused: 0
    Stopped: 0
  Images: 153
  Server Version: 20.10.23
  Storage Driver: overlay2
    Backing Filesystem: extfs
    Supports d_type: true
    Native Overlay Diff: true
    userxattr: false
  Logging Driver: json-file
  Cgroup Driver: cgroupfs
  Cgroup Version: 2
  Plugins:
    Volume: local
    Network: bridge host ipvlan macvlan null overlay
    Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
  Swarm: inactive
  Runtimes: runc io.containerd.runc.v2 io.containerd.runtime.v1.linux
  Default Runtime: runc
  Init Binary: docker-init
  containerd version: 2456e983eb9e37e47538f59ea18f2043c9a73640
  runc version: v1.1.4-0-g5fd4c4d
  init version: de40ad0
  Security Options:
    seccomp
    Profile: default
    cgroupns
  Kernel Version: 5.15.49-linuxkit
  Operating System: Docker Desktop
  OSType: linux
  Architecture: aarch64
  CPUs: 6
  Total Memory: 11.68GiB
  Name: docker-desktop
  ID: RLIZ:YP3S:XWNF:65BY:TGXT:ADTA:VJHC:KT6N:SKAP:GWHF:FU3C:TMCB
  Docker Root Dir: /var/lib/docker
  Debug Mode: false
  HTTP Proxy: http.docker.internal:3128
  HTTPS Proxy: http.docker.internal:3128
  No Proxy: hubproxy.docker.internal
  Registry: https://index.docker.io/v1/
  Experimental: false
  Insecure Registries:
    hubproxy.docker.internal:5000
    127.0.0.0/8
  Registry Mirrors:
    https://hub.artifactory.…/
    https://gcr.artifactory.…/
  Live Restore Enabled: false
  

• OS (e.g. from /etc/os-release):

  ❯ uname -a
  Darwin T409FVWQCM 22.4.0 Darwin Kernel Version 22.4.0: Mon Mar  6 20:59:28 PST 2023; root:xnu-8796.101.5~3/RELEASE_ARM64_T6000 arm64
  

• Kubernetes version: (use kubectl version):

  With broken kind cluster:

  ❯ kubectl version
  Found existing alias for "kubectl". You should use: "k"
  WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
  Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.1", GitCommit:"4c9411232e10168d7b050c49a1b59f6df9d7ea4b", GitTreeState:"clean", BuildDate:"2023-04-14T13:14:41Z", GoVersion:"go1.20.3", Compiler:"gc", Platform:"darwin/arm64"}
  Kustomize Version: v5.0.1
  Server Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.3", GitCommit:"9e644106593f3f4aa98f8a84b23db5fa378900bd", GitTreeState:"clean", BuildDate:"2023-03-30T06:36:32Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/arm64"}
  

  With working kind cluster:

  ❯ kubectl version
  WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
  Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.1", GitCommit:"4c9411232e10168d7b050c49a1b59f6df9d7ea4b", GitTreeState:"clean", BuildDate:"2023-04-14T13:14:41Z", GoVersion:"go1.20.3", Compiler:"gc", Platform:"darwin/arm64"}
  Kustomize Version: v5.0.1
  Server Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.3", GitCommit:"434bfd82814af038ad94d62ebe59b133fcb50506", GitTreeState:"clean", BuildDate:"2022-11-02T03:24:50Z", GoVersion:"go1.19.2", Compiler:"gc", Platform:"linux/arm64"}
  WARNING: version difference between client (1.27) and server (1.25) exceeds the supported minor version skew of +/-1
  

• Any proxies or other special environment settings?:

  n/a

[BenTheElder]

Can you please share the kind export logs output as described in the bug template for failed cluster creation? That will have more useful information

[BenTheElder]

Also for completeness:

Please be forewarned that using images like this is not recommended vs digest pinning and using images across kind release may not work. We announce known breakages in the release notes and try to avoid them but can't guarantee it.

See text here
https://kind.sigs.k8s.io/docs/user/quick-start/#creating-a-cluster
https://github.com/kubernetes-sigs/kind/releases/tag/v0.18.0#new-features

[BenTheElder]

I think I know what happened here, however, and it's more generally a problem with breaking airgapping in kindnetd reference
/assign

[BenTheElder]

This should be fixed in upcoming images, sorry about that.

[BenTheElder]

Try using --image=kindest/node:v1.27.1@sha256:c44686bf1f422942a21434e5b4070fc47f3c190305be2974f91444cd34909f1b with v0.18.0

[anzdaddy]

I didn't send kind export log because it didn't seem that the creation process was actually failing, just that the end result wasn't fully functional. I'm happy to send them through if you still need them.

The --image=kindest/node:v1.27.1@… works for now. Thank you!

[BenTheElder]

For the record because someone else will find this issue: the kind export logs files might've helped clarify the issue quicker, the logs are useful for non-functional clusters, but it's not necessary now, thanks!

#3177 has more context on what's wrong with the v0.18 images in ~air-gapped environments.

I'd guess in this case not strictly airgapped but proxied or otherwise setup in some way that the host can pull from dockerhub but the nodes can't.

It's considered a bug if fully airgapped doesn't work, so we'll probably have to push more images and a release soon with this fix https://kind.sigs.k8s.io/docs/user/working-offline/

[BenTheElder]

v0.19 should not have this issue https://github.com/kubernetes-sigs/kind/releases/tag/v0.19.0