Kind fails kubadm command: unknown flag --skip-phases

[FraBle]

What happened:

We've upgraded the local tooling on macOS from kind 0.8.1 to 0.9.0. Creating clusters now breaks with a kubeadm error.

ERROR: failed to create cluster: failed to join node with kubeadm: command "docker exec --privileged skynet-worker kubeadm join --config /kind/kubeadm.conf --skip-phases=preflight --v=6" failed with error: exit status 1
Command Output: Error: unknown flag: --skip-phases

What you expected to happen:

Cluster coming up healthy.

How to reproduce it (as minimally and precisely as possible):

kind.json:

{
    "kind": "Cluster",
    "apiVersion": "kind.x-k8s.io/v1alpha4",
    "nodes": [
        {
            "role": "control-plane",
            "image": "$KUBE_NODE_IMAGE",
            "extraMounts": [
                {
                    "hostPath": "/Users/$USER/.docker/config.json",
                    "containerPath": "/var/lib/kubelet/config.json"
                }
            ]
        },
        {
            "role": "worker",
            "image": "$KUBE_NODE_IMAGE",
            "extraMounts": [
                {
                    "hostPath": "/Users/$USER/.docker/config.json",
                    "containerPath": "/var/lib/kubelet/config.json"
                }
            ]
        }
    ]
}

setup.sh:

#!/bin/bash
set -e

export KUBE_NODE_IMAGE="kindest/node:v1.13.12@sha256:214476f1514e47fe3f6f54d0f9e24cfb1e4cda449529791286c7161b7f9c08e7"
config="$(cat kind.json | envsubst)"
kind create cluster --name skynet --config=<(echo "$config")

Terminal output

./setup.sh
Creating cluster "skynet" ...
 ✓ Ensuring node image (kindest/node:v1.13.12) 🖼
 ✓ Preparing nodes 📦 📦
 ✓ Writing configuration 📜
 ✓ Starting control-plane 🕹️
 ✓ Installing CNI 🔌
 ✓ Installing StorageClass 💾
 ✗ Joining worker nodes 🚜
ERROR: failed to create cluster: failed to join node with kubeadm: command "docker exec --privileged skynet-worker kubeadm join --config /kind/kubeadm.conf --skip-phases=preflight --v=6" failed with error: exit status 1
Command Output: Error: unknown flag: --skip-phases
Usage:
  kubeadm join [flags]

Flags:
      --apiserver-advertise-address string            If the node should host a new control plane instance, the IP address the API Server will advertise it's listening on.
      --apiserver-bind-port int32                     If the node should host a new control plane instance, the port for the API Server to bind to. (default 6443)
      --config string                                 Path to kubeadm config file.
      --cri-socket string                             Specify the CRI socket to connect to. (default "/var/run/dockershim.sock")
      --discovery-file string                         A file or URL from which to load cluster information.
      --discovery-token string                        A token used to validate cluster information fetched from the API server.
      --discovery-token-ca-cert-hash strings          For token-based discovery, validate that the root CA public key matches this hash (format: "<type>:<value>").
      --discovery-token-unsafe-skip-ca-verification   For token-based discovery, allow joining without --discovery-token-ca-cert-hash pinning.
      --experimental-control-plane                    Create a new control plane instance on this node
  -h, --help                                          help for join
      --ignore-preflight-errors strings               A list of checks whose errors will be shown as warnings. Example: 'IsPrivilegedUser,Swap'. Value 'all' ignores errors from all checks.
      --node-name string                              Specify the node name.
      --token string                                  Use this token for both discovery-token and tls-bootstrap-token when those values are not provided.

Global Flags:
      --log-file string   If non-empty, use this log file
      --rootfs string     [EXPERIMENTAL] The path to the 'real' host root filesystem.
      --skip-headers      If true, avoid header prefixes in the log messages
  -v, --v Level           log level for V logs

error: unknown flag: --skip-phases

Anything else we need to know?:

• N/A

Environment:

• kind version: (use kind version): kind v0.9.0 go1.15.2 darwin/amd64
• Kubernetes version: (use kubectl version):

  Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.12", GitCommit:"a8b52209ee172232b6db7a6e0ce2adc77458829f", GitTreeState:"clean", BuildDate:"2019-10-15T12:12:15Z", GoVersion:"go1.11.13", Compiler:"gc", Platform:"darwin/amd64"}
  

• Docker version: (use docker info):

  Server Version: 19.03.12
  Kernel Version: 4.19.76-linuxkit
  Operating System: Docker Desktop
  

• OS (e.g. from /etc/os-release): macOS Catalina 10.15.6 (19G2021)

[BenTheElder]

See existing discussion: #1744 (comment), #1744 (comment)

The release notes have been updated. 1.14+ will be required for multi-node clusters for now.

[FraBle]

@BenTheElder Was just about to close this issue since I discovered this comment as well 😅
Thanks for the references!

[BenTheElder]

This one is really unfortunate and surprising, I will have to dig later to find out what the thinking was upstream 😞

I think we may be able to re-introduce 1.13 support in the future, but it will require a somewhat involved rewrite of bootstrapping to use the phases directly. It's something on our radar but we have some other large changes to tackle first. We also need to speak to the kubeadm team more before moving on this as it alters the upstream test coverage.

[FraBle]

Ack. Thanks for the details! 🙌
I will update our tooling to pin to kind v0.8.1 in the meantime.

For anyone running into the same issue, this is how we pinned it via homebrew:

if ! brew list | grep kind@0.8.1 > /dev/null ; then
    echo "[$(date +"%T")] Installing kind v0.8.1..."
    if ! brew tap | grep $USER/kind > /dev/null; then
        brew tap-new $USER/kind
        brew extract --version=0.8.1 kind $USER/kind
    fi
    brew install kind@0.8.1
fi

[neolit123]

This one is really unfortunate and surprising, I will have to dig later to find out what the thinking was upstream

kubeadm init phases were added in 1.13 and join phases in 1.14, because it was a lot of work and a matter of developer / reviewer bandwidth:

kubernetes/kubeadm#1163
kubernetes/kubeadm#1204