CoreDNS/kubeadm incompatibility

[jonathan-conder-sm]

What happened:
I was more or less following these instructions for EKS-D 1-21-4: https://distro.eks.amazonaws.com/users/install/kubeadm-onsite/
But CoreDNS fails to start with these errors in the logs:

pkg/mod/k8s.io/client-go@v0.20.2/tools/cache/reflector.go:167: Failed to watch *v1beta1.EndpointSlice: failed to list *v1beta1.EndpointSlice: endpointslices.discovery.k8s.io is forbidden: User "system:serviceaccount:kube-system:coredns" cannot list resource "endpointslices" in API group "discovery.k8s.io" at the cluster scope

What you expected to happen:
CoreDNS should be Running and Ready

How to reproduce it (as minimally and precisely as possible):
Host system is ubuntu 20.04.
Install kubeadm=1.21.2-00, kubectl=1.21.2-00 and kubelet=1.21.2-00 following https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/
Then do this:

$ for binary in kubeadm kubectl kubelet; do curl -O "https://distro.eks.amazonaws.com/kubernetes-1-21/releases/4/artifacts/kubernetes/v1.21.2/bin/linux/amd64/${binary}"; done
$ chmod +x kubeadm kubectl kubelet
# cp kubeadm kubectl kubelet /usr/bin/
# docker pull public.ecr.aws/eks-distro/kubernetes/pause:v1.21.2-eks-1-21-4
# docker tag public.ecr.aws/eks-distro/kubernetes/pause:v1.21.2-eks-1-21-4 public.ecr.aws/eks-distro/kubernetes/pause:3.4.1

Create kubeadm-config.yaml with the following contents (replace podSubnet with something available on your network):

apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
imageRepository: public.ecr.aws/eks-distro/kubernetes
kubernetesVersion: v1.21.2-eks-1-21-4
dns:
  imageRepository: public.ecr.aws/eks-distro/coredns
  imageTag: v1.8.3-eks-1-21-4
etcd:
  local:
    imageRepository: public.ecr.aws/eks-distro/etcd-io
    imageTag: v3.4.16-eks-1-21-4
networking:
  podSubnet: 10.177.0.0/16
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cgroupDriver: systemd

Finally run:

# kubeadm init --config kubeadm-config.yaml
$ mkdir -p $HOME/.kube
$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
$ sudo chown $(id -u):$(id -g) $HOME/.kube/config
$ kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml

and check the logs of one of the coredns pods.

Anything else we need to know?:
The error is due to a change in CoreDNS 1.8.1: https://coredns.io/2021/01/20/coredns-1.8.1-release/ and has been fixed in upstream kubeadm but not backported to 1.21 (which uses 1.8.0 by default): kubernetes/kubernetes@74feb07#diff-80bea83c0faf0435d38773c725ba352bfd0e7e0aee6d0cdaa1d223ec5a4189b4
I suggest you cherry-pick that commit or downgrade your CoreDNS version accordingly.

Environment:
I used versions from here: https://github.com/aws/eks-distro/blob/v1-21-eks-4/development/pull-all.sh

[jaxesn]

Good callout! We stay in sync with the versions that EKS uses in the cloud this is why we bumped to CoreDNS 1.8.3 with 1.20 and then 1.21. Bumping ahead of upstream like this can/will create issues like this which are not ideal. On the EKS side and during our e2e tests using kops, we apply the necessary changes to the role. We can add this to add our docs to make that more clear.

Since there is a fix in the 1.22 branch upstream now, we probably would accept a backport patch for 1.20 and 1.21 if you are interested in contributing.

[jonathan-conder-sm]

Yeah I could probably make a PR when I get the chance

[jaxesn]

Fixed in #547