Kubeadm reset with Calico and Containerd

[dkoshkin]

Is this a BUG REPORT or FEATURE REQUEST?

BUG REPORT

Versions

kubeadm version (use kubeadm version):

kubeadm version: &version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.3", GitCommit:"5e53fd6bc17c0dec8434817e69b04a25d8ae0ff0", GitTreeState:"clean", BuildDate:"2019-06-06T01:41:54Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.3", GitCommit:"5e53fd6bc17c0dec8434817e69b04a25d8ae0ff0", GitTreeState:"clean", BuildDate:"2019-06-06T01:44:30Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration:
  AWS
• OS (e.g. from /etc/os-release):

[root@ip-10-0-192-121 centos]# cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

• Kernel (e.g. uname -a):
  Linux ip-10-0-192-121.us-west-2.compute.internal 3.10.0-957.1.3.el7.x86_64 kubeadm join on slave node fails preflight checks #1 SMP Thu Nov 29 14:49:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

What happened?

Installing with Containerd 1.2.5 and Calico v3.6.3 and then running kubeadm reset on one of the control-plane nodes.

$ kubeadm reset --cri-socket unix:///run/containerd/containerd.sock -v 5
...
E0626 17:49:39.212047   23718 reset.go:192] [reset] Failed to remove containers: [failed to stop running pod 69a9f78e15d3638bdb54d3a801fd4a2e103cefc03740c9664c186afa2fb28cb9: output: time="2019-06-26T17:49:37Z" level=fatal msg="stopping the pod sandbox \"69a9f78e15d3638bdb54d3a801fd4a2e103cefc03740c9664c186afa2fb28cb9\" failed: rpc error: code = Unknown desc = failed to destroy network for sandbox \"69a9f78e15d3638bdb54d3a801fd4a2e103cefc03740c9664c186afa2fb28cb9\": error getting ClusterInformation: Get https://[10.0.0.1]:443/apis/crd.projectcalico.org/v1/clusterinformations/default: dial tcp 10.0.0.1:443: connect: connection refused"
, error: exit status 1, failed to stop running pod 24495fddb8c0020b82bfad9178a2fa6fb47be0f592ed3be931e3f10ba6697ba5: output: time="2019-06-26T17:49:37Z" level=fatal msg="stopping the pod sandbox \"24495fddb8c0020b82bfad9178a2fa6fb47be0f592ed3be931e3f10ba6697ba5\" failed: rpc error: code = Unknown desc = failed to destroy network for sandbox \"24495fddb8c0020b82bfad9178a2fa6fb47be0f592ed3be931e3f10ba6697ba5\": error getting ClusterInformation: Get https://[10.0.0.1]:443/apis/crd.projectcalico.org/v1/clusterinformations/default: dial tcp 10.0.0.1:443: connect: connection refused"
, error: exit status 1]

What you expected to happen?

kubeadm reset should remove all pods.

How to reproduce it (as minimally and precisely as possible)?

• Install on a node with Containerd and use - containerd config default to get the default settings.
• Install Calico
• Run kubeadm reset

Anything else we need to know?

This happens because ListKubeContainers() returns a list of pod IDs, based on the order of those IDs, trying to delete any pods after kube-apiserver is deleted will fail. On every stopp the calico CNI plugin tries to reach out to the apiserver.

A possible simple solution to try to increase the chances of all pods being deleted is order the returned list with kube-apiserver always being the last pod.
This still would fail in a scenario when the kube-apiserver cannot be reached.

[neolit123]

/assign @bart0sh
could you please take a look.

[timothysc]

/cc @vincepri @chuckha @detiber - b/c it could affect CAPA.

[dkoshkin]

It would also be great to get the perspective from the Calico team /cc @caseydavenport

[caseydavenport]

Get https://[10.0.0.1]:443/apis/crd.projectcalico.org/v1/clusterinformations/default: dial tcp 10.0.0.1:443: connect: connection refused"

Ah, looks to me like the Calico CNI plugin is trying to access the apiserver and failing. It does this as part of tearing down networking for pods.

This might be because the kube-proxy has been removed, and the iptables cleared out? That will cut anyone off trying to access the apiserver via the Service.

[dkoshkin]

@caseydavenport right in this case its actually the api-server being stopped before other pods and the Calico CNI won't ever be able to get that CRD.

This behavior makes sense during normal execution when the del command comes from the Kubelet, but in this case when a user is trying to clean up a node with kubeadm reset (or using crictl stopp) it won't be possible to stop that pod.

Looking at the code it seems that it uses that CRD here, Im wondering if there is a workaround to prevent the Calico CNI from trying to access the api-server?

[caseydavenport]

Im wondering if there is a workaround to prevent the Calico CNI from trying to access the api-server?

I will think it through. At first glance, I'm worried that we'd be sacrificing the mainline case to cover for an exceptional one. But, there may be ways to safely remove that dependency.

What options do we have on the kubeadm side? Could we make sure the apiserver is the last resource to be destroyed?

[dkoshkin]

Ok so some more info:
Deleting the kube-apiserver last did not always work. When resetting control-plane nodes in an HA setup, etcd will lose quorum and the api-server will stop responding on whatever the last control-plane node is leaving, some pods behind.
Deleting all workload pods first, then the api-servers, then the etcd seems to be properly cleaning up the pods more consistently.

Please let me know if you would like to me to provide any more evidence.

[bart0sh]

I can't reproduce this. Here is what I did:

1. Installed containerd
2. Created new cluster: sudo -E ./kubeadm init --cri-socket unix:///run/containerd/containerd.sock --pod-network-cidr=192.168.0.0/16
3. copied config: sudo cp /etc/kubernetes/admin.conf $HOME/.kube/config && sudo chown $(id -u):$(id -g) $HOME/.kube/config
4. installed calico as suggested above: kubectl apply -f https://docs.projectcalico.org/v3.6/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml
5. waited until all pods are in running state
6. run sudo -E ./kubeadm reset --cri-socket unix:///run/containerd/containerd.sock -v5
7. checked if any kubernetes pods are still running: sudo crictl -r unix:///run/containerd/containerd.sock pods.

@dkoshkin I suspect that you're using different calico version.

[dkoshkin]

@bart0sh please try deploying a bunch of other non static pods, lets assume nginx for now, depending on the order of the pods being deleted if kube-apiserver or etcd pods are stopped first the other nginx pods wont be able to be stopped.

You can also reproduce this with crictl directly by stopping the kubelet (so it doesnt restart the pods) deleting the api-server pod first and then trying to delete any of the nginx pods.

[caseydavenport]

I've thought about this a little bit more and I'm worried it's a bit deeper than just the bit of code linked above. Even if we stop checking that CRD, fundamentally Calico uses the apiserver to store information about pods it has networked in the apiserver, and on delete it needs to clean that up. For this scenario, obviously the apiserver is not coming back, but in normal operation you can experience intermittent loss of access to the api.

The option we have is to make Calico essentially ignore errors on tear down, thus orphaning state in the apiserver, and then writing some external controller do some correlation to clean it up later if the apiserver ever comes back. This approach is relatively complicated and I worry will reduce Calico's robustness in normal operation.

[neolit123]

please re-open if found a bug on the side of kubeadm.
thanks!

[zhucan]

Is there any updates for this?