Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ingress not able to reach one of the pods because of missing flannel annotations on the nodes. #13644

Closed
sangeethah opened this issue May 22, 2018 · 23 comments

Comments

@sangeethah
Copy link
Member

commented May 22, 2018

**Rancher versions: v2.0.1-rc5

Steps to Reproduce:

Created a cluster with following node configuration:
2 control nodes
3 etcd nodes
3 worker nodes

Created a daemonset .
Created an ingress pointing to this daemonset.

When ingress url is accessed , it does not direct traffic to one of the pods in the daemonset.

Not able to access this pod using its pod ip from the host (from which the ingress url was accessed) ,

root@ip-172-31-5-12:/home/ubuntu# ping 10.42.6.2
PING 10.42.6.2 (10.42.6.2) 56(84) bytes of data.
^C
--- 10.42.6.2 ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 3000ms
@leodotcloud

This comment has been minimized.

Copy link
Member

commented May 22, 2018

A route for one of the pod CIDRs is missing causing connectivity problems:

default via 172.31.0.1 dev eth0
10.42.0.0/24 via 10.42.0.0 dev flannel.1 onlink
10.42.1.0/24 via 10.42.1.0 dev flannel.1 onlink
10.42.2.0/24 via 10.42.2.0 dev flannel.1 onlink
10.42.3.0/24 via 10.42.3.0 dev flannel.1 onlink
10.42.4.0/24 via 10.42.4.0 dev flannel.1 onlink
10.42.5.0/24 via 10.42.5.0 dev flannel.1 onlink
10.42.7.2 dev calif5bca118720  scope link
10.42.7.3 dev calidc02448b8cc  scope link
172.17.0.0/16 dev docker0  proto kernel  scope link  src 172.17.0.1 linkdown
172.31.0.0/20 dev eth0  proto kernel  scope link  src 172.31.5.12
root@ip-172-31-5-12:~#

10.42.6.0/24 via 10.42.6.0 dev flannel.1 onlink --> This should have been present in the above output.

@leodotcloud

This comment has been minimized.

Copy link
Member

commented May 22, 2018

Problematic node: ip-172-31-8-167

All the annotations related to flannel are missing on this node.

> kubectl get nodes | grep worker
ip-172-31-0-134   Ready     worker         1d        v1.9.7
ip-172-31-5-12    Ready     worker         1d        v1.9.7
ip-172-31-8-167   Ready     worker         1d        v1.9.7
>
>
>
>
>
> kubectl describe node ip-172-31-0-134
Name:               ip-172-31-0-134
Roles:              worker
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    kubernetes.io/hostname=ip-172-31-0-134
                    node-role.kubernetes.io/worker=true
Annotations:        field.cattle.io/publicEndpoints=[{"nodeName":"c-9xxcq:m-6c8e064e584d","addresses":["18.191.113.242","18.191.30.66"],"port":32355,"protocol":"TCP","serviceName":"test-96868:ingress-e20ae0ca19ae1f931ee5...
                    flannel.alpha.coreos.com/backend-data={"VtepMAC":"1a:cf:13:37:a8:b1"}
                    flannel.alpha.coreos.com/backend-type=vxlan
                    flannel.alpha.coreos.com/kube-subnet-manager=true
                    flannel.alpha.coreos.com/public-ip=172.31.0.134
                    node.alpha.kubernetes.io/ttl=0
                    rke.cattle.io/external-ip=18.191.113.242
                    rke.cattle.io/internal-ip=172.31.0.134
                    volumes.kubernetes.io/controller-managed-attach-detach=true
Taints:             <none>
CreationTimestamp:  Mon, 21 May 2018 23:06:51 +0000
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  OutOfDisk        False   Tue, 22 May 2018 23:53:56 +0000   Mon, 21 May 2018 23:06:51 +0000   KubeletHasSufficientDisk     kubelet has sufficient disk space available
  MemoryPressure   False   Tue, 22 May 2018 23:53:56 +0000   Mon, 21 May 2018 23:06:51 +0000   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Tue, 22 May 2018 23:53:56 +0000   Mon, 21 May 2018 23:06:51 +0000   KubeletHasNoDiskPressure     kubelet has no disk pressure
  Ready            True    Tue, 22 May 2018 23:53:56 +0000   Mon, 21 May 2018 23:07:21 +0000   KubeletReady                 kubelet is posting ready status
Addresses:
  InternalIP:  172.31.0.134
  Hostname:    ip-172-31-0-134
Capacity:
 cpu:     2
 memory:  4045104Ki
 pods:    110
Allocatable:
 cpu:     2
 memory:  3942704Ki
 pods:    110
System Info:
 Machine ID:                 40beb5eb909e171860ceee669da56e1d
 System UUID:                EC287B4A-0E98-34B0-63A3-5D671B674836
 Boot ID:                    4992b40c-d675-45fd-85d6-95379c91e103
 Kernel Version:             4.4.0-1049-aws
 OS Image:                   Ubuntu 16.04.3 LTS
 Operating System:           linux
 Architecture:               amd64
 Container Runtime Version:  docker://17.3.2
 Kubelet Version:            v1.9.7
 Kube-Proxy Version:         v1.9.7
PodCIDR:                     10.42.5.0/24
ExternalID:                  ip-172-31-0-134
Non-terminated Pods:         (5 in total)
  Namespace                  Name                              CPU Requests  CPU Limits  Memory Requests  Memory Limits
  ---------                  ----                              ------------  ----------  ---------------  -------------
  cattle-system              cattle-node-agent-5cv4m           0 (0%)        0 (0%)      0 (0%)           0 (0%)
  default                    add-ssh-key-mb5tf                 0 (0%)        0 (0%)      0 (0%)           0 (0%)
  ingress-nginx              nginx-ingress-controller-xtshb    0 (0%)        0 (0%)      0 (0%)           0 (0%)
  kube-system                canal-j7ctg                       250m (12%)    0 (0%)      0 (0%)           0 (0%)
  test-96868                 default-35428-bq2dl               0 (0%)        0 (0%)      0 (0%)           0 (0%)
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  CPU Requests  CPU Limits  Memory Requests  Memory Limits
  ------------  ----------  ---------------  -------------
  250m (12%)    0 (0%)      0 (0%)           0 (0%)
Events:         <none>
>
> kubectl describe node ip-172-31-8-167
Name:               ip-172-31-8-167
Roles:              worker
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    kubernetes.io/hostname=ip-172-31-8-167
                    node-role.kubernetes.io/worker=true
Annotations:        field.cattle.io/publicEndpoints=[{"nodeName":"c-9xxcq:m-719be9b84a42","addresses":["18.191.30.66","18.217.251.154"],"port":32355,"protocol":"TCP","serviceName":"test-96868:ingress-e20ae0ca19ae1f931ee5...
                    node.alpha.kubernetes.io/ttl=0
                    rke.cattle.io/external-ip=18.217.251.154
                    rke.cattle.io/internal-ip=172.31.8.167
                    volumes.kubernetes.io/controller-managed-attach-detach=true
Taints:             <none>
CreationTimestamp:  Mon, 21 May 2018 23:06:53 +0000
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  OutOfDisk        False   Tue, 22 May 2018 23:55:14 +0000   Mon, 21 May 2018 23:06:53 +0000   KubeletHasSufficientDisk     kubelet has sufficient disk space available
  MemoryPressure   False   Tue, 22 May 2018 23:55:14 +0000   Mon, 21 May 2018 23:06:53 +0000   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Tue, 22 May 2018 23:55:14 +0000   Mon, 21 May 2018 23:06:53 +0000   KubeletHasNoDiskPressure     kubelet has no disk pressure
  Ready            True    Tue, 22 May 2018 23:55:14 +0000   Mon, 21 May 2018 23:07:23 +0000   KubeletReady                 kubelet is posting ready status
Addresses:
  InternalIP:  172.31.8.167
  Hostname:    ip-172-31-8-167
Capacity:
 cpu:     2
 memory:  4045104Ki
 pods:    110
Allocatable:
 cpu:     2
 memory:  3942704Ki
 pods:    110
System Info:
 Machine ID:                 40beb5eb909e171860ceee669da56e1d
 System UUID:                EC2FC25D-1A94-E239-9147-32FBB3823ED4
 Boot ID:                    0011236d-a573-4fa6-8979-7871c2508374
 Kernel Version:             4.4.0-1049-aws
 OS Image:                   Ubuntu 16.04.3 LTS
 Operating System:           linux
 Architecture:               amd64
 Container Runtime Version:  docker://17.3.2
 Kubelet Version:            v1.9.7
 Kube-Proxy Version:         v1.9.7
PodCIDR:                     10.42.6.0/24
ExternalID:                  ip-172-31-8-167
Non-terminated Pods:         (5 in total)
  Namespace                  Name                              CPU Requests  CPU Limits  Memory Requests  Memory Limits
  ---------                  ----                              ------------  ----------  ---------------  -------------
  cattle-system              cattle-node-agent-l7vrx           0 (0%)        0 (0%)      0 (0%)           0 (0%)
  default                    add-ssh-key-kgk7q                 0 (0%)        0 (0%)      0 (0%)           0 (0%)
  ingress-nginx              nginx-ingress-controller-79jlh    0 (0%)        0 (0%)      0 (0%)           0 (0%)
  kube-system                canal-thbs2                       250m (12%)    0 (0%)      0 (0%)           0 (0%)
  test-96868                 default-35428-579dd               0 (0%)        0 (0%)      0 (0%)           0 (0%)
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  CPU Requests  CPU Limits  Memory Requests  Memory Limits
  ------------  ----------  ---------------  -------------
  250m (12%)    0 (0%)      0 (0%)           0 (0%)
Events:         <none>
>
>
@leodotcloud

This comment has been minimized.

Copy link
Member

commented May 23, 2018

Workaround: Killing the flannel container on the node fixed the problem.

root@ip-172-31-8-167:~# docker ps | grep coreos-flannel
c700e92a2c50        rancher/coreos-flannel@sha256:93952a105b4576e8f09ab8c4e00483131b862c24180b0b7d342fb360bbe44f3d             "/opt/bin/flanneld..."   25 hours ago        Up 25 hours                             k8s_kube-flannel_canal-thbs2_kube-system_a68f724f-5d4b-11e8-b2fd-022a9dd27086_0

root@ip-172-31-8-167:~# docker rm -f c700e92a2c50
c700e92a2c50
> kubectl describe node ip-172-31-8-167
Name:               ip-172-31-8-167
Roles:              worker
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    kubernetes.io/hostname=ip-172-31-8-167
                    node-role.kubernetes.io/worker=true
Annotations:        field.cattle.io/publicEndpoints=[{"nodeName":"c-9xxcq:m-719be9b84a42","addresses":["18.191.30.66","18.217.251.154"],"port":32355,"protocol":"TCP","serviceName":"test-96868:ingress-e20ae0ca19ae1f931ee5...
                    flannel.alpha.coreos.com/backend-data={"VtepMAC":"56:96:5f:0e:c5:47"}
                    flannel.alpha.coreos.com/backend-type=vxlan
                    flannel.alpha.coreos.com/kube-subnet-manager=true
                    flannel.alpha.coreos.com/public-ip=172.31.8.167
                    node.alpha.kubernetes.io/ttl=0
                    rke.cattle.io/external-ip=18.217.251.154
                    rke.cattle.io/internal-ip=172.31.8.167
                    volumes.kubernetes.io/controller-managed-attach-detach=true
Taints:             <none>

@deniseschannon deniseschannon modified the milestones: v2.0.3, v2.0.4 Jun 9, 2018

@sangeethah

This comment has been minimized.

Copy link
Member Author

commented Jul 12, 2018

This issue was seen again when testing with custom clusters using v2.0.6 release.

@cjellick

This comment has been minimized.

Copy link
Member

commented Sep 12, 2018

@leodotcloud is this a flannel bug or something rancher is doing wrong?

@cjellick cjellick assigned cjellick and unassigned leodotcloud Sep 12, 2018

@cjellick

This comment has been minimized.

Copy link
Member

commented Sep 12, 2018

The impact/scope of this issue appears to be limited and infrequent. @orangedeng please investigate this with explicit testing for flannel and canal. See if it is broke specifically for one or the other (or if it is even reproducible anymore).
Since QA will be doing network testing as part of the 2.1, we are going to move this specific issue out of the release and allow @orangedeng to research as part of 2.1.

@cjellick cjellick assigned orangedeng and unassigned cjellick Sep 12, 2018

@cjellick cjellick modified the milestones: v2.1, v2.2 Sep 12, 2018

@superseb

This comment has been minimized.

Copy link
Member

commented Sep 12, 2018

Not reproducible yet, used command below to check for annotations.

kubectl get nodes -o json | jq '.items[].metadata | select(.annotations["flannel.alpha.coreos.com/public-ip"] == null or .annotations["flannel.alpha.coreos.com/kube-subnet-manager"] == null or .annotations["flannel.alpha.coreos.com/backend-type"] == null or .annotations["flannel.alpha.coreos.com/backend-data"] == null) | .name'

Workaround still valid.

Was discovered as one of the kube-dns pods landed on a broken node, and every X DNS request would hang.

@leodotcloud

This comment has been minimized.

Copy link
Member

commented Sep 17, 2018

is this a flannel bug or something rancher is doing wrong?

@cjellick this is a flannel bug, nothing to do with Rancher.

@cjellick cjellick modified the milestones: v2.2, Backlog Sep 18, 2018

@superseb

This comment has been minimized.

Copy link
Member

commented Nov 3, 2018

See #16440 for more info

@ukinau

This comment has been minimized.

Copy link
Contributor

commented Nov 3, 2018

Sorry I didn't know there was ticket for this.
Yesterday we faced this issue and it turns out this is not the bug in flannel. This is the kind of bug in rancher more specifically NodeController and NodeSyncer. This behaviour is still in master branch.
Inside the ticket (#16440) I wrote what's happening and
the solution proposal to prevent from causing this problem, PR will be available in next week.

@superseb

This comment has been minimized.

Copy link
Member

commented Nov 3, 2018

There needs be a decision on a design first, we will review this now we know it seems to be a bug in Rancher and not upstream.

@superseb

This comment has been minimized.

Copy link
Member

commented Nov 3, 2018

@orangedeng can you take another look at this?

@twoodall

This comment has been minimized.

Copy link

commented Nov 7, 2018

Am seeing the same issue on clusters running in aws, where some of the worker nodes run in a spot instance fleet, such that new nodes periodically come and go. A good percentage of these are losing the flanneld annotations on the node, causing other nodes to drop/not create routes, arp entries, etc to the nodes pod network. Restarting the canal/flanneld pod will resolve the issue. Appears to be a race condition at node startup as ticket (#16440) indicated.

@superxor

This comment has been minimized.

Copy link

commented Nov 13, 2018

We seen this issue too multiple times in the last 48h running endless test setups. Custom Node Cluster setup. Latest stable Rancher + Docker 17.03 CE. We also came to the conclusion this is definitely a rancher bug, not flannel

@cjellick

This comment has been minimized.

Copy link
Member

commented Nov 13, 2018

Agreed. We are actively investigating and trying to track down the root cause.

@ukinau

This comment has been minimized.

Copy link
Contributor

commented Nov 15, 2018

Hi did anyone take a look at this ticket? #16440
I meant to explain about the root cause in that ticket.

And actually our environment already changed the behavior of nodesyncer with following patch.
The reason why we need to change the behaviour of nodesyncer is already mentioned in #16440 .

diff --git a/pkg/controllers/user/nodesyncer/nodessyncer.go b/pkg/controllers/user/nodesyncer/nodessyncer.go
index 11cc9c4e..64526ccf 100644
--- a/pkg/controllers/user/nodesyncer/nodessyncer.go
+++ b/pkg/controllers/user/nodesyncer/nodessyncer.go
@@ -143,7 +143,19 @@ func (m *NodesSyncer) syncLabels(key string, obj *v3.Node) error {
 			toUpdate.Labels = obj.Spec.DesiredNodeLabels
 		}
 		if updateAnnotations {
-			toUpdate.Annotations = obj.Spec.DesiredNodeAnnotations
+			// NOTE: This is just workaround.
+			// There are multiple solutions to solve the problem of https://github.com/rancher/rancher/issues/13644
+			// and this problem is kind of desigin bugs. So solving the root cause of problem need design decistions.
+			// That's why for now we solved the problem by the soltion which don't have to change many places. Because
+			// We don't wanna create/maintain large change which have high possibility not to be merged in upstream
+			// The solution is that we would change NodeSyncer so as not to replace annotation with desiredAnnotations but just update annotations which
+			// is specified in desiredAnnotation. This change have side-effects that disable for user to delete exisiting annotation
+			// via desiredAnnotation. but we belived this case is not so famous so we chose this solution
+			for k, v := range obj.Spec.DesiredNodeAnnotations {
+				toUpdate.Annotations[k] = v
+			}
 		}
 		logrus.Infof("Updating node %v with labels %v and annotations %v", toUpdate.Name, toUpdate.Labels, toUpdate.Annotations)
 		if _, err := m.nodeClient.Update(toUpdate); err != nil {

and after we applied this patch, I tried to create new 10 clusters (OpenStack+RKE) which have 50 nodes for each (altogether 500 nodes) and all cluster didn't have this reachability problem and seems to be fixed.

I understand above patch is just workaround but at least the root cause of this problem is already obvious. If I need to provide more information about this problem, I'm willing to provide more information and if anyone agreed on my idea to fix this problem which is described in #16440 not this workaround, I'm willing to provide PR as well.

I hope it will be fixed soon.

@superseb

This comment has been minimized.

Copy link
Member

commented Nov 15, 2018

As noted by @cjellick , this is currently being worked on.

@sangeethah

This comment has been minimized.

Copy link
Member Author

commented Nov 29, 2018

Tested with latest build from master - Nov 26.

This issue is not seen anymore when testing with RKE clusters.

Was also able to test clusters with 30 nodes multiple times and did not hit the issue of flannel annotations missing.

@sangeethah sangeethah closed this Nov 29, 2018

@leodotcloud

This comment has been minimized.

Copy link
Member

commented Nov 29, 2018

@ukinau Thanks for the awesome write-up! Your inputs are highly appreciated.

@ukinau

This comment has been minimized.

Copy link
Contributor

commented Dec 3, 2018

No worries, thanks for quick fix :)

@loganhz loganhz added the team/ca label Dec 18, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.