Pod Networking issue

[jakoguta]

Most of my pods have the following error in the log file:

Get https://<cluster-ip>:<port>/containerLogs/kube-system/digitalocean-flexplugin-deploy-47d5g/flex-deploy?tailLines=5000&amp;timestamps=true: dial tcp <cluster-ip>:<port>: connect: no route to host

I am seeing this is pods like: digitalocean-flexplugin-deploy, digitalocean-provisioner, monitoring-grafana and even the busy-pod.

Running: kubectl get po -n kube-system returns:

NAME                                        READY     STATUS    RESTARTS   AGE
coredns-78fcdf6894-lpnkg                    1/1       Running   0          1d
coredns-78fcdf6894-n8bbp                    1/1       Running   0          1d
digitalocean-flexplugin-deploy-47d5g        1/1       Running   0          1d
digitalocean-flexplugin-deploy-7dchm        1/1       Running   0          1d
digitalocean-flexplugin-deploy-jnflb        1/1       Running   0          1d
digitalocean-provisioner-6dc747b88d-hh7b5   1/1       Running   0          1d
digitalocean-provisioner-6dc747b88d-nn49l   1/1       Running   0          1d
etcd-k8s-master-node                        1/1       Running   0          1d
heapster-6cc56b8ff9-gzmvx                   1/1       Running   0          1d
kube-apiserver-k8s-master-node              1/1       Running   0          1d
kube-controller-manager-k8s-master-node     1/1       Running   0          1d
kube-flannel-ds-gpbbx                       1/1       Running   0          1d
kube-flannel-ds-jmjv8                       1/1       Running   0          1d
kube-flannel-ds-xspq6                       1/1       Running   0          1d
kube-proxy-l5fpv                            1/1       Running   0          1d
kube-proxy-vzb6z                            1/1       Running   0          1d
kube-proxy-zr4vm                            1/1       Running   0          1d
kube-scheduler-k8s-master-node              1/1       Running   0          1d
kubernetes-dashboard-6948bdb78-b49t4        1/1       Running   0          1d
monitoring-grafana-764779695-lcqc6          1/1       Running   0          1d
monitoring-influxdb-6d7bc5546f-dd2k8        1/1       Running   0          1d

In the video I see you are running kube v1.10.* and I am running v1.11.0.

Suggestion: Cloud the move from kubedns to coredns be the issue?

Also: This may be causing the pod to fail to mount the volume:

pod has unbound PersistentVolumeClaims (repeated 2 times)
Unable to mount volumes for pod "busy-pod_default(97000798-9669-11e8-984f-e2571728a743)": timeout expired waiting for volumes to attach or mount for pod "default"/"busy-pod". list of unmounted volumes=[vol1]. list of unattached volumes=[vol1 default-token-72khc]

[chukaofili]

@jakoguta Possibly (on the coredns), although i do know this is a routing issue. Have you tried the setup with kubedns?

[jakoguta]

@chukaofili in kube v1.11, CoreDNS is the default dns server. Due to the fact am not an expert with Kube, I am not confident to change from CoreDNS to KubeDNS. I will have to do some research and see if I can get a fix on this.

[chukaofili]

@jakoguta Ok, but what IP did you use as your advertise address? For your Master?

[jakoguta]

@chukaofili I managed to get it working. It was a mistake on my part that messed the network.

ssh core@$MASTER_IP sudo sysctl net.bridge.bridge-nf-call-iptables=1
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/v0.10.0/Documentation/kube-flannel.yml

I believe I did not run sudo sysctl net.bridge.bridge-nf-call-iptables=1

Thanks for the support, you may close

Edit: I found out that the problem is from the networking part

## Please note that this assumes KUBELET_EXTRA_ARGS hasn’t already been set in the unit file.
echo "Fix Digital Oceans private ip for routing"
IFACE=eth1  # change to eth1 for DO's private network
DROPLET_IP_ADDRESS=$(ip addr show dev $IFACE | awk 'match($0,/inet (([0-9]|\.)+).* scope global/,a) { print a[1]; exit }')
echo $DROPLET_IP_ADDRESS  # check this, just in case
sed -i '2s/^/Environment="KUBELET_EXTRA_ARGS=--node-ip='$DROPLET_IP_ADDRESS'"\n/' /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
systemctl daemon-reload
systemctl restart kubelet

@chukaofili In Kube v1.11 putting Environment="KUBELET_EXTRA_ARGS=--node-ip='$DROPLET_IP_ADDRESS'" in the /etc/systemd/system/kubelet.service.d/10-kubeadm.conf will not work. However, I put it in the file /etc/default/kubelet as shown below and it worked.

IFACE=eth1  # change to eth1 for DO's private network
DROPLET_IP_ADDRESS=$(ip addr show dev $IFACE | awk 'match($0,/inet (([0-9]|\.)+).* scope global/,a) { print a[1]; exit }')
echo $DROPLET_IP_ADDRESS  # check this, just in case
echo "KUBELET_EXTRA_ARGS=--node-ip=$DROPLET_IP_ADDRESS  " > /etc/default/kubelet
systemctl daemon-reload
systemctl restart kubelet

Credits to @jazoom from this issue thread kubernetes/kubeadm#203.

[chukaofili]

@jakoguta Ah i see, thanks for info.

[jakoguta]

@chukaofili the KUBELET_EXTRA_ARGS values below for the persistent volumes should also be placed in the file /etc/default/kubelet for Kube v1.11. I hope you may find time to add this observation to your documentation for those running Kube v1.11+. The busy-pod was not mounting until I moved the KUBELET_EXTRA_ARGS values.

#!/bin/bash

sudo su
mkdir -p /etc/kubernetes/kubelet-plugins/volume
sed -i -e 's#\(KUBELET_EXTRA_ARGS=\)#\1--volume-plugin-dir=/etc/kubernetes/kubelet-plugins/volume #' /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
systemctl daemon-reload
systemctl restart kubelet

[chukaofili]

@jakoguta Yes, just updated the previous one, will update this too now. Thanks for bringing this to my attention.